Metrics question

A launched Sierra agent has increased containment significantly, but CSAT is flat and escalations are rising. How would you diagnose the issue? What metrics and instrumentation would you review at the conversation, intent, and handoff levels, how would you segment the problem, and what improvements would you prioritize first?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests structured diagnosis of a specific automation paradox, rising containment but flat CSAT and rising escalations, using layered instrumentation.

Review conversation level metrics first: are escalations happening after multiple failed automated attempts, suggesting the agent is trying too hard to contain difficult cases.
Review intent level metrics: segment containment and CSAT by intent to see if certain intents are being over automated relative to their real complexity.
Review handoff level metrics: when escalation does happen, check if the human agent has full context, since a poor handoff itself can suppress CSAT even after a correct escalation.
Check if containment gains came from stricter definitions of what counts as contained, which could mask cases that were technically closed but left the customer unsatisfied.
Segment the problem by customer type or issue complexity to see if the paradox concentrates in a specific segment rather than being uniform.
Prioritize fixes based on where the data points: narrowing automation scope on over automated intents, or fixing handoff context if that is the bigger driver.

Checks whether the definition of contained itself changed, a subtle but common cause where a metric improves on paper while the real customer experience does not.
Reviews handoff context quality specifically, recognizing that CSAT can suffer from a bad escalation experience even when the escalation decision itself was correct.
Segments by intent and customer type to find where the paradox concentrates rather than treating it as a uniform, system wide problem.

Assuming rising containment is unambiguously good without checking whether the definition of contained itself may be too loose.
Fixing the automation broadly without checking whether poor handoff context, not the escalation decision itself, is driving flat CSAT.

How would you tell if the containment definition itself needs tightening?
What would you do if the paradox is concentrated in one specific intent rather than spread evenly?