Metrics question

One live agent has high conversation volume but low containment, with many users escalating to human support. What metrics would you inspect first, how would you segment the problem, and how would you determine whether the main issue is conversation design, model behavior, or the customer’s backend workflow integration?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests structured diagnosis of a containment problem using metrics segmentation to isolate whether the cause is design, model, or backend integration.

Check volume-to-containment ratio by intent category first, since high overall volume with low containment likely concentrates in a few specific intents, not evenly across all.
Segment escalations by stated reason if available, users explicitly saying 'this isn't working' points to model or design failure; silent drop-off points elsewhere.
Check conversation transcripts for the highest-escalation intents to see if the agent is looping, misunderstanding, or correctly identifying it cannot help.
If the agent correctly identifies it cannot help but escalates anyway, check backend integration, it may lack the system access needed to complete the action.
If the agent seems confident but wrong, treat it as a model or conversation-design issue, not an integration gap.
Prioritize fixing the highest-volume, highest-escalation intent first, since that will move the containment number the most.

Segments by intent category instead of treating 'low containment' as one undifferentiated metric.
Uses transcript review to distinguish an agent that is confidently wrong from one that correctly defers due to a backend gap.
Prioritizes the fix by volume times escalation rate, targeting the intervention that moves the metric most.

Treating containment as one aggregate number without segmenting by intent.
Assuming the cause is always model quality without checking backend integration gaps.

How would you validate your diagnosis before committing engineering time to a fix?
What would you do if the top escalating intent turns out to need a new integration entirely?