AI & Technical question

A major customer reports that Sierra’s agent resolves simple requests well but breaks down in complex, multi-step conversations that depend on business-process rules. How would you diagnose the failure modes, decide whether the root cause is in instructions, knowledge, tool use, workflow design, or escalation logic, and define the evaluation metrics you’d use to know the agent is actually improving?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Whether you can decompose a complex agent-failure complaint across the right layer of the stack and define metrics that reflect genuine improvement.

Reproduce the failure pattern with real transcripts of complex, multi-step conversations that broke down, rather than relying on the customer's summary alone.
Check instructions first: are the business-process rules actually captured correctly in the agent's configuration, since a missing or ambiguous rule looks identical to a reasoning failure from the outside.
Check knowledge next: does the agent have access to the specific business-process documentation needed for this customer's edge cases.
Check tool use and workflow design: does the agent call the right tools in the right order for multi-step processes, and does the workflow definition itself account for branching business logic.
Check escalation logic last: when the agent is uncertain in a complex case, does it escalate cleanly, or does it guess or loop.
Define evaluation metrics matched to this diagnosis: task success rate specifically on multi-step, rule-dependent conversations, plus escalation appropriateness (escalating when it should, not when it should not), tracked separately from simple-request success rate.

Separates instructions, knowledge, tool use, workflow design, and escalation logic as five distinct, individually testable causes rather than one vague 'the agent is bad at complex tasks'.
Notes that missing instructions can look like a reasoning failure, a genuinely useful diagnostic insight.
Proposes a metric split between simple and complex, rule-dependent conversations, since blending them would hide the real problem.
Names escalation appropriateness as its own metric, since escalating too much or too little are both failures worth tracking separately.

Treating this as purely a model reasoning problem without checking instructions and knowledge first.
Using one blended success-rate metric that does not separate simple from complex conversations.
No distinct metric for escalation quality, which is central to this kind of failure.

How would you tell a missing instruction apart from a genuine reasoning failure?
What would you do if the business-process rules themselves are inconsistent or undocumented?
How would you validate that a fix actually improved complex-conversation success rate?