AI & Technical question

One of Decagon's largest customers has an agent live, but adoption has plateaued: simple cases are automated well, while high-value complex cases still escalate to humans. How would you diagnose whether the main bottleneck is workflow design, knowledge retrieval, tool/API reliability, policy ambiguity, or model behavior, and how would you prioritize the next set of improvements?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests root-cause diagnosis of an adoption plateau across multiple candidate causes and prioritizing fixes by how much each actually explains.

Pull a sample of escalated complex cases and read the actual transcripts, since aggregate escalation rate alone won't reveal the cause.
Check knowledge retrieval first, since a common silent cause is the agent finding stale or incomplete documentation for complex cases.
Check tool and API reliability next, looking for failed or timed-out calls that force escalation regardless of reasoning quality.
Check policy ambiguity, where the agent isn't sure it's allowed to act and escalates defensively even when it could resolve the case.
Only after ruling those out, attribute remaining escalations to model reasoning limits, and prioritize fixes by how many escalations each explains.

Reads actual escalated transcripts to categorize root cause rather than relying on an aggregate escalation metric alone.
Checks retrieval and tool reliability before blaming the model, since those are cheaper to fix and often the real cause.
Quantifies how many escalations each root cause explains, such as assuming 40 percent trace to stale knowledge base articles, to prioritize.
Proposes a fix sequence matched to root-cause share, not a generic recommendation to improve the model.

Assuming the model itself is the bottleneck without first ruling out retrieval, tool reliability, and policy ambiguity.
Prioritizing fixes by intuition instead of quantifying how many escalations each candidate cause actually explains.

How would you distinguish a knowledge-retrieval failure from a genuine model reasoning failure in a transcript?
What would you fix first if the causes were evenly split across all four categories?