AI & Technical question

Sierra has launched a mortgage pre-approval or lost-card support agent that now handles thousands of conversations per day. Conversion is flat, complaint rate is rising, and human reviewers are finding inconsistent responses. How would you build an evaluation framework for the agent, identify the highest-risk failure modes, and decide what to fix first? Include the metrics, offline and online evals, and guardrails you would use.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests the ability to build an evaluation framework for a struggling live agent and prioritize fixes using both metrics and human review.

Segment the flat conversion and rising complaints by intent, customer segment, and conversation length to find where the agent is actually failing.
Build offline evals against a curated set of historical conversations, scoring accuracy, policy adherence, and tone consistency.
Run online evals in production, sampling live conversations for human review, since offline sets cannot catch every real world edge case.
Identify highest risk failure modes first, such as incorrect pre approval terms or unauthorized card actions, since those carry regulatory and brand risk.
Set guardrails, like mandatory human review above a certain loan amount or hold thresholds, to bound risk while fixes are in progress.
Prioritize fixes by a combination of frequency and severity, fixing high frequency low severity issues in parallel with rare high severity ones.

Separates offline and online evals clearly and explains why production sampling catches issues a static eval set misses.
Names concrete high risk failure modes specific to mortgage or card support, like misquoted terms or wrongful card locks, not generic errors.
Proposes a guardrail, such as a dollar or risk threshold for mandatory human review, that bounds harm while the team investigates.

Treating flat conversion and rising complaints as the same problem instead of investigating them as separate signals.
Fixing the most visible complaints first without checking whether they are the highest severity or highest frequency issue.

How would you decide the threshold for mandatory human review on a financial decision like this?
What would you do if the inconsistent responses trace back to the underlying model rather than your prompts or workflow?