Metrics question

For an agent that collects and routes fraud, waste, and abuse reports, what success metrics would you define for both the institution and the end user? If report volume and completion rates are high but downstream resolution quality is poor, how would you diagnose the problem and prioritize fixes?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests defining two-sided success metrics for a reporting agent and diagnosing a completion-quality gap in downstream resolution.

For the institution, define metrics on report quality and actionability: percent of reports with sufficient detail for investigation, and percent leading to a resolved case.
For the end user, define metrics on experience: completion rate, time to submit, and perceived clarity of what happens after submission.
If volume and completion are high but resolution quality is poor, first check whether reports are structurally complete, since a completed form can still be low-signal.
Check whether the agent is asking the right clarifying questions for triage, like specifics needed for investigators, versus just collecting whatever the user offers.
Check the handoff itself: are completed reports actually reaching the right downstream team with the right priority tagging, or getting lost in an unrouted queue.
Prioritize fixing intake question design first if structural completeness is the gap, since that is the cheapest, highest-leverage fix before touching routing infrastructure.

Separates institution-side and user-side metrics instead of one blended satisfaction score.
Diagnoses the high-completion, low-resolution gap by checking structural report quality before jumping to routing infrastructure.
Prioritizes the cheapest high-leverage fix, better intake questions, before larger routing changes.

Treating high completion rate alone as proof the agent is working.
Jumping to a routing-infrastructure rebuild without checking report quality first.

How would you measure whether an intake-question redesign actually improved resolution?
What would you do if routing is the real bottleneck instead?