Metrics question

A national health plan wants an agent that can answer questions like "What is my copay for a primary care visit?" and "How many physical therapy visits do I have left?" How would you define the MVP scope, fallback and escalation paths, and launch criteria when source data may be inconsistent and mistakes could erode trust? What metrics would you track in the first 90 days after launch?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests MVP scoping and trust focused launch criteria for a benefits Q&A agent where inaccurate answers carry real financial and trust consequences.

Scope the MVP to the highest confidence data sources first, for example clear copay amounts, deferring ambiguous benefit categories where source data is known to be inconsistent.
Define a confidence threshold below which the agent must decline to answer and escalate to a human rather than guess.
Design fallback language that is honest about uncertainty, for example directing the member to a phone line for edge cases, instead of a generic apology.
Set launch criteria requiring a minimum accuracy rate on a held out sample of common questions, verified by human review against source plan documents.
Track containment rate, accuracy rate against verified answers, escalation rate, and member complaint rate in the first 90 days.
Review a sample of escalated and declined cases weekly early on to catch source data inconsistencies before they erode trust broadly.

Ties the confidence threshold and fallback directly to the stated risk, inconsistent source data, rather than assuming the model will just handle it.
Requires human verified accuracy on a held out sample as a launch gate, not just internal confidence in the model.
Names complaint rate specifically as a 90 day metric, since trust erosion in benefits Q&A shows up there before it shows up in usage metrics.

Launching without a clear confidence threshold, letting the agent guess on ambiguous benefits questions where source data is unreliable.
Measuring only containment rate in the first 90 days without a paired accuracy or trust metric.

How would you decide the confidence threshold for declining to answer?
What would you do if source data inconsistency turns out to be worse than expected after launch?