Metrics question

What metrics would you use to judge whether a legal AI product is working in a 5-customer pilot versus a scaled rollout? Be specific about user-value, trust/quality, operational, and business metrics, and explain which ones are leading indicators versus launch gates.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Whether the candidate can define different success bars for an early pilot versus a scaled rollout, not one static scorecard.

Propose pilot-stage metrics, deliberately small and qualitative: direct user-value feedback from the five customers, like time saved per task, and trust and quality signals from close-read review of a sample of outputs by legal experts.
Propose operational pilot metrics: task completion rate and manual correction rate per output, tracked closely given the small, observable sample.
Propose scaled-rollout metrics: aggregate business metrics like customer retention and expansion, alongside statistically robust quality metrics no longer needing full manual review of every output, just a monitored sample.
Classify leading versus launch gates: pilot-stage expert quality review is a launch gate for scaling, while aggregate retention and usage growth are leading indicators once scaled.
Note the transition point explicitly: moving from pilot to scaled rollout requires the pilot's quality bar to hold steady as volume grows, not just that early adopters were satisfied.

Distinguishes pilot metrics, which can be closely observed and qualitative given small scale, from scaled metrics, which must be statistically robust and operationally lighter-touch.
Names a concrete pilot-stage metric, expert legal review of outputs, as a hard gate rather than a nice-to-have signal.
Proposes business metrics, retention and expansion, specifically for the scaled stage, since a five-customer pilot is too small to draw reliable business conclusions from.
Explains the transition condition clearly, that quality must hold as volume grows, addressing the real risk of a pilot succeeding on hand-holding rather than the product itself.

Using the same metrics identically for both stages, ignoring that a five-customer pilot allows close observation that scaled rollout cannot sustain.
Proposing only business metrics with no mention of quality or trust signals, missing the point of measuring whether the product is actually working.

How would you decide the pilot is ready to scale beyond five customers?
What would you do if quality held in the pilot but degraded once volume increased?