Metrics question

A new core-model capability shows strong offline gains but increases latency and cost. How would you define launch gates that combine offline evaluations and online product metrics, and what criteria would determine full launch, limited rollout, or no-ship?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Ability to design a launch gating framework that weighs model quality gains against real product costs like latency and infrastructure spend.

Define the offline gate first: the capability must clear a minimum quality improvement threshold on the relevant eval set, a floor below which no launch conversation happens.
Define the online gate: run a limited rollout measuring the actual user facing tradeoff, quality improvement in real usage against latency increase and cost per request.
Set explicit thresholds for each outcome, for example full launch if user facing quality metrics improve and latency stays within an acceptable added second budget, limited rollout if quality improves but cost is high and only justified for a premium tier, no ship if online quality gains do not materialize despite offline gains.
Segment by use case, since latency sensitive contexts like chat may tolerate less added delay than an asynchronous batch use case.
Include a cost per quality point calculation so leadership can compare this investment against other roadmap options.
Define the rollback trigger if the limited rollout later shows the tradeoff was not worth it.

Sets concrete threshold values, even as flagged assumptions, rather than vague if it looks good criteria.
Explicitly separates the three outcomes, full launch, limited rollout, no ship, with a distinct trigger for each.
Segments by use case, recognizing latency tolerance differs by context.
Includes a cost per quality point framing to make the tradeoff legible to leadership.

Using only the offline eval to decide launch and ignoring real latency and cost impact on users.
Treating the decision as binary, ship or not, without considering limited rollout as a real option.

Chapters of the AI PM course that teach what this question tests.