AI & Technical question

A newer frontier model produces better agent responses for Ghostwriter, but it is more expensive, slower, and less predictable. As PM, how would you structure the decision on whether, where, and for whom to use that model in the product?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Whether you can structure a model-selection decision as a segmented product tradeoff rather than a single blanket switch.

Quantify the tradeoff concretely: how much better are the responses (using existing quality evals), how much more expensive, how much slower, and how much less predictable (variance in output quality or latency) the newer model is.
Segment by use case and customer tier rather than deciding for the whole product at once, since a CX manager doing casual drafting has different tolerance than a technical team running high-volume production agents.
Identify use cases where quality improvement matters most and cost or latency sensitivity is lowest, for example complex drafting tasks with lower volume, as the first candidates for the newer model.
Identify use cases where cost and predictability matter most, for example high-volume production agent responses, where the current model likely stays the default until the newer model's cost and reliability improve.
Pilot the newer model on the identified low-risk, high-value segment first, measuring real quality lift and operational cost before expanding.
Set a review cadence to reassess as the newer model's cost and reliability improve over time, since model tradeoffs shift quickly in this space.

Segments the decision by use case and customer tier instead of treating it as one binary switch-or-not decision for the whole product.
Quantifies all three tradeoff dimensions, quality, cost, and predictability, rather than focusing only on the most visible one (quality).
Picks a specific low-risk, high-value pilot segment with clear reasoning, rather than a vague 'we would test it'.
Builds in a recurring reassessment, recognizing model economics and quality change quickly.

Treating this as an all-or-nothing decision instead of segmenting by use case and risk tolerance.
Focusing only on quality improvement without weighing cost and predictability.
No plan to pilot before a broader rollout.

How would you measure predictability concretely for a new model?
What would you do if the newer model shows strong quality but unpredictable cost spikes?
How would you communicate a segmented model strategy to customers who ask which model they are using?