AI & Technical question

If you launched a new simulation and evaluation system for Ghostwriter, what success metrics would you use to prove it improves agent quality and business outcomes, and how would you account for the fact that LLM behavior is non-deterministic?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests defining success metrics for an agent simulation and evaluation system, including how to handle non-deterministic LLM behavior in measurement.

Define what the simulation system is meant to catch before a change goes live, such as regressions in task success or policy violations.
Define agent-quality metrics: task success rate, journey adherence, and appropriate escalation, measured across many simulated conversations per scenario.
Define business-outcome metrics the simulation should predict, like containment rate, and validate correlation with real production outcomes.
Account for non-determinism by running each scenario multiple times and reporting a distribution rather than a single pass or fail.
Use the simulation as a gate, so a change only ships if its outcome distribution clears a set bar versus the live agent's baseline.

Explicitly runs each test scenario multiple times to get a distribution rather than trusting a single non-deterministic run.
Validates that simulation scores actually predict real-world outcomes against a held-out sample of production data.
Separates agent-quality metrics, like journey adherence, from business metrics, like case resolution, since a correct answer can still fail the customer.
Sets a statistically meaningful comparison threshold against the current baseline agent, not just whether it passed.

Treating a single simulation run as ground truth despite known LLM non-determinism.
Measuring only technical task success without validating correlation to real business outcomes.

How many simulation runs per scenario would you consider statistically sufficient?
What would you do if simulation scores improved but production satisfaction didn't?