AI & Technical question

Before launching a new AI workflow for a high-volume support use case, what quality bar would you set? Define the offline and online eval framework, launch criteria, and post-launch monitors you would use to measure task success, reliability, safety and groundedness, latency, fallback behavior, and customer trust at scale.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests defining a rigorous pre-launch quality bar for a high-volume AI workflow, spanning offline evals, launch criteria, and post-launch monitoring.

Build an offline eval set covering task success on representative real conversations, including edge cases like ambiguous requests and adversarial inputs.
Define offline thresholds for safety and groundedness specifically, not just task success, since a fluent but ungrounded or unsafe answer can pass a naive accuracy check.
Set launch criteria combining offline eval scores with a limited online pilot, comparing task success, latency, and fallback rate against the human-handled baseline.
Define fallback behavior explicitly as part of the launch bar: the workflow must gracefully hand off on low-confidence cases, not guess.
Set post-launch monitors: real-time task-success sampling, latency percentiles, and a safety-incident flag reviewed daily during the first weeks.
Set a rollback trigger tied to specific thresholds, for example safety-flag rate above a set level, so the response to degradation is pre-decided, not improvised.

Separates safety and groundedness thresholds from task-success accuracy, since a workflow can look accurate while still being ungrounded or unsafe.
Requires an online pilot comparison against the human baseline before full launch, not offline evals alone.
Pre-defines a rollback trigger with a specific threshold, avoiding improvised decisions during a live incident.

Relying on offline evals alone without an online pilot comparison.
No pre-defined rollback trigger, leaving the response to a live degradation improvised.

How would you sample conversations for daily monitoring at high volume?
What would you do if offline evals passed but the online pilot showed a latency regression?