Metrics question

After launching an enterprise AI agent, what primary success metric and guardrail metrics would you track to determine whether it is succeeding? How would you balance automation goals with resolution quality, CSAT, escalation rate, and trust-related failures such as incorrect or unsafe answers?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Ability to pick a primary metric for an AI agent that reflects true success, not just automation volume, and pair it with the right guardrails.

Choose the primary metric as a quality adjusted resolution rate, for example the percentage of conversations fully and correctly resolved without human escalation, rather than raw automation or containment rate alone.
Add CSAT as a guardrail specifically on automated conversations, since a high resolution rate achieved with poor customer experience is not real success.
Add escalation rate as a second guardrail, watching for both too high escalation, indicating the agent is not adding value, and suspiciously low escalation, which can indicate the agent is not escalating when it should.
Add a trust specific guardrail tracking incorrect or unsafe answers, measured through periodic human audit of a sample of automated conversations, since this will not reliably surface through CSAT or resolution metrics alone.
Set thresholds that trigger a pause or rollback, for example if the unsafe answer rate crosses a defined threshold, that overrides a strong resolution rate number.
Review all metrics together on a regular cadence so automation gains are never interpreted as success in isolation from the quality guardrails.

Chooses a quality adjusted primary metric instead of raw automation or containment rate, avoiding the trap of rewarding volume over correctness.
Names a specific trust guardrail, an audited unsafe answer rate, that would not be caught by CSAT or resolution numbers alone.
Flags both directions of escalation rate risk, too high and suspiciously too low, showing nuanced understanding.
Defines an override rule where a guardrail breach caps a good primary metric result.

Picking raw automation or containment rate as the sole success metric, which rewards the agent for resolving things badly rather than well.
No mechanism to catch unsafe or incorrect answers beyond relying on CSAT, which will not reliably surface every trust failure.

Chapters of the AI PM course that teach what this question tests.