Metrics question

Sierra aims to deliver a better, more human customer experience with AI. For a newly launched enterprise agent, what north-star metric, supporting metrics, and guardrails would you track? How would you balance automation/containment against customer trust, quality, and escalation outcomes?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests north star metric design for a support agent that must balance automation with genuine customer trust and quality outcomes.

Choose a north star that captures both efficiency and human quality, such as resolution rate weighted by customer satisfaction, rather than containment alone.
Track supporting metrics for automation: containment rate, average handle time, and cost per resolution.
Track supporting metrics for trust: CSAT, complaint rate, and repeat contact rate for the same issue.
Track guardrails: escalation appropriateness, meaning cases correctly routed to a human, and critical error rate.
Explicitly monitor for the tradeoff where containment rises while CSAT falls, since that signals over automation of cases better handled by a human.
Review these metrics together weekly, using containment and CSAT as a paired check rather than either one in isolation.

Chooses a north star that combines efficiency and quality rather than optimizing purely for containment, matching Sierra's stated human centered goal.
Explicitly names the containment versus CSAT tradeoff as the key thing to monitor, since it is the most common failure pattern in this kind of launch.
Includes escalation appropriateness as a distinct guardrail from critical error rate, since wrongly withheld escalation is its own failure mode.

Optimizing the launch purely for containment rate without a paired quality or trust metric to catch over automation.
Treating CSAT and containment as independent metrics instead of reviewing them together to catch the tradeoff early.

What threshold would make you scale back automation on a specific intent?
How would you separate a genuine quality problem from customers who simply prefer talking to a human?