Metrics question
A new Arabic-speaking AI agent has been live for 30 days. What metrics would you use to judge whether the launch is successful, and if CSAT improves while containment and resolution efficiency do not, how would you interpret that tradeoff and decide what to do next?
- Sierra
- Metrics
- Medium
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests defining launch-success metrics and interpreting a specific metric divergence, rising CSAT with flat containment and resolution efficiency.
How to approach it
- Define the 30-day success metrics: CSAT, containment rate, resolution efficiency (time and steps to resolve), and escalation rate, as a set, not a single headline number.
- Interpret rising CSAT with flat containment and resolution efficiency carefully: users may be satisfied with the interaction quality even when the agent resolves fewer things independently.
- Check whether CSAT is being driven by politeness and clarity of communication rather than actual task completion, a common pattern in early-stage language launches.
- Check containment and resolution efficiency data by intent to see if a subset of intents is dragging the average down while others perform well.
- Decide the next step based on the divergence: if CSAT is genuine but containment lags, invest in expanding intent coverage rather than in tone or communication style.
- Avoid over-indexing on CSAT alone as proof of success, since it can mask operational inefficiency that will show up in cost later.
What a strong answer includes
- Explains the CSAT-versus-containment divergence with a concrete mechanism, satisfaction from communication quality outpacing actual task completion.
- Segments containment and resolution by intent to find where the operational lag concentrates instead of treating it as uniform.
- Prioritizes intent-coverage investment over tone tuning, correctly reading what the divergence implies about the real gap.
Common mistakes
- Treating rising CSAT alone as proof the launch is successful.
- Not investigating whether the containment lag concentrates in specific intents.
Likely follow-up questions
- How would you distinguish genuine satisfaction from politeness bias in CSAT surveys?
- What would you do if containment stayed flat even after expanding intent coverage?
More metrics questions
- What metrics prove ROI to a Fortune 500 company deploying Sierra?Sierra · Metrics · Hard
- How would you measure customer satisfaction with an AI support agent?Sierra · Metrics · Medium
- What are the most important metrics for an infrastructure platform powering enterprise AI agents, and how would you organize them into a scorecard? Include how you would measure latency, availability, fault tolerance, and developer productivity, and explain which leading indicators you would monitor to catch problems before they show up in customer impact.Sierra · Metrics · Medium
- After launch, how would you measure whether Sierra’s Agent SDK is succeeding? Define the leading and lagging metrics you would use across developer adoption, implementation quality, and downstream end-user outcomes, and explain how those metrics would influence roadmap decisions.Sierra · Metrics · Medium
- For an agent that collects and routes fraud, waste, and abuse reports, what success metrics would you define for both the institution and the end user? If report volume and completion rates are high but downstream resolution quality is poor, how would you diagnose the problem and prioritize fixes?Sierra · Metrics · Hard
- One live agent has high conversation volume but low containment, with many users escalating to human support. What metrics would you inspect first, how would you segment the problem, and how would you determine whether the main issue is conversation design, model behavior, or the customer’s backend workflow integration?Sierra · Metrics · Hard
More questions from Sierra
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Measuring AI Product Economics
- Chapter 2: Using SQL and Logs
- Chapter 14: Interviewing for AI PM Roles