Metrics question
You are launching an AI agent for a complex support workflow across voice, chat, and SMS. What north-star and guardrail metrics would you set for the first 90 days, how would those metrics differ by channel, and what thresholds would convince you the agent is ready to expand to additional channels, intents, or geographies?
- Decagon
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can set north-star and guardrail metrics for a multi-channel AI agent launch and adapt them meaningfully by channel and by maturity stage.
How to approach it
- Define the north-star metric: task resolution rate without a quality-compromising escalation, tracked consistently across voice, chat, and SMS so channels are comparable.
- Define guardrail metrics: customer satisfaction, response accuracy, and a hard safety guardrail like rate of incorrect actions taken, since a wrong action is worse than a slow or missed response.
- Adapt metrics by channel: voice needs latency and turn-taking quality guardrails that chat and SMS do not; SMS needs a message-length and response-time guardrail specific to that medium's expectations.
- Set 90-day thresholds as go or no-go gates for expansion, for example a minimum resolution rate and a maximum incorrect-action rate that must hold steady, not just trend positively, before expanding to new intents or geographies.
- Require the guardrail metrics to hold across all three channels independently before considering the agent ready to expand, rather than averaging across channels and hiding a weak one.
- Review metrics weekly during the first 90 days given the launch is new and higher-risk, shifting to a lighter cadence once the thresholds are consistently met.
What a strong answer includes
- Uses one comparable north-star metric across channels while layering channel-specific guardrails on top, rather than a single generic metric set.
- Names channel-specific guardrails concretely, latency and turn-taking for voice, message length and response time for SMS, showing real domain fluency.
- Requires guardrails to hold independently per channel before expansion, preventing a strong chat channel from masking a weak voice channel.
- Sets a weekly review cadence appropriate for a new, higher-risk 90-day launch window.
Common mistakes
- Using one identical metric set across voice, chat, and SMS without channel-specific guardrails.
- Averaging metrics across channels, which can hide a channel performing poorly.
- No clear threshold for what triggers expansion versus holding back.
Likely follow-up questions
- How would you weigh a strong chat channel against a weak voice channel when deciding on expansion?
- What would you do if the SMS channel shows good resolution rate but poor customer satisfaction?
- How would you set thresholds differently for a new geography versus a new intent?
More metrics questions
- A large customer has an AI agent live in production, but adoption is below plan and leadership is hesitating on expansion. What metrics would you review first, how would you isolate whether the issue is workflow selection, agent quality, operational rollout, or stakeholder buy-in, and what actions would you take in the next 30 days to improve adoption and demonstrate business impact?Decagon · Metrics · Hard
- You have inherited a new strategic account and must choose the first customer-support workflows to automate in production. What prioritization framework would you use to decide where the agent goes live first, and which adoption, quality, and business metrics would you require before recommending expansion into additional workflows or channels?Decagon · Metrics · Hard
- A live enterprise agent is generating strong customer demand for expansion, but engineers report unresolved reliability gaps in the current design. How would you decide what to ship next, including what evidence or thresholds you would require to expand safely, what you would defer, and how you would manage the conversation with the customer’s leadership team and internal engineering partners?Decagon · Metrics · Hard
- How would you define a metrics framework for Decagon’s developer experience across APIs, SDKs, and headless deployments? Specify the leading and lagging metrics you’d track from integration start through production launch, and explain how those metrics would change your roadmap priorities.Decagon · Metrics · Hard
- One of Decagon's largest customers has launched an agent, but adoption has plateaued because internal teams will not let it handle higher-value interactions. How would you diagnose whether the bottleneck is model quality, workflow design, integration gaps, or change management, and how would you decide which intervention to make first?Decagon · Metrics · Hard
- You own a customer support agent from first production launch through enterprise-wide expansion. What success metrics would you track in the first 30-60 days versus six months later, and how would you balance business outcomes, customer experience, and operational reliability when those metrics conflict?Decagon · Metrics · Hard
More questions from Decagon
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Measuring AI Product Economics
- Chapter 2: Using SQL and Logs
- Chapter 14: Interviewing for AI PM Roles