AI & Technical question
An enterprise agent is live for a large customer, and containment is improving but CSAT is flat. How would you identify the highest-priority workflow failures? Specify the telemetry you would use, how you would separate model-quality issues from workflow or integration issues, and how you would direct Forward Deployed Engineers on the order of fixes.
- Decagon
- AI & Technical
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can diagnose a flat-CSAT-despite-improving-containment problem using real telemetry and direct forward-deployed engineers on a prioritized fix order.
How to approach it
- Check for the classic false-resolution trap first: rising containment paired with flat CSAT often means the agent is closing conversations without truly resolving the customer's issue, so review a sample of contained conversations for actual resolution quality.
- Use telemetry that separates model-quality issues (wrong or unhelpful responses despite correct context) from workflow or integration issues (correct response but the underlying action failed or data was stale), for example response accuracy scores versus downstream action success logs.
- Segment CSAT by conversation type to find where the gap concentrates, since a blended CSAT score can hide a badly underperforming segment offsetting strong performance elsewhere.
- Cross-reference contained-but-low-CSAT conversations against telemetry to identify whether the pattern points to model quality or workflow and integration issues.
- Direct forward-deployed engineers to fix the highest-frequency, highest-CSAT-impact issue first, whichever layer it is in, rather than defaulting to model tuning as the first lever.
- Set a new success bar: track CSAT specifically on contained conversations going forward, not just overall containment rate, to catch this failure pattern earlier next time.
What a strong answer includes
- Names the false-resolution trap explicitly as the leading hypothesis, a precise, non-obvious diagnostic insight directly relevant to this exact metric pattern.
- Separates model-quality telemetry from workflow and integration telemetry as two distinct diagnostic tracks.
- Segments CSAT by conversation type instead of relying on the blended aggregate, which is what hides the real problem.
- Proposes tracking CSAT specifically on contained conversations as a new standing metric, directly preventing this exact failure pattern from going unnoticed again.
Common mistakes
- Assuming rising containment is automatically good news without checking for false resolution.
- Not separating model-quality from workflow and integration telemetry, leading to the wrong fix.
- Using only blended CSAT, which can hide a badly underperforming segment.
Likely follow-up questions
- How would you sample conversations at scale to check for false resolution?
- What would you do if the issue turns out to be a stale integration your team does not own?
- How would you prevent containment rate from being over-weighted as a success signal again?
More ai & technical questions
- During pre-sales for a Fortune 500 support organization, the buyer believes Decagon’s AI agent vision is compelling but doubts it can safely automate high-volume interactions across chat, email, SMS, and voice. How would you structure the technical proof process: which workflows would you include in the initial pilot, what evals, guardrails, and human-escalation design would you use to demonstrate safety and quality, and how would you translate the results into a scope the customer is willing to sign?Decagon · AI & Technical · Hard
- How would you design an evaluation framework for Duet that measures two things separately: whether it correctly identifies agent failures in production conversations, and whether its proposed fixes are actually high quality before they reach production? Define the labels, offline and online evals, human review criteria, and failure modes you’d want the team to track.Decagon · AI & Technical · Hard
- Decagon wants enterprises to control how agents change in production. Define the product requirements for versioning, approvals, rollout/rollback, evaluation gates, and auditability. Then describe the minimum API surface, core data model, and architecture choices you would align on with engineering to support safe agent updates at enterprise scale.Decagon · AI & Technical · Hard
- Design a product that analyzes conversations across voice, chat, email, and SMS and gives support teams actionable recommendations (for example, workflow fixes, agent behavior changes, or knowledge-base gaps). What jobs-to-be-done would you prioritize first, what should the UI show so teams trust and act on the recommendations, and how would you handle confidence, evidence/citations, and feedback loops to improve recommendation quality over time?Decagon · AI & Technical · Hard
- A Fortune 500 prospect wants to automate high-value support interactions across chat, email, SMS, and voice, but the C-suite doubts an AI agent can do this safely. How would you structure a pre-sales pilot: which workflows would you start with, what model failure modes and guardrails would you test, what offline and online evals and success thresholds would you require, and how would those results translate into a full production rollout?Decagon · AI & Technical · Hard
- One of Decagon's largest customers has an agent live, but adoption has plateaued: simple cases are automated well, while high-value complex cases still escalate to humans. How would you diagnose whether the main bottleneck is workflow design, knowledge retrieval, tool/API reliability, policy ambiguity, or model behavior, and how would you prioritize the next set of improvements?Decagon · AI & Technical · Hard
More questions from Decagon
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Making AI Product Decisions
- Chapter 8: Designing Evals for AI Products
- Chapter 6: Designing AI Agents and Tooling