AI & Technical question
A Computer workflow saves users time, but in 5-10% of sessions the agent takes the wrong action or needs correction. How would you define product quality for this workflow, set launch thresholds, and design an eval and feedback loop to improve reliability after launch?
- Perplexity
- AI & Technical
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can define product quality for an imperfect but net-positive AI workflow, set a realistic launch bar, and design a feedback loop that actually improves reliability after ship.
How to approach it
- Define quality as a composite: time saved on correct sessions weighed against cost of wrong actions, since a wrong action can be more costly than several correct ones are valuable.
- Set the launch threshold based on reversibility: if wrong actions are easily undone, a 5-10% error rate may be acceptable with clear undo; if actions are hard to reverse, the bar should be much stricter.
- Build a feedback loop where every correction or undo is logged as a labeled failure case, feeding a recurring eval set instead of being discarded.
- Add a lightweight in-flow confirmation for higher-risk action types, informed by whichever categories dominate the 5-10% failure rate.
- Review the eval set and error rate weekly post-launch, and set a target trajectory, like halving avoidable errors within a defined number of weeks, not just a static bar.
What a strong answer includes
- Ties the launch threshold to reversibility of the action rather than a single flat error rate for everything.
- Turns every correction into eval data automatically, creating a real improvement loop instead of a one-time launch gate.
- Sets a post-launch trajectory target, showing the quality bar is a process, not a single checkpoint.
Common mistakes
- Sets one flat accuracy threshold regardless of how costly a wrong action actually is.
- Treats the eval and feedback loop as a nice-to-have instead of the core mechanism for improving reliability.
Likely follow-up questions
- How would you categorize wrong actions by severity in the eval set.
- What would you do if error rate stays flat after several improvement cycles.
More ai & technical questions
- You launch a new artifact-generation capability, and users say the outputs are occasionally excellent but too inconsistent to rely on. How would you define quality, build an evaluation framework across model and product layers, and decide whether to improve, constrain, or roll back the feature?Perplexity · AI & Technical · Hard
- An agentic workflow has inconsistent model behavior: for the same user intent, sometimes the agent takes efficient steps and sometimes it opens irrelevant pages or asks unnecessary follow-ups. User feedback is mixed because some sessions feel magical and others fail badly. How would you work with research, engineering, and data science to define quality, instrument the workflow, and decide what to fix first?Perplexity · AI & Technical · Hard
- Design a simple load balancer for Google.com. What data structures would you use?Google · AI & Technical · Hard
- You’re part of the Google Search web spam team. How would you detect duplicate websites?Google · AI & Technical · Hard
- Build a system to predict if something can go wrong in a cab ride.Lyft · AI & Technical · Hard
- Explain the data pipeline for the last AI project you worked on. What were the top challenges in getting data, and how did you resolve them?Google · AI & Technical · Hard
More questions from Perplexity
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Foundations: the model and the decisions it forces on you
- Chapter 8: Evals: define good and make the number defensible
- Chapter 6: Agents and agentic architecture