AI & Technical question
An agentic workflow has inconsistent model behavior: for the same user intent, sometimes the agent takes efficient steps and sometimes it opens irrelevant pages or asks unnecessary follow-ups. User feedback is mixed because some sessions feel magical and others fail badly. How would you work with research, engineering, and data science to define quality, instrument the workflow, and decide what to fix first?
- Perplexity
- AI & Technical
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests defining agentic-workflow quality with research and engineering, instrumenting the workflow, and deciding what to fix first amid inconsistent behavior.
How to approach it
- Define quality dimensions explicitly: step efficiency (unnecessary actions taken), task success (goal achieved), and unnecessary clarification requests, since 'inconsistent' bundles several distinct failure modes.
- Instrument the workflow to log every action taken, its rationale if available, and whether it was necessary for the eventual outcome.
- Segment sessions into 'magical' and 'failed' groups and compare their action traces directly to find the specific decision point where they diverge.
- Work with data science to quantify how often unnecessary actions occur and whether they correlate with a specific input pattern or ambiguity type.
- Prioritize fixing the highest-frequency divergence point first, likely a specific ambiguous-intent pattern causing wasted exploration, rather than a general model retrain.
- Set an ongoing eval metric, like average unnecessary-action count per session, to track whether the fix actually reduced inconsistency.
What a strong answer includes
- Breaks 'inconsistent' into specific measurable dimensions, step efficiency and task success, instead of treating it as one vague quality problem.
- Uses direct trace comparison between magical and failed sessions to localize the actual divergence point.
- Prioritizes the highest-frequency failure pattern rather than defaulting to a broad model retrain as the first move.
Common mistakes
- Treating 'inconsistent behavior' as one problem without breaking it into measurable dimensions.
- Jumping to a model retrain before localizing the specific divergence pattern.
Likely follow-up questions
- How would you validate the fix generalizes beyond the sampled divergence cases?
- What would you do if the divergence correlates with input ambiguity that is hard to detect upfront?
More ai & technical questions
- You launch a new artifact-generation capability, and users say the outputs are occasionally excellent but too inconsistent to rely on. How would you define quality, build an evaluation framework across model and product layers, and decide whether to improve, constrain, or roll back the feature?Perplexity · AI & Technical · Hard
- A Computer workflow saves users time, but in 5-10% of sessions the agent takes the wrong action or needs correction. How would you define product quality for this workflow, set launch thresholds, and design an eval and feedback loop to improve reliability after launch?Perplexity · AI & Technical · Hard
- Design a simple load balancer for Google.com. What data structures would you use?Google · AI & Technical · Hard
- You’re part of the Google Search web spam team. How would you detect duplicate websites?Google · AI & Technical · Hard
- Build a system to predict if something can go wrong in a cab ride.Lyft · AI & Technical · Hard
- Explain the data pipeline for the last AI project you worked on. What were the top challenges in getting data, and how did you resolve them?Google · AI & Technical · Hard
More questions from Perplexity
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Making AI Product Decisions
- Chapter 8: Designing Evals for AI Products
- Chapter 6: Designing AI Agents and Tooling