AI & Technical question
A banking or consulting client wants Harvey to automate a high-stakes workflow where user intent is ambiguous, source data is incomplete, and mistakes are costly. How would you decompose the workflow, decide which steps can be fully agentic versus human-in-the-loop, and define the evals, guardrails, and launch criteria needed to judge technical viability?
- Harvey
- AI & Technical
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests decomposing a high-stakes ambiguous workflow into agentic and human-in-the-loop steps, with the evals and guardrails needed to judge technical viability.
How to approach it
- Decompose the workflow into discrete steps first, separating steps with clear, verifiable inputs and outputs from steps requiring judgment on incomplete data.
- Classify each step: fully agentic where outputs are verifiable and mistakes are low-cost or reversible; human-in-the-loop where data is incomplete or errors are costly.
- For agentic steps, define task-level evals measuring accuracy against a labeled reference set, plus a confidence threshold below which the step defers to a human.
- For human-in-the-loop steps, define the agent's role as drafting or flagging, with a clear handoff format the human can quickly verify rather than redo from scratch.
- Define guardrails: hard stops on irreversible actions without explicit human sign-off, and audit logging on every step for post-hoc review.
- Set launch criteria combining step-level eval accuracy, human reviewer agreement rate on flagged cases, and a pilot period before scaling beyond the pilot client.
What a strong answer includes
- Classifies each workflow step individually by verifiability and reversibility, rather than making one blanket agentic-or-not decision for the whole workflow.
- Designs human-in-the-loop steps around fast verification, not full redo, respecting the human reviewer's time while still catching errors.
- Sets a hard guardrail on irreversible actions requiring explicit sign-off, appropriate for high-stakes banking or consulting use.
Common mistakes
- Deciding agentic-versus-human at the whole-workflow level instead of per step.
- No confidence threshold or fallback for agentic steps when the model is uncertain.
Likely follow-up questions
- How would you set the confidence threshold, and how would you validate it is calibrated correctly?
- What would you do if human reviewers start rubber-stamping flagged cases without real review?
More ai & technical questions
- How would you prevent and handle hallucinated case law in Harvey's answers?Harvey · AI & Technical · Hard
- How would you measure the accuracy of Harvey's legal research outputs?Harvey · AI & Technical · Hard
- Vault’s architecture must balance fine-grained document permissions, fast search latency, fresh indexes, and AI workflows over sensitive data. As the PM, how would you define the product requirements, decision criteria, and tradeoffs for this architecture with engineering, and where would you draw the line between PM ownership and engineering ownership?Harvey · AI & Technical · Hard
- Suppose Harvey is building an Outlook or Word integration that drafts grounded work product from matter documents and email history. How would you partner with engineering to make architecture tradeoffs among retrieval depth vs latency, precomputed vs on-demand indexing, model quality vs cost, and strict permissioning vs implementation speed? What would you ship first, and why?Harvey · AI & Technical · Hard
- Harvey can integrate Vault with iManage, SharePoint, Box, and Dropbox, but engineering capacity is limited and each system differs in APIs, authentication, sync models, and permission semantics. How would you prioritize the integrations roadmap, define what is 'good enough' for v1, and make tradeoffs between speed, reliability, permissions fidelity, and user experience?Harvey · AI & Technical · Hard
- Design the first version of Harvey’s cross-system legal research experience for an enterprise customer using systems like iManage, SharePoint, and internal knowledge bases. Which user workflow would you support first, and how would you handle permissions, source freshness, citations or provenance, and failure modes so legal professionals trust the output?Harvey · AI & Technical · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Making AI Product Decisions
- Chapter 8: Designing Evals for AI Products
- Chapter 6: Designing AI Agents and Tooling