Product design question
A top law firm says Vault’s AI search is promising, but attorneys still avoid using it in high-stakes matters. How would you diagnose whether the trust gap comes from retrieval quality, citation and explanation UX, permissioning concerns, workflow fit, or change management, and how would you prioritize the first product changes?
- Harvey
- Product design
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests diagnosing a trust gap in a high-stakes professional tool across five plausible causes, and prioritizing fixes for attorneys who bear real professional risk if they rely on a wrong answer.
How to approach it
- Interview attorneys who avoid Vault on high-stakes matters specifically, since general usage data won't reveal why the most consequential work is excluded.
- Check retrieval quality: does Vault surface the actual controlling or most relevant documents for complex matters, or does it miss key precedent buried in large document sets.
- Check citation and explanation UX: can an attorney quickly verify a claim against the source document, since unverifiable answers are unusable in high-stakes legal work.
- Check permissioning: do attorneys worry Vault might surface privileged or conflicted material it shouldn't have access to, undermining trust regardless of answer quality.
- Check workflow fit and change management: is the trust gap really about the tool, or about firm-wide caution and lack of partner-level endorsement for high-stakes use.
- Prioritize citation and verifiability first if attorneys can't quickly confirm an answer, since that blocks trust even when retrieval is accurate, then address retrieval gaps on complex matters.
What a strong answer includes
- Interviews the specific avoiding segment rather than generalizing from broad usage data that hides this exact gap.
- Separates a real retrieval failure from a verifiability failure, since attorneys may distrust even correct answers they can't quickly check.
- Flags permissioning anxiety as a distinct trust barrier from answer quality, relevant given legal confidentiality obligations.
- Sequences fixes starting with citation and verification UX, since it's often the fastest lever to unlock trust without waiting on deeper retrieval work.
Common mistakes
- Assuming the trust gap is purely a retrieval quality problem without checking verifiability or permissioning concerns.
- Using general usage metrics instead of directly interviewing the attorneys who specifically avoid high-stakes use.
- Ignoring firm-level change management and partner endorsement as a real, non-technical barrier.
Likely follow-up questions
- How would you validate that citation UX is really the top blocker versus retrieval quality?
- What would change management support from firm leadership look like in practice?
More product design questions
- How would you improve Harvey's Vault for bulk document review at large law firms?Harvey · Product design · Hard
- Design a citation system so lawyers can fully trust Harvey's outputs.Harvey · Product design · Hard
- Design a Workflow Builder that lets non-technical lawyers automate recurring tasks.Harvey · Product design · Medium
- Several top law firm customers say they will not expand Harvey without stronger identity and access management. How would you prioritize and scope SSO, SCIM, RBAC, and audit logging for the first two releases of Command Center, and how would you balance enterprise security requirements against implementation complexity and admin usability?Harvey · Product design · Hard
- Lawyers will rarely ask for 'better queuing' or 'stronger isolation,' but they will care deeply about accuracy, latency, reliability, auditability, and security. How would you uncover those needs and translate them into clear infrastructure product requirements and prioritized engineering work? Be specific about discovery methods, requirement framing, and how you would separate must-haves from nice-to-haves.Harvey · Product design · Hard
- You have 4 weeks with a strategic customer in a new professional-services vertical to identify one workflow for an AI-agent pilot. How would you run discovery, score candidate workflows, and decide whether to solve the problem by configuring Harvey’s existing capabilities, building a reusable extension, or not pursuing a bespoke solution at all?Harvey · Product design · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 4: Discovery and strategy for AI products
- Chapter 7: AI UX and human oversight: design for a system that is wrong sometimes
- Chapter 14: Get the job: the AI PM interview loop