Metrics question
You are leading a Harvey pilot with a corporate legal team on a recurring workflow. What success metrics would you define before launch, how would you separate adoption, workflow quality, and business impact, and what evidence would you require before calling the pilot successful?
- Harvey
- Metrics
- Medium
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests ability to define a rigorous, multi layered success framework for an enterprise pilot before launch, separating adoption from quality from business impact.
How to approach it
- Clarify the recurring workflow in scope, for example contract review, and what success looks like for the legal team specifically.
- Define adoption metrics, such as share of eligible matters routed through Harvey and repeat usage by the same associates week over week.
- Define workflow quality metrics, such as accuracy of flagged issues against a lawyer's own review and time saved per document.
- Define business impact metrics, such as matter turnaround time, tying the pilot to something the client's leadership cares about.
- Set the evidence bar for success up front, requiring quality and business impact to hold, not adoption alone.
What a strong answer includes
- Separates the three metric layers cleanly, for example adoption is a leading indicator while business impact is the metric that justifies expansion.
- Names a concrete quality benchmark, such as comparing Harvey's flagged issues against a senior associate's independent review on a sample of matters.
- Requires business impact evidence, not just usage, before calling the pilot a success, since usage alone does not justify a firm wide rollout.
- Sets the evidence bar before launch, avoiding the trap of retroactively picking whichever metric looks best after the pilot ends.
Common mistakes
- Treating high adoption alone as proof of success without validating quality or business impact.
- Choosing metrics after the pilot ends instead of committing to them beforehand, which invites cherry picking.
Likely follow-up questions
- What would you do if adoption was high but quality metrics were mixed?
- How would you isolate Harvey's impact from other changes happening on the legal team at the same time?
More metrics questions
- What metrics prove Harvey's value to a firm like PwC or Paul Weiss?Harvey · Metrics · Hard
- What KPI hierarchy would you use for Vault, from account-level adoption and active matters to search success, document coverage, and workflow outcomes, to measure value for law firms and enterprises? How would those metrics change your roadmap if usage is high but repeat usage in critical workflows is low?Harvey · Metrics · Hard
- What metrics would you define for Command Center across adoption, admin efficiency, and governance coverage, and how would you use those metrics to decide whether the product is actually improving Harvey’s enterprise land-and-expand motion?Harvey · Metrics · Hard
- Harvey is considering expanding its platform to additional regions. How would you decide whether multi-region expansion is the right product investment now versus later? Explain the customer signals, business considerations, and success metrics you would use, and how you would compare this investment against competing platform priorities.Harvey · Metrics · Hard
- For Vault’s search and Q&A experience, what metrics would you define for adoption, engagement, trust, and business value at the workspace and firm level? Which leading indicators would you watch in the first 90 days, and how would changes in those metrics alter your product decisions?Harvey · Metrics · Hard
- A pilot agent deployment has strong qualitative feedback from senior stakeholders, but end-user adoption is inconsistent and the customer’s data environment is messy. How would you structure the deployment plan, define the success metrics, diagnose whether the issue is workflow fit vs data readiness vs change management, and decide if this should graduate into a repeatable product for the broader vertical?Harvey · Metrics · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop