Metrics question
What metrics prove Harvey's value to a firm like PwC or Paul Weiss?
- Harvey
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests metrics design for proving enterprise ROI to sophisticated, skeptical buyers in a professional services context.
How to approach it
- Identify what a firm like PwC or Paul Weiss actually cares about: billable hour efficiency, associate throughput, and risk reduction, not raw usage.
- Build a metric tree: time saved per matter as the core driver, split into research time, drafting time, and review time.
- Add an adoption layer: active users per practice group and tasks completed per lawyer per week, since unused seats mean no value delivered.
- Add a quality guardrail: error or hallucination rate on delivered work, since a firm will not credit time saved if it introduces rework risk.
- Translate time saved into dollars using each firm's own billing rates, since the same hours saved are worth more at a higher rate firm.
- Confirm with the interviewer whether the audience is procurement, who cares about cost, or partners, who care about associate leverage and risk.
What a strong answer includes
- Converts time saved directly into a dollar figure using the client's own rates, which is the language procurement and finance actually buy on.
- Separates adoption metrics from outcome metrics, since a firm can have high logins but low real time savings if lawyers just skim outputs.
- Includes a quality or error guardrail metric, since a firm evaluating Harvey cares as much about avoided malpractice risk as about speed.
- Segments by practice group, since litigation, M&A, and in house teams get different value and need different proof points.
Common mistakes
- Reporting only usage numbers like logins or queries, which do not prove business value to a firm's leadership.
- Ignoring quality or accuracy as a metric, when a single bad citation can outweigh hours saved in a law firm's risk calculus.
- Using a generic dollar value instead of the specific firm's own billing rates.
Likely follow-up questions
- How would you prove time saved without just trusting a self reported survey?
- How would this metric set differ for an in house legal team versus a law firm?
- What would convince a skeptical partner who does not trust AI generated work?
More metrics questions
- What KPI hierarchy would you use for Vault, from account-level adoption and active matters to search success, document coverage, and workflow outcomes, to measure value for law firms and enterprises? How would those metrics change your roadmap if usage is high but repeat usage in critical workflows is low?Harvey · Metrics · Hard
- What metrics would you define for Command Center across adoption, admin efficiency, and governance coverage, and how would you use those metrics to decide whether the product is actually improving Harvey’s enterprise land-and-expand motion?Harvey · Metrics · Hard
- Harvey is considering expanding its platform to additional regions. How would you decide whether multi-region expansion is the right product investment now versus later? Explain the customer signals, business considerations, and success metrics you would use, and how you would compare this investment against competing platform priorities.Harvey · Metrics · Hard
- For Vault’s search and Q&A experience, what metrics would you define for adoption, engagement, trust, and business value at the workspace and firm level? Which leading indicators would you watch in the first 90 days, and how would changes in those metrics alter your product decisions?Harvey · Metrics · Hard
- A pilot agent deployment has strong qualitative feedback from senior stakeholders, but end-user adoption is inconsistent and the customer’s data environment is messy. How would you structure the deployment plan, define the success metrics, diagnose whether the issue is workflow fit vs data readiness vs change management, and decide if this should graduate into a repeatable product for the broader vertical?Harvey · Metrics · Hard
- If you joined Harvey, what 2 or 3 AI-assisted workflows would you implement first for the product communications team, and why those first? For each workflow, explain the job to be done, the human review points, the main failure modes or brand risks, and the metrics you would use to decide whether it improved speed or quality enough to scale.Harvey · Metrics · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop