Metrics question
What KPI hierarchy would you use for Vault, from account-level adoption and active matters to search success, document coverage, and workflow outcomes, to measure value for law firms and enterprises? How would those metrics change your roadmap if usage is high but repeat usage in critical workflows is low?
- Harvey
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests building a KPI hierarchy from account adoption down to workflow outcomes, and using it diagnostically when usage is high but repeat use in critical workflows is low.
How to approach it
- Build the hierarchy top-down: account adoption, active-matters coverage, search success (query-to-useful-answer rate), document coverage, and workflow outcomes like time saved in due diligence review.
- Define search success narrowly, a query followed by a citation click with no immediate re-query, as a proxy for a genuinely useful answer.
- Treat workflow outcomes as the top KPI for renewal, since general activity can be high while the product never touches billable, high-stakes work.
- If usage is high but critical-workflow repeat use is low, drill into document coverage first, since attorneys won't trust Vault if key documents aren't indexed.
- If coverage is fine, check search success specifically on critical-workflow queries, since aggregate success can mask poor performance on complex questions.
- Use the diagnosis to redirect the roadmap toward coverage or search quality for critical workflows, not broad features that boost general activity without fixing the gap.
What a strong answer includes
- Builds a genuine hierarchy from coarse adoption down to workflow outcomes, rather than a flat list of disconnected metrics.
- Defines search success operationally, no immediate re-query after a citation click, instead of a vague satisfaction proxy.
- Uses the high-usage-low-critical-repeat pattern as a real diagnostic signal pointing at coverage or workflow-specific search quality.
Common mistakes
- Treating overall usage as sufficient proof of value without checking whether it reaches critical, billable workflows.
- Measuring search success in aggregate, which can hide poor performance on the complex queries that matter most.
- Adding general features in response to weak critical-workflow adoption instead of diagnosing the specific root cause.
Likely follow-up questions
- How would you segment matters by criticality to make this diagnosis sharper?
- What would you do if coverage and search quality both look fine but adoption is still low?
More metrics questions
- What metrics prove Harvey's value to a firm like PwC or Paul Weiss?Harvey · Metrics · Hard
- What metrics would you define for Command Center across adoption, admin efficiency, and governance coverage, and how would you use those metrics to decide whether the product is actually improving Harvey’s enterprise land-and-expand motion?Harvey · Metrics · Hard
- Harvey is considering expanding its platform to additional regions. How would you decide whether multi-region expansion is the right product investment now versus later? Explain the customer signals, business considerations, and success metrics you would use, and how you would compare this investment against competing platform priorities.Harvey · Metrics · Hard
- For Vault’s search and Q&A experience, what metrics would you define for adoption, engagement, trust, and business value at the workspace and firm level? Which leading indicators would you watch in the first 90 days, and how would changes in those metrics alter your product decisions?Harvey · Metrics · Hard
- A pilot agent deployment has strong qualitative feedback from senior stakeholders, but end-user adoption is inconsistent and the customer’s data environment is messy. How would you structure the deployment plan, define the success metrics, diagnose whether the issue is workflow fit vs data readiness vs change management, and decide if this should graduate into a repeatable product for the broader vertical?Harvey · Metrics · Hard
- If you joined Harvey, what 2 or 3 AI-assisted workflows would you implement first for the product communications team, and why those first? For each workflow, explain the job to be done, the human review points, the main failure modes or brand risks, and the metrics you would use to decide whether it improved speed or quality enough to scale.Harvey · Metrics · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop