Metrics question
For Vault’s search and Q&A experience, what metrics would you define for adoption, engagement, trust, and business value at the workspace and firm level? Which leading indicators would you watch in the first 90 days, and how would changes in those metrics alter your product decisions?
- Harvey
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests defining a metrics framework across adoption, engagement, trust, and business value at two organizational levels, with leading indicators for the first 90 days.
How to approach it
- Define workspace-level metrics: percent of workspaces with at least one search or Q&A query weekly, and query volume per active user.
- Define firm-level metrics: percent of the firm's licensed attorneys who have used Vault at least once, and repeat usage rate over 30 days.
- Define trust metrics specifically: citation-click-through rate, since lawyers verifying sources is a strong proxy for whether they trust the answer enough to act on it.
- Define business value metrics: time saved per query versus a manual research baseline, and matters where Vault findings were cited in real work product.
- In the first 90 days, watch leading indicators: week-over-week active user growth and citation-click-through rate, since both predict whether trust and habit are forming early.
- If citation-click-through is low despite high query volume, treat it as a trust or answer-quality problem and prioritize accuracy and citation clarity over new features.
What a strong answer includes
- Uses citation-click-through rate specifically as a trust proxy, a concrete metric suited to a legal research tool, not a generic engagement number.
- Separates workspace-level and firm-level metrics, since a single power-user workspace can mask weak firm-wide adoption.
- Ties a specific leading-indicator pattern, high volume with low click-through, to a concrete roadmap response.
Common mistakes
- Using only firm-wide averages that hide which workspaces are actually engaged.
- Treating query volume alone as proof of trust without checking citation verification behavior.
Likely follow-up questions
- How would you distinguish real trust from lawyers rubber-stamping citations without checking them?
- What would you do if adoption is strong but time-saved data is inconclusive?
More metrics questions
- What metrics prove Harvey's value to a firm like PwC or Paul Weiss?Harvey · Metrics · Hard
- What KPI hierarchy would you use for Vault, from account-level adoption and active matters to search success, document coverage, and workflow outcomes, to measure value for law firms and enterprises? How would those metrics change your roadmap if usage is high but repeat usage in critical workflows is low?Harvey · Metrics · Hard
- What metrics would you define for Command Center across adoption, admin efficiency, and governance coverage, and how would you use those metrics to decide whether the product is actually improving Harvey’s enterprise land-and-expand motion?Harvey · Metrics · Hard
- Harvey is considering expanding its platform to additional regions. How would you decide whether multi-region expansion is the right product investment now versus later? Explain the customer signals, business considerations, and success metrics you would use, and how you would compare this investment against competing platform priorities.Harvey · Metrics · Hard
- A pilot agent deployment has strong qualitative feedback from senior stakeholders, but end-user adoption is inconsistent and the customer’s data environment is messy. How would you structure the deployment plan, define the success metrics, diagnose whether the issue is workflow fit vs data readiness vs change management, and decide if this should graduate into a repeatable product for the broader vertical?Harvey · Metrics · Hard
- If you joined Harvey, what 2 or 3 AI-assisted workflows would you implement first for the product communications team, and why those first? For each workflow, explain the job to be done, the human review points, the main failure modes or brand risks, and the metrics you would use to decide whether it improved speed or quality enough to scale.Harvey · Metrics · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Measuring AI Product Economics
- Chapter 2: Using SQL and Logs
- Chapter 14: Interviewing for AI PM Roles