Metrics question
What metrics would you use to evaluate whether a newly launched Command Center workflow, such as user provisioning or access review, actually improved enterprise adoption and reduced admin burden? Include leading and lagging indicators, and how you would avoid false positives from vanity usage.
- Harvey
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can define an outcome-focused metrics set for an admin workflow feature and specifically guard against vanity usage looking like real adoption.
How to approach it
- Leading indicators: percentage of eligible admins who complete setup of the new workflow within their first month, and time to complete a common admin task like a user provisioning request.
- Lagging indicators: reduction in average time spent on admin tasks per week, and reduction in support or IT tickets related to access requests or user management.
- Define a true adoption signal as repeated, self-initiated use over multiple weeks, not a one-time setup event, since setup completion alone can look like adoption without ongoing value.
- Guard against false positives by checking whether admins who set up the workflow are actually using it for real provisioning events, not just exploring it once and reverting to manual processes.
- Segment by firm size and admin team maturity, since a large firm's admin burden reduction will look different from a small firm's, and blending them can hide whether the feature is genuinely working for the target segment.
What a strong answer includes
- Defines true adoption as sustained, repeated use rather than a one-time setup completion, directly addressing the vanity-usage concern the question raises.
- Pairs leading indicators (setup, time to complete a task) with lagging outcome indicators (ticket reduction, time saved), rather than relying on one or the other alone.
- Segments by firm size, recognizing admin burden and workflow value differ meaningfully across law firm sizes.
Common mistakes
- Counts feature setup or logins as adoption without checking for sustained real usage.
- Uses one blended metric across all firm sizes, hiding whether the feature works for the actual target segment.
Likely follow-up questions
- How would you distinguish a workflow that is used once and abandoned from one delivering ongoing value.
- What would you do if ticket volume drops but admins report the workflow still feels slow.
More metrics questions
- What metrics prove Harvey's value to a firm like PwC or Paul Weiss?Harvey · Metrics · Hard
- What KPI hierarchy would you use for Vault, from account-level adoption and active matters to search success, document coverage, and workflow outcomes, to measure value for law firms and enterprises? How would those metrics change your roadmap if usage is high but repeat usage in critical workflows is low?Harvey · Metrics · Hard
- What metrics would you define for Command Center across adoption, admin efficiency, and governance coverage, and how would you use those metrics to decide whether the product is actually improving Harvey’s enterprise land-and-expand motion?Harvey · Metrics · Hard
- Harvey is considering expanding its platform to additional regions. How would you decide whether multi-region expansion is the right product investment now versus later? Explain the customer signals, business considerations, and success metrics you would use, and how you would compare this investment against competing platform priorities.Harvey · Metrics · Hard
- For Vault’s search and Q&A experience, what metrics would you define for adoption, engagement, trust, and business value at the workspace and firm level? Which leading indicators would you watch in the first 90 days, and how would changes in those metrics alter your product decisions?Harvey · Metrics · Hard
- A pilot agent deployment has strong qualitative feedback from senior stakeholders, but end-user adoption is inconsistent and the customer’s data environment is messy. How would you structure the deployment plan, define the success metrics, diagnose whether the issue is workflow fit vs data readiness vs change management, and decide if this should graduate into a repeatable product for the broader vertical?Harvey · Metrics · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop