Metrics question
You have 90 days to launch a new integration that brings matter data from a strategic partner into Harvey. How would you structure the launch across product, engineering, applied AI, platform, security, and partnerships, and what metrics would you use in the first 30 and 90 days to decide whether to expand, iterate, or stop?
- Harvey
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can structure a fast, cross-functional partner integration launch and define staged metrics that drive a real expand, iterate, or stop decision.
How to approach it
- Break the 90 days into phases with named owners: technical integration and security review (product, engineering, security), applied AI tuning for the new data source (applied AI, platform), and partnership terms and support readiness (partnerships).
- Define 30-day metrics focused on technical health: integration uptime, data-sync accuracy, and time to surface partner matter data inside Harvey's workflow.
- Define 90-day metrics focused on adoption and value: percent of eligible lawyers using the integrated data, frequency of reuse in active matters, and qualitative feedback on relevance.
- Set explicit decision criteria in advance for each checkpoint, for example if 30-day technical metrics are weak, delay the adoption push rather than launching broadly on a shaky foundation.
- Run a tight cross-functional standup (product, engineering, applied AI, platform, security, partnerships) through the 90 days to catch integration or security blockers early given the compressed timeline.
- At day 90, use the adoption and reuse data against pre-set thresholds to decide expand, iterate, or stop, rather than defaulting to continue by inertia.
What a strong answer includes
- Splits metrics cleanly into a technical-health 30-day gate and an adoption-and-value 90-day gate, matching what is actually knowable at each stage.
- Sets decision criteria before the data comes in, avoiding a post-hoc rationalization of whatever the numbers show.
- Names reuse in active matters, not just login or query count, as the real adoption signal for legal professionals, showing domain judgment.
- Structures the cross-functional cadence around the compressed 90-day timeline specifically, not a generic weekly sync.
Common mistakes
- Using the same metrics for both the 30-day and 90-day checkpoints instead of differentiating technical health from adoption.
- No pre-set decision thresholds, leading to an ambiguous or biased expand/iterate/stop call.
- Vague cross-functional structure with no named owners for the compressed timeline.
Likely follow-up questions
- What would you do if technical metrics are strong at 30 days but adoption is weak at 90?
- How would you decide between iterate and stop if the data is mixed?
- How would you handle a security or accreditation delay within this 90-day window?
More metrics questions
- What metrics prove Harvey's value to a firm like PwC or Paul Weiss?Harvey · Metrics · Hard
- What KPI hierarchy would you use for Vault, from account-level adoption and active matters to search success, document coverage, and workflow outcomes, to measure value for law firms and enterprises? How would those metrics change your roadmap if usage is high but repeat usage in critical workflows is low?Harvey · Metrics · Hard
- What metrics would you define for Command Center across adoption, admin efficiency, and governance coverage, and how would you use those metrics to decide whether the product is actually improving Harvey’s enterprise land-and-expand motion?Harvey · Metrics · Hard
- Harvey is considering expanding its platform to additional regions. How would you decide whether multi-region expansion is the right product investment now versus later? Explain the customer signals, business considerations, and success metrics you would use, and how you would compare this investment against competing platform priorities.Harvey · Metrics · Hard
- For Vault’s search and Q&A experience, what metrics would you define for adoption, engagement, trust, and business value at the workspace and firm level? Which leading indicators would you watch in the first 90 days, and how would changes in those metrics alter your product decisions?Harvey · Metrics · Hard
- A pilot agent deployment has strong qualitative feedback from senior stakeholders, but end-user adoption is inconsistent and the customer’s data environment is messy. How would you structure the deployment plan, define the success metrics, diagnose whether the issue is workflow fit vs data readiness vs change management, and decide if this should graduate into a repeatable product for the broader vertical?Harvey · Metrics · Hard
More questions from Harvey
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop