AI & Technical question

How would you measure the accuracy of Harvey's legal research outputs?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests AI evaluation design specifically for legal research accuracy, where correctness has a precise, checkable standard.

Define accuracy precisely for this domain: did the research correctly identify the controlling law, and does every citation actually support the stated proposition.
Build a gold standard test set: real research questions with attorney verified correct answers and citations, ideally across multiple practice areas.
Score on multiple dimensions: citation existence, citation relevance to the claim, and completeness, since research can be technically correct but miss a key case.
Bring in domain experts for evaluation, since only practicing attorneys can reliably judge whether a legal proposition is actually correctly supported.
Track accuracy over time and by practice area, since performance on tax law may differ meaningfully from performance on employment law.
Confirm with the interviewer whether the evaluation is for internal quality tracking or for a claim made externally to law firms, since the rigor bar differs.

Separates citation existence from citation relevance, since a real citation attached to the wrong proposition is a distinct and dangerous failure mode from a fabricated one.
Uses attorney graders rather than automated scoring alone, since legal correctness often requires domain judgment a generic evaluator cannot make.
Segments accuracy by practice area, avoiding a single blended score that could hide weak performance in a specific domain.
Proposes tracking completeness, not just precision, since missing a controlling case is as serious an error as citing a wrong one.

Using only citation existence as the accuracy measure, missing the more common and more dangerous failure of a real citation misapplied.
Relying solely on automated scoring without domain expert review for a field where correctness is nuanced.
Reporting one blended accuracy number across all practice areas, hiding weak spots.

How would you build the gold standard test set without enormous attorney time cost?
How often would you refresh the evaluation set as case law changes?
What accuracy threshold would you require before trusting an answer in a client facing draft?