AI & Technical question

You own AI-powered contract analysis that extracts pricing terms, renewal dates, and risk flags from vendor agreements. Design the pre-launch evaluation framework: what labeled datasets and test slices would you use, how would you score extraction accuracy and explanation quality, what failure modes must trigger fallback or human review, and what launch gate would you set for enterprise readiness?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Whether you can design a pre-launch evaluation framework for a document-extraction AI feature with real financial and legal consequences for errors.

Build labeled test slices by contract type (SaaS, services, leases), extraction field (price, renewal date, auto-renew clause, liability terms), and document quality (clean PDF versus scanned or poorly formatted).
Score extraction accuracy per field, not as one blended number, since a missed renewal date is worse than a mis-parsed vendor name.
Score explanation quality separately: does the tool cite the exact clause and page it extracted from, so a reviewer can verify quickly.
Define failure modes that must trigger fallback: low-confidence extraction, conflicting terms across amendments, and any risk-flag field below a confidence threshold.
Route anything under threshold to mandatory human review before it reaches a customer-facing risk summary.
Set the launch gate: minimum field-level accuracy (for example 95% on renewal date and price, since those drive financial decisions) plus a required human-review rate ceiling that engineering can support.

Separates field-level accuracy targets by consequence, treating renewal date and auto-renew terms as higher bar than descriptive fields.
Requires provenance (clause citation) as part of the launch bar, not just accuracy, since enterprise legal and finance reviewers will not trust an unexplained number.
Names a concrete gate, for example assume 95% precision on the highest-risk fields and under 5% of documents routed to full manual review before GA.
Plans for test-set drift: scanned and non-English contracts are a distinct slice that must pass separately, not be averaged into the overall score.

Reporting one blended accuracy number instead of breaking it down by field and risk level.
No provenance or explainability requirement, which enterprise buyers will reject.
Setting a launch gate without defining what happens to documents that fail it.

How would you build the labeled dataset for contract types you have little historical data on?
What would you do if accuracy is high on clean PDFs but drops sharply on scanned documents?
How would you decide when a customer can turn off human review for a given field?