AI & Technical question

How would you measure whether Glean's AI answers are accurate and well-cited?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

AI and technical metrics for answer accuracy and citation quality, central to trust in an enterprise AI-answering product.

Define accuracy concretely: does the answer correctly reflect what the underlying source documents say, without fabricated or misattributed claims.
Track citation quality: percentage of answers where every claim is backed by a clickable, correct source citation, not just a general reference.
Add a human evaluation sample: have reviewers periodically score a random set of answers against the source documents for factual correctness.
Add an implicit signal: track how often users click through to verify a citation, and how often they immediately reformulate the query, which can suggest a poor answer.
Add explicit feedback: a simple thumbs up or down per answer, aggregated over time to catch drift in accuracy as sources or models change.
Define success as a minimum accuracy threshold on the human-evaluated sample, tracked continuously, not just at launch.

Separates accuracy (is the claim true) from citation quality (is the source correctly and specifically attributed) as distinct measurable dimensions.
Proposes a human evaluation sample, which is necessary since accuracy cannot be fully automated for open-ended answers.
Uses reformulation rate as a scalable implicit proxy that does not require manual review of every query.
Treats this as an ongoing monitoring problem, not a one-time launch check, since source data and models change.
Ties the metric to trust, the core value proposition for an enterprise answer engine.

Relying only on explicit user feedback, which is sparse and does not scale to catch drift.
Treating citation presence alone as sufficient without verifying the citation actually supports the claim.

How would you build the human evaluation sample without it becoming a bottleneck?
How would you detect accuracy drift after a model update?
What would you do if accuracy dropped for one specific data source?