AI & Technical question

How would you measure the quality of AI-generated research from Claygent?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests AI evaluation design for judging the quality of open ended, agentic research output rather than a simple factual answer.

Define quality dimensions specifically for research output: factual accuracy, completeness relative to what was asked, and correct sourcing of the information provided.
Build a test set of real research tasks with known correct or verifiable answers, covering the range of tasks users actually ask Claygent to perform.
Score using a mix of automated checks, like verifying a returned fact against a trusted source, and human review for judgment heavy dimensions like completeness.
Track failure modes separately: distinguish a wrong answer from a right answer with no source, since both are quality problems but need different fixes.
Add a real world proxy: track how often users manually correct or discard Claygent's output, as a signal that complements the structured evaluation.
Confirm with the interviewer whether the evaluation should focus on one data type, like firmographic research, or general open ended research quality across many task types.

Separates accuracy, completeness, and sourcing as distinct quality dimensions rather than a single blended quality score that hides which aspect is actually weak.
Builds a real test set with verifiable answers rather than relying purely on subjective human impressions of quality.
Distinguishes a wrong answer from an unsourced answer as different failure modes, since each points to a different underlying fix.
Adds a real world usage proxy, like manual correction rate, to complement structured evaluation with a signal drawn from actual production usage.

Using a single blended quality score that does not reveal whether the weakness is accuracy, completeness, or sourcing.
Relying only on human judgment without any verifiable test set, making evaluation inconsistent and hard to track over time.
No real world usage signal, evaluating quality purely offline without checking whether it matches what users experience in production.

How would you build a verifiable test set for open ended research tasks at scale?
What would you do if accuracy was high but completeness was consistently weak?
How would you weight sourcing quality against raw factual correctness in an overall score?