AI & Technical question
How would you design an experiment to evaluate a generative AI feature when outputs are non-deterministic?
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests experimentation design specific to generative AI: can you adapt standard A/B testing rigor to handle non-deterministic, open-ended outputs where a single 'correct' response does not exist.
How to approach it
- Clarify the challenge: a generative feature can produce different valid outputs for the same input, so standard click-through-only testing is insufficient.
- Define the outcome metrics: a mix of automated quality proxies, relevance and groundedness scoring, and human or behavior-based signals like thumbs up/down and follow-up rate.
- Design the test structure: a randomized A/B test at the session level, not per-response since responses vary too much, aggregated across many interactions.
- Increase sample size and duration versus a typical test: higher per-response variance needs a larger sample to reach significance than a deterministic change would.
- Add human evaluation on a sampled subset in parallel, since automated proxies alone can miss subtly wrong but confident-sounding answers.
- Define success as significant improvement in aggregate quality and behavioral metrics, combined with no regression in a hallucination guardrail.
What a strong answer includes
- Explicitly names the core challenge, non-deterministic outputs make single-response comparison meaningless, so evaluation happens at the aggregate level.
- Combines automated quality proxies with human evaluation and behavioral signals, since no single measurement type suffices for generative quality.
- Addresses statistical power directly, noting higher output variance needs a larger sample to detect a real effect.
- Adds an explicit hallucination guardrail to the experiment design, not just an engagement-lift metric.
Common mistakes
- Proposing a standard A/B test with no acknowledgment of the added statistical noise from non-deterministic outputs.
- Relying only on automated metrics or only on human evaluation, rather than combining both for a fuller quality picture.
- No guardrail metric for safety or hallucination risk, treating this like any other feature experiment with only an engagement metric.
Likely follow-up questions
- How would you determine the right sample size given the added output variance?
- How would you catch a hallucination that a user does not notice or report?
- How would you decide whether a quality regression on a small user subset is enough to halt the test?
More ai & technical questions
- You’re given a new model that improves accuracy by 20% but doubles latency. Would you ship it? Walk me through your decision.OpenAI · AI & Technical · Hard
- In what situations would you explicitly avoid using RAG and choose prompting or fine-tuning instead?OpenAI · AI & Technical · Hard
- How should OpenAI handle hallucinations in ChatGPT for high-stakes use cases like medical or legal questions?OpenAI · AI & Technical · Hard
- How would you design guardrails for OpenAI's Operator (browser agent) to prevent harmful actions?OpenAI · AI & Technical · Hard
- Before launching a new Codex capability that can write code or trigger deployments, what evaluation plan and launch gates would you require to validate permission boundaries, prompt-injection resistance, stale authorization handling, secret protection, partner-dependency failure modes, and audit completeness?OpenAI · AI & Technical · Hard
- Suppose you are scoping a first product for in-house legal teams to review contracts with AI assistance. What requirements would you lock first around target use case, acceptable error rates, human-review steps, citations/provenance, and data handling, and what would have to be true before you let customers use it on real matters?OpenAI · AI & Technical · Hard
More questions from these companies
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Foundations: the model and the decisions it forces on you
- Chapter 8: Evals: define good and make the number defensible
- Chapter 6: Agents and agentic architecture