Metrics question

Suppose OpenAI is testing a new learning feature in ChatGPT, such as guided practice or personalized explanations. How would you design the experiment so you can tell whether it improves learning outcomes rather than just session length or retention, and what success metrics would you use?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests experiment design skill: isolating true learning impact from engagement proxies like session length or return rate.

Define the actual outcome of interest, measured learning gain on a pre and post assessment, rather than proxy engagement metrics.
Design a randomized controlled test between the new feature and the current experience, not just a before and after comparison.
Include a delayed post test, not just immediate assessment, to check whether the learning gain persists rather than just improving short term recall.
Track session length and retention as secondary metrics, explicitly separate from the primary learning outcome metric, to avoid conflating the two.
Control for selection effects, since students who opt into a new feature may already be more motivated learners.
Set a pre registered success threshold for the learning gain metric before running the experiment, to avoid post hoc rationalization of engagement wins.

Uses a pre and post assessment with a delayed retest, which is what actually separates real learning from short term engagement.
Explicitly separates primary outcome, learning gain, from secondary engagement metrics instead of letting engagement substitute for the real question.
Controls for selection bias, since more motivated students opting into a new feature would inflate apparent learning gains.

Using session length or return rate as the primary success metric for a learning claim.
Skipping a delayed retest, which would miss cases where a feature helps short term recall but not durable learning.

How would you handle a case where engagement is high but the delayed retest shows no real gain?
What would you do if a true randomized test is not feasible in a live classroom setting?