Metrics question

For a retrieval or ranking improvement, how would you determine whether an offline metric is actually predictive of user value rather than just easy to optimize, and how would you validate that relationship through online experiments?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Rigor in connecting an offline proxy metric to real user value and designing the online validation to prove or disprove it.

Name the specific offline metric, for example a relevance score on a labeled dataset, and the user value it is meant to proxy, such as task success or satisfaction.
Check the labeling process for bias, since offline labels can reward things easy to game, like keyword overlap, rather than true relevance.
Run a held out online experiment where the ranking change is shipped to a subset of users, measuring downstream behavioral metrics like click through, task completion, or reformulation rate.
Compare the offline metric's movement against the online outcome across multiple experiments over time, not just once, to see if they consistently correlate.
Watch for cases where offline improves but online user value does not, which signals the offline metric is being gamed or missing what users actually want.
Decide whether to keep, recalibrate, or replace the offline metric based on that correlation, and document the finding for future launch decisions.

Names a concrete behavioral proxy for online validation, like task success or reformulation rate, not just click through alone.
Explicitly tests for gaming, cases where the offline metric rises but real user value does not.
Requires multiple experiments over time to establish correlation, not a single data point.
States what action follows from a mismatch, recalibrating or replacing the offline metric.

Trusting the offline metric permanently without ever validating it online.
Using only one online metric like click through, which can also be gamed by superficial changes.

What would you do if the offline metric and online results disagreed for the first time after years of correlation?
How would you choose the right online experiment duration to avoid a false signal?