Metrics question

How would you measure the success of the Netflix recommendation engine?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Metric design for a machine learning system, balancing engagement outcomes against catalog diversity and discovery health.

State the goal, help users find content they will enjoy quickly, increasing watch time and reducing decision fatigue.
Define the primary metric, percent of viewing sessions initiated from a recommended title, combined with completion rate for those titles.
Define a quality guardrail, recommendation diversity, since an engine that only recommends the same popular titles repeatedly would look good on completion rate but fail at real discovery value.
Define a satisfaction proxy, rating or thumbs up rate on recommended content, catching cases where users finish content but did not actually enjoy it.
Consider long term impact, whether users who engage more with recommendations show higher 90 day retention than those who do not.

Names a specific primary metric, recommended title completion rate, distinct from overall watch time, better isolating the engine's specific contribution.
Adds a diversity guardrail explicitly, since an engine could maximize short term completion by only ever recommending obvious blockbusters.
Ties the metric to retention, testing whether good recommendations causally improve 90 day retention, not just immediate viewing behavior.

Using only overall watch time as the metric, which does not isolate the recommendation engine's specific effect.
No diversity or catalog health guardrail, risking an engine that only ever suggests the most popular titles.
No connection made between recommendation quality and longer term retention.

How would you isolate the recommendation engine's effect from a user's own active searching?
How would you handle the trade off between short term completion rate and long term catalog diversity?