Netflix · Metrics · Senior · 45 to 60 minutes

How should Netflix measure recommendation value?

Netflix is a subscription streaming service that helps members discover and watch movies and TV shows from a large catalog. Answer this product management case and get it graded line by line against a weighted rubric.

Metric design
Causal reasoning
Experimentation
Strategic thinking
Stakeholder judgment

The situation

You are a senior PM in Netflix personalization. Leadership is deciding how much to invest in the next generation of recommender systems, but a basic measurement question is unresolved: how much value do personalized recommendations create beyond the inherent appeal of the titles themselves? Today, teams often point to the share of viewing that comes from recommended titles, but several partners argue that this is not a causal metric because the system is designed to surface titles members were already likely to watch. A clean answer would be to remove personalization for a large group of members and compare outcomes, but that would likely harm the member experience and create pushback from product leadership. Meanwhile, content and finance partners want to know not only whether recommendations increase total viewing, but also whether they change which titles get watched across the catalog, including possible concentration on already popular content. Your director asks you to propose a measurement approach that is credible, practical, and decision useful for product investment.

What to produce

Write a 1 to 2 page product strategy memo.

Your answer must

Define the core causal question and explain why common proxy metrics can mislead in this case.
Recommend a metric framework with primary success metrics, guardrails, and at least one way to capture catalog concentration or diversity.
Propose a measurement approach that can estimate incremental value without broadly removing personalization, and state the key assumptions behind it.
Describe how you would validate that the approach captures substitution effects, including what members watch instead or whether they stop watching.
Suggest 2 to 4 comparison baselines or counterfactuals that would help leadership judge the value of the current recommender system.
Explain how the output of this work would inform product and investment decisions, including key risks and next steps.

Constraints

You cannot run a broad test that turns off personalization for a large share of members.
Any approach must scale to millions of members, a large catalog, and changing member tastes over time.
You have rich recommendation and viewing logs, but only limited capacity for targeted experiments.
The recommendation team needs an initial answer in time for the next planning cycle.

Format

Use sections for problem framing, metrics, methodology, validation, baselines, risks, and decision use.

How Netflix actually played it

Netflix argues that valuing recommendations by the share of viewing from recommended titles is misleading, since titles are recommended because they are likely to be watched. The team builds a model to separate title appeal from recommendation value and finds, for example, that random recommendations cut engagement 16 percent versus the current system. A rigorous view of measuring personalization.

Share of viewing from recommendations is not a causal metric.
Separate the appeal of the item from the value of showing it.
Compare against random and popularity baselines.

Built from Measuring the Impact of Personalized Recommendations by Netflix Technology Blog.