Netflix · Metrics · Senior · 45 to 60 minutes

What should Netflix optimize in experiments?

Netflix is a subscription entertainment service that helps people watch films and TV shows online, and it originally mailed DVDs. Answer this product management case and get it graded line by line against a weighted rubric.

metric design
experimentation strategy
product judgment
systems thinking
executive communication

The situation

It is 2012. Netflix has moved from DVD by mail to streaming and now serves members across many devices and countries. Recommendations and search increasingly shape what people watch, but content licensing costs are rising and leadership is debating how to evaluate product experiments. Some teams want to optimize for immediate clicks on titles, artwork, or play starts because those signals arrive fast. Others argue that local wins can backfire if members abandon a show quickly, experience more buffering, or become less likely to stay subscribed over time. The challenge is harder because one account often represents a household with different tastes, and experiments now run concurrently across surfaces and markets. Reed Hastings and the product leadership team want a clear metrics framework for experimentation before they scale more work in discovery, personalization, and catalog presentation. You are the senior PM asked to recommend what Netflix should optimize for, how to measure short term wins without losing sight of long term member value, and what guardrails every experiment should respect.

What to produce

A 1 to 2 page experiment measurement memo for the executive product review.

Your answer must

Propose one primary north star metric for discovery and personalization experiments, and define it clearly.
Show a metric hierarchy with leading indicators, lagging outcomes, and why each belongs in the framework.
Define 3 to 5 guardrail metrics that protect streaming quality, business health, and member experience.
Explain how short term experimental readouts would be validated against long term outcomes such as retention, satisfaction, or durable engagement.
Describe how you would segment or interpret results for households, devices, and markets, and include rollout decision rules.

Constraints

Retention and revenue effects are slow to observe, so most experiments need useful readouts within weeks.
One Netflix account may represent multiple viewers in a household with different tastes.
Experiments run across web, mobile, TVs, and game consoles, where instrumentation quality and user behavior differ.
Changes that improve a local metric but increase buffering or content spend per useful viewing session may be unacceptable.

Format

Use headings for north star, proxy metrics, guardrails, segmentation, and decision rules.

How Netflix actually played it

Seven lessons from Netflix experimentation covering personalization tests, business-model experiments, building a platform and choosing the right north star for experiments.

Experiment on business models, not just UI
Choose experiment metrics tied to long-term value
Invest in an experimentation platform

Built from Netflix: Lessons in Experimentation by Aakash Gupta (Product Growth).