Netflix · Metrics · Senior · 45 to 60 minutes

How should Netflix test title artwork?

Netflix is a subscription streaming service that helps people discover and watch movies and TV shows online. Answer this product management case and get it graded line by line against a weighted rubric.

Experiment design
Metrics strategy
Tradeoff analysis
Personalization thinking
Rollout planning

The situation

Netflix's discovery and personalization leaders ask you to recommend how the company should choose artwork for each show or movie on the home screen. For many titles, the creative team can supply several viable images, and different members may respond better to different artwork. The problem is that each member only sees one image for a given title impression, so every exposure both serves the product and could teach the team something. Assume a typical title has 4 candidate images and receives enough traffic that a bad testing choice could affect millions of browse impressions before a decision is made. Leaders want a testing approach that can improve click to play behavior without creating a long period where many members see weak artwork. They also want confidence that the method can scale across a large catalog, not just a single marquee title. You need to propose how Netflix should evaluate and choose artwork variants, which metrics matter, and how to handle the tradeoff between learning quickly and showing the best available image in the moment.

What to produce

Write a 1 to 2 page product strategy memo.

Your answer must

Define the core decision, the primary success metric, and at least 2 guardrail metrics.
Compare at least 2 experimentation approaches for this problem and recommend 1, with a clear reason it fits the one image slot constraint.
Explain how learning and serving should work over time, including when and how traffic would shift across artwork options.
Address what should be personalized or segmented, for example by title, member cohort, or context, and what should stay global.
Discuss second order effects and risks, such as member experience, trust in recommendations, or overfitting to short term clicks.
Outline rollout, monitoring, and stopping rules, including how you would know the approach can scale across the catalog.

Constraints

For each title impression, Netflix can show only one artwork image to a member.
The recommendation must work for a large and changing catalog, not only a single title.
Testing should limit member exposure to clearly weak artwork while still producing reliable learning.

Format

Use short sections for decision, metrics, experiment approach, tradeoffs, and rollout.

How Netflix actually played it

Summarizes how Netflix chose artwork for each title when only one image can be shown, and why a bandit approach fit better than classic A/B tests. A concrete case of experiment design under constraints.

Pick the test method that fits the decision
Only one option can be shown, so learn while serving

Built from Netflix Artwork Personalization Through Multi-Armed Bandit Testing by Reforge.