Duolingo · Metrics · Mid · 45 to 60 minutes

How should Duolingo judge hundreds of experiments?

Duolingo is a language learning app and website used by millions of learners around the world. Answer this product management case and get it graded line by line against a weighted rubric.

Experiment design
Metrics judgment
Prioritization
Analytical communication

The situation

Your manager, a product leader working with Duolingo's data team, asks you to propose how the company should evaluate and operationalize product experiments across the app and website. Duolingo already runs a few hundred A/B tests in a typical week, from small UI changes to larger features such as social competition and subscription prompts. Teams move quickly, but experiment readouts are inconsistent. Some teams focus on local wins, such as taps or upgrade rate, while others worry about learning quality, retention, or bugs. Because so many tests run at once, custom analysis for each experiment is too slow, yet a one size fits all dashboard would bury teams in irrelevant metrics. You need to recommend a metrics and reporting framework for an internal experiments service that helps teams set up tests, monitor gradual rollouts, and decide whether to launch, pause, or iterate. The hard part is balancing speed, rigor, and product values when different metrics can move in opposite directions.

What to produce

Write a 1 to 2 page product memo.

Your answer must

Define a metric hierarchy that includes one primary success metric, universal guardrail metrics, and experiment-specific secondary metrics.
Propose how report templates should differ by experiment type, including how they would support a test with more than two arms.
Recommend rollout monitoring and stop, continue, or pause rules for gradual exposure before full launch.
Explain how the framework should handle conflicting results, for example when a revenue or engagement metric improves while retention or learning metrics worsen.
List the key decisions the experiments service must support from experiment setup through final recommendation.

Constraints

The approach must scale to a few hundred concurrent experiments with mostly automated analysis.
Nightly reports are the main recurring analysis surface, not custom deep dives for every test.
The framework must work across mobile and desktop product changes, including experiments with more than two arms.

Format

Use short sections and include one simple table for metric tiers and decision rules.

How Duolingo actually played it

Duolingo runs hundreds of A/B tests at once under the principle 'test everything', and this post explains how its data team analyzes them. It covers the tooling and process for turning experiment data into decisions. A glimpse into running experimentation at scale.

Test everything but standardize analysis.
Automate common analysis so decisions are fast.
A culture of experimentation needs shared tooling.

Built from Improving Duolingo, one experiment at a time by Lavanya Aprameya.