Stripe · Metrics · Senior · 45 to 60 minutes

How should Stripe measure payment method impact?

Stripe provides payments infrastructure and checkout tools that businesses use to accept money online. Answer this product management case and get it graded line by line against a weighted rubric.

Experiment design
Metrics judgment
Platform strategy
Trade off analysis
Executive communication

The situation

You are a senior PM in Stripe’s checkout organization. Stripe can surface many payment methods beyond cards, but merchants keep asking a basic question: which methods actually increase conversion and revenue, and for which buyers? Leadership wants a credible answer that can inform product defaults, merchant guidance, and future investment in payment method coverage. The challenge is that Stripe operates across many buyer countries and supports many payment methods, so the number of possible method, country, and device combinations is huge. A simple global A/B test may miss important differences, while a fully custom test for every segment is not practical. There is also a product experience risk. If a buyer refreshes checkout or returns later and sees a different set of payment methods, that could hurt trust and contaminate the data. Some checkout surfaces have stable session identifiers, others do not. Your GM asks you to propose the experiment and measurement strategy before any test launches. Assume Stripe wants insights that are statistically credible, operationally feasible, and safe for merchants and buyers.

What to produce

Write a 1 to 2 page experiment strategy memo.

Your answer must

Define the decision the experiment should inform, and specify the primary metric, secondary metrics, and guardrail metrics.
Propose an experiment design that can estimate the causal impact of adding or withholding payment methods, including the unit of randomization and how you will keep the buyer experience consistent across refreshes and related sessions.
Explain how you would handle the large combination space across payment methods, buyer countries, and device types, and still produce useful segmented insights.
Include a sample sizing and validation approach, with the phases you would run before full launch and what each phase is meant to de risk.
Call out the biggest risks to validity, privacy, and merchant trust, and how you would mitigate them.
State how you would report results to internal stakeholders and to merchants, especially if conversion and revenue move differently across segments.

Constraints

The plan must work across both hosted checkout flows with persistent session IDs and flows where a page reload creates a new session.
Leadership needs results broken out at least by buyer country and device type, not only a single global average.
You cannot run a separate manually managed experiment for every payment method, country, and merchant combination.
The test should minimize disruption to merchants and avoid a visibly inconsistent checkout experience for the same buyer journey.

Format

Use five sections: objective, metrics, experiment design, analysis plan, risks and rollout.

How Stripe actually played it

Stripe tested more than 50 payment methods across 200 buyer countries and found that surfacing at least one relevant extra method lifted revenue 12 percent and conversion 7.4 percent on average. The post explains how the test handled millions of method and country combinations. Good for seeing how a large-scale experiment is designed.

Large combinatorial spaces need a deliberate experiment design.
Relevance matters: local methods and wallets drove the largest lift.
Report both revenue and conversion effects.

Built from How we tested the conversion impact of global payment methods by Stripe.