Shopify · Metrics · Mid · 45 to 60 minutes

How should Shopify measure merchant retention?

Shopify is a commerce platform that lets merchants run online stores and manage orders, payments, and customer data. Answer this product management case and get it graded line by line against a weighted rubric.

Metric design
Analytical judgment
Product strategy
Communication

The situation

You are a PM on Shopify's merchant reporting team. Merchant-facing teams want a new retention report that helps merchants decide which customers to try to win back and which customers are still healthy. The challenge is that many Shopify merchants are not subscription businesses, so there is no explicit cancellation event when an end customer leaves, they simply stop buying. Today, some stakeholders want a simple rule like no purchase in 30, 60, or 90 days because it is easy to explain and fast to ship. Others argue that this will misclassify customers with very different buying rhythms, like weekly grocery shoppers versus people who buy only a few times a year. The solution also has to work across hundreds of thousands of shops with very different patterns, and it must be simple enough for non-technical merchants to trust and use. Your director wants a recommendation for how Shopify should define and report retention, and how to validate that the output is accurate and actionable enough to launch.

What to produce

Write a 1 to 2 page product strategy memo.

Your answer must

Define the core measurement problem for retention and churn in non-contractual commerce, and explain why a fixed inactivity window can fail.
Recommend a merchant-facing retention approach and the specific outputs Shopify should show, such as probabilities, segments, rankings, or expected future value, and explain why those outputs are useful.
Specify the minimum data inputs and decision logic needed for a first version, including how the method should work across shops with different purchase patterns.
Describe how merchants would use the output in practice, including which customer groups deserve attention first and what actions the product should support.
Propose a validation plan and launch success metrics that cover model quality, merchant understanding, product adoption, and business impact.

Constraints

Must rely primarily on historical transaction data Shopify already has for each shop.
Results must be explainable to non-technical merchants inside reports and support conversations.
Assume the first version runs as a recurring batch report, not a real-time system.
The approach must scale across Shopify's full merchant base without manual tuning for each shop.

Format

Use short section headers. Include one table for merchant-facing outputs and one table for success metrics.

How Shopify actually played it

Shopify set out to help merchants see churn in non-contractual businesses where customers just stop buying. The team infers churn probabilities and expected values across over 500 million historical merchant customers and points to customers near 50 percent probability as the group to target. A clear look at building a retention product for customers' customers.

In non-subscription businesses churn has to be inferred, not observed.
Probability bands let teams choose who to win back.
Build analytics your customers can act on, not only report on.

Built from How Shopify Merchants can Measure Retention by Shopify Engineering.