Metrics question
There is a large non-profit church frequented by many visitors. Donations are currently done by visitors who donate money to collection box. You need to launch an existing paypal like collection product for church donation instead of the current mode of donation box. Design an A/B experiment to test results and complete the launch within three months.
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests experiment design, framing a hypothesis, choosing metrics, and sequencing a launch with a hard deadline.
How to approach it
- State the hypothesis: a digital donation option increases total donation amount and donor participation versus a cash-only collection box.
- Design the test by splitting comparable services or locations, alternating weeks or splitting by physical area, since randomizing individuals isn't practical in a shared setting.
- Choose metrics: primary, total donation amount per attendee; secondary, percent of attendees who donate at all; guardrail, donor complaints or friction reports.
- Sequence to fit 3 months: month 1 build and pilot with one congregation, month 2 run the test with enough Sundays for statistical power, month 3 analyze and roll out.
- Define the minimum lift in donation amount needed to justify full rollout before running the test, not after.
What a strong answer includes
- Proposes a randomization design that fits the physical and social nature of a collection-box setting, splitting by service or location.
- Defines the success threshold, minimum lift, before running the test, avoiding after-the-fact rationalization.
- Sequences the plan explicitly against the 3-month deadline instead of leaving timing vague.
Common mistakes
- Proposing to randomly assign individuals, which isn't practical in a shared collection-box setting.
- Not pre-defining a success threshold before seeing the results.
Likely follow-up questions
- How would you ensure statistical significance given a limited number of Sundays in 3 months?
- How would you handle older congregants less comfortable with digital payments?
More metrics questions
- How would you measure the performance and health of a Netflix Original?Microsoft · Metrics · Hard
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Facebook Ads dropped by 20%. Mark Zuckerberg calls you and asks you to fix it. What do you do?Meta · Metrics · Hard
- Imagine you're the product manager for Facebook Marketplace. Since many sellers don't mark items as sold, what existing functionality and metrics could you use to determine whether an item has likely sold?Meta · Metrics · Hard
- Uber has seen a dip in the ride confirmation rate. As a PM at Uber, you are responsible to investigate and fix the issue. How will you go about it?Uber · Metrics · Hard
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop