Metrics question
I ran an A/B test on Instagram. It resulted in a 10% increase in usage for Instagram stories and a 10% decrease in overall usage for Instagram. As a product manager, what metrics will you ask from me to make trade-offs on the A/B test?
- Meta
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Understanding how to weigh a feature's cannibalization of the core product, and what data resolves the trade off.
How to approach it
- Clarify what overall usage means, total time spent or total sessions, since interpretation depends on it.
- Ask for the absolute size of each metric, since a 10 percent lift on a small surface can be outweighed by a drop on a much bigger one.
- Ask for retention data, whether Stories usage correlates with higher long term retention even if short term feed usage drops.
- Ask whether the drop is cannibalization, users shifting existing usage from feed to Stories, or a true net loss.
- Ask for monetization impact, since Stories and feed may carry different ad load and revenue per minute.
What a strong answer includes
- Requests the absolute usage numbers behind both percentages, since Stories may be a small fraction of total usage, making the net effect additive rather than a wash.
- Requests a cohort retention comparison, checking whether users exposed to more Stories show higher 30 day retention despite lower feed time.
- Requests revenue per minute by surface, since a shift to Stories could still be revenue positive if monetization there is comparable.
Common mistakes
- Treating the two 10 percent figures as directly comparable without checking absolute base sizes.
- Ignoring retention and only looking at immediate usage numbers.
- Not asking about monetization differences between the two surfaces.
Likely follow-up questions
- If retention improved but revenue dropped, would you still ship the change?
- How would you design a longer term holdout to settle this?
More metrics questions
- How would you measure the success of Facebook Likes?Meta · Metrics · Medium
- How would you measure improvements made to Facebook messenger?Meta · Metrics · Easy
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Define the metrics for YouTube search.Google · Metrics · Medium
- You are the PM of Instagram app. The MAU has been constant but DAU has declined. What will you do?Meta · Metrics · Medium
More questions from Meta
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop