Metrics question
As the product manager responsible for Facebook's newsfeed, your task is to increase the number of posts users see per day. To test this, the company conducted an A/B test with a randomized 1% sample of the population. Which metrics would be affected by this change, and what type of data should you anticipate as a result?
- Meta
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Ability to reason about the downstream metric and data implications of a specific product change tested via a randomized experiment.
How to approach it
- Identify the primary metrics likely affected: time spent in feed, total post impressions, and engagement rate (likes, comments, shares) per session.
- Consider secondary/guardrail metrics: content diversity per session, ad impressions and revenue (since more posts could dilute ad density), and user-reported satisfaction or complaint rate.
- Anticipate the type of data: continuous, per-user metrics like session time and engagement rate, alongside count data like total posts seen, all analyzable at the 1% sample level with standard A/B significance testing.
- Flag a likely trade-off: engagement rate per post may decline even if total engagement rises, since seeing more posts can dilute attention per post.
- Note that a 1% sample may be too small to detect subtle drops in metrics with high variance, like long-term retention, requiring a longer observation window or larger sample for those.
What a strong answer includes
- Distinguishes primary engagement metrics from guardrail metrics like ad revenue and complaint rate, showing broader systems thinking.
- Anticipates the specific trade-off, more posts diluting per-post engagement, a nuanced and realistic prediction.
- Addresses statistical power explicitly, noting a 1% sample may not detect effects on slower-moving metrics like retention.
Common mistakes
- Naming only one obvious metric, like total engagement, without considering dilution effects or guardrails like ad revenue.
- Ignoring the statistical/data quality dimension of the question entirely, which explicitly asks about data type and anticipated results.
Likely follow-up questions
- How would you decide if the test should run longer to detect retention effects?
- What would make you recommend not shipping this even if engagement went up?
More metrics questions
- How would you measure the success of Facebook Likes?Meta · Metrics · Medium
- How would you measure improvements made to Facebook messenger?Meta · Metrics · Easy
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Define the metrics for YouTube search.Google · Metrics · Medium
- You are the PM of Instagram app. The MAU has been constant but DAU has declined. What will you do?Meta · Metrics · Medium
More questions from Meta
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop