Metrics question
How would you measure improvements made to Facebook messenger?
- Meta
- Metrics
- Easy
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests defining a measurement framework for iterative product changes to a messaging app, distinguishing engagement quality from raw usage.
How to approach it
- Clarify what 'improvement' means: could be a UI change, a new feature (reactions, disappearing messages), or a performance fix; assume a general framework applicable to any of these.
- Define primary engagement metrics: messages sent per DAU, session frequency, and time-to-first-response between users.
- Define a quality-of-experience metric: message delivery latency and app crash/error rate, since performance directly affects usage.
- Define a retention metric: D1/D7 retention of users exposed to the change versus a holdout group.
- Run the improvement as an A/B test against these metrics, not just before/after, to control for seasonality and external factors.
- Add a guardrail: ensure the change doesn't increase unsend/delete rate or reported spam, which would signal a negative experience despite higher raw usage.
What a strong answer includes
- Separates raw usage metrics (messages sent) from quality metrics (latency, crash rate) and retention, giving a fuller picture than one number.
- Insists on A/B testing rather than simple before/after comparison, controlling for confounds.
- Adds a guardrail metric (unsend/delete/report rate) to catch improvements that increase usage but harm experience.
- Gives a concrete example: a typing-indicator improvement should be measured by response latency and session frequency, not just message count.
Common mistakes
- Using only 'messages sent' as the single success metric without any quality or retention check.
- Comparing before/after without a control group, missing seasonality or external confounds.
- No guardrail metric to catch negative side effects of the change.
Likely follow-up questions
- How would you isolate this feature's impact from other things shipping at the same time?
- What would you do if messages sent went up but D7 retention dropped?
- How would you measure success differently for a performance fix versus a new feature?
More metrics questions
- How would you measure the success of Facebook Likes?Meta · Metrics · Medium
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Define the metrics for YouTube search.Google · Metrics · Medium
- You are the PM of Instagram app. The MAU has been constant but DAU has declined. What will you do?Meta · Metrics · Medium
- How do you measure success of the Facebook Fundraising feature?Coinbase · Metrics · Medium
More questions from Meta
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop