Metrics question
You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests handling conflicting A/B test signals: a primary metric improving while a guardrail metric (retention) worsens, requiring a structured decision, not a gut call.
How to approach it
- Restate the conflict clearly: profile completion up 8%, but 7-day retention down 2%, so the question is whether the short-term data gain is worth the retention cost.
- Check statistical validity first: confirm both results are statistically significant and not noise, and check the sample size and test duration were adequate.
- Form a hypothesis for the link: the new flow likely adds friction or feels invasive (asking for more personal info upfront), causing some users to drop off or feel less comfortable before they've found value in the product.
- Segment the retention drop: check whether it's concentrated in specific user segments (e.g., users asked for sensitive info like phone number) versus a broad decline, to localize the cause.
- Decide based on business priority: retention is generally a stronger long-term signal than profile completion, which is only a means to an end (better personalization/matching), so lean toward not shipping as-is.
- Propose next step: iterate the flow to keep profile-completion gains while removing the specific friction point (e.g., make the added fields optional or move them later in onboarding), then re-test.
What a strong answer includes
- Explicitly weighs the two metrics against each other and states retention as the higher-priority signal since it's closer to core business value, rather than picking one without justification.
- Proposes a concrete hypothesis (added friction/invasiveness) for why more profile info could hurt retention, not just noting the conflict.
- Recommends segmenting the retention drop rather than treating it as monolithic, showing analytical depth.
- Ends with an actionable next step (iterate and re-test) instead of a binary ship/don't-ship answer.
Common mistakes
- Picking one metric to chase (usually profile completion, since it 'improved') without weighing the retention cost.
- Not checking statistical significance or test validity before drawing conclusions.
- Treating this as a binary ship-or-kill decision with no iteration path.
Likely follow-up questions
- How would you determine if the retention drop is worth the profile data gain long term?
- What test would you run next to isolate the cause?
- How would you explain this trade-off to a stakeholder who only looks at the completion metric?
More metrics questions
- How do you define success for Yelp reviews?Google · Metrics · Medium
- Utilization went down by 45% on app XYZ in Italy for the month of August. Give a reason why and draft a plan to fix it.Spotify · Metrics · Medium
- Define the metrics for YouTube search.Google · Metrics · Medium
- You walk in to your office and find that Google cloud subscription has gone down by 20%. What do you do as a product manager?Google · Metrics · Medium
- What would be the top six metrics for WhatsApp? Now pick one from the six. Tell me three things you would do to improve that metric.Google · Metrics · Medium
- How do you measure the success of Google Search?Google · Metrics · Medium
More questions from these companies
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop