Metrics question
Instagram team is planning to start a display campaign on subways in NYC. How would you design the test and measure the campaign success?
- Meta
- Metrics
- Medium
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Experimentation design skill, checking whether you can structure a valid test for an offline-to-online marketing channel with a clear success metric and control group.
How to approach it
- Clarify the goal: measure whether the subway campaign drives incremental Instagram app opens, signups, or a specific in-app action among NYC subway riders.
- Define the test structure: run the campaign in select subway stations or lines while holding out comparable stations as a geographic control group.
- Choose the measurement method: use unique tracking, like QR codes with campaign-specific UTM parameters or geo-based app open lift analysis for the test versus control areas.
- Address confounders: control for seasonality and other concurrent marketing by comparing the test period against both the control group and the same period last year.
- Define the primary metric: incremental app opens or signups attributable to the campaign, and a guardrail metric like cost per incremental user against the media spend.
- Set a decision threshold in advance: define what lift would justify scaling the campaign to other cities before analyzing results, to avoid post-hoc rationalization.
What a strong answer includes
- Uses a geographic control group, a valid design for out-of-home advertising where individual-level randomization isn't possible.
- Names concrete measurement mechanisms, QR tracking and geo-based lift analysis, rather than vague 'track engagement' language.
- Sets a decision threshold before running the test, avoiding the common mistake of rationalizing whatever result comes back.
Common mistakes
- Proposing individual-level A/B testing, which isn't feasible for an out-of-home campaign with no way to randomize exposure at the person level.
- Failing to control for seasonality or concurrent campaigns, which would confound the measured lift.
Likely follow-up questions
- How would you isolate this campaign's effect from other concurrent marketing running in NYC?
- What would make you recommend scaling this to other cities?
More metrics questions
- How would you measure the success of Facebook Likes?Meta · Metrics · Medium
- How would you measure improvements made to Facebook messenger?Meta · Metrics · Easy
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Define the metrics for YouTube search.Google · Metrics · Medium
- You are the PM of Instagram app. The MAU has been constant but DAU has declined. What will you do?Meta · Metrics · Medium
More questions from Meta
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop