Metrics question
A week after launch, CAC in one region spikes 35% while blended global CAC is flat. Walk me through how you’d detect it early, decompose the change by funnel stage and channel, separate mix-shift vs signal-quality vs creative-fatigue causes, and decide whether to cut spend, change targeting, refresh creative, or fix measurement.
- Replit
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests rapid metric decomposition skill under a real regional anomaly, separating mix shift, signal quality, and creative fatigue as distinct causes.
How to approach it
- Confirm the spike is real and not a reporting artifact by checking raw spend and conversion counts, not just the calculated CAC ratio.
- Decompose by funnel stage first, checking whether the spike is in cost per click, click to signup, or signup to activation, since each points to a different cause.
- Check for mix shift by reviewing whether channel or audience allocation in that region changed recently, which can raise blended CAC without anything being broken.
- Check for signal quality issues by reviewing whether conversion events from that region dropped or became delayed, which would make the bidding algorithm overspend chasing bad signal.
- Check for creative fatigue by reviewing frequency and click through rate trends for the region's active ad creative over the past two weeks.
- Match the fix to the diagnosed cause: cut spend if it is a genuine demand or cost problem, fix tracking if it is signal quality, or refresh creative if fatigue is confirmed.
What a strong answer includes
- Decomposes the spike by funnel stage before jumping to a cause, since cost per click and activation rate problems require completely different fixes.
- Explicitly separates three distinct causes, mix shift, signal quality, and creative fatigue, with a specific check for each rather than guessing.
- Matches the corrective action precisely to the confirmed cause instead of defaulting to a spend cut as the safe default response.
Common mistakes
- Reacting immediately by cutting spend before diagnosing whether the cause is even a real cost or demand problem.
- Treating blended global CAC being flat as reassurance, when a regional spike can still represent a real, isolated issue worth investigating.
Likely follow-up questions
- How would you distinguish a signal quality issue from a genuine cost increase using only platform reported data?
- What would you do if all three hypotheses show partial evidence?
More metrics questions
- What metrics would you track for Replit Agent's success?Replit · Metrics · Medium
- How would you measure whether AI is actually helping users ship apps faster on Replit?Replit · Metrics · Medium
- Replit launches a new natural-language app builder for both existing users and new prospects. How would you decide which growth motions should come from product-led loops vs paid acquisition, which segments or geos you’d target first, and what first 3-5 experiments you’d run to test whether paid is incrementally driving durable activation rather than just more signups?Replit · Metrics · Hard
- How would you design a product-led land-and-expand motion for Replit enterprise so that a small group of internal builders grows into a company-wide rollout? What product milestones, leading indicators, and account-level metrics would you use to decide whether an account is ready to expand?Replit · Metrics · Hard
- You are the single DRI for paid search, paid social, ASO, and lifecycle. Design the operating loop you would build in your first 90 days: what inputs you’d ingest, how you’d segment users and geos, how creative insights feed into new variants, how budget allocation decisions get made, and which parts must be automated vs human-reviewed. Be explicit about the metrics and decision rules that make the system improve over time.Replit · Metrics · Hard
- How would you measure the performance and health of a Netflix Original?Microsoft · Metrics · Hard
More questions from Replit
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop