Metrics question
Replit launches a new natural-language app builder for both existing users and new prospects. How would you decide which growth motions should come from product-led loops vs paid acquisition, which segments or geos you’d target first, and what first 3-5 experiments you’d run to test whether paid is incrementally driving durable activation rather than just more signups?
- Replit
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests designing a growth measurement plan that distinguishes durable product-led growth from acquisition-driven vanity signups.
How to approach it
- Clarify the launch goal first: is this optimizing for new-user activation, existing-user expansion, or both, since it changes which motion should lead.
- For product-led loops, target segments where the builder already has a distribution advantage, existing users referring or sharing generated apps.
- For paid acquisition, target segments with clear intent signals, like developers searching for app-builder tools, rather than broad awareness spend.
- Design experiments that isolate incrementality: a holdout region or geo without paid spend, compared against a paid-spend region on activation and retention, not just signups.
- Run a second experiment testing whether paid-acquired users convert to habitual use at the same rate as organic users within 30 days.
- Run a third experiment on referral-loop virality, k-factor from existing users, to see if product-led growth alone can hit the target without paid spend.
What a strong answer includes
- Insists on incrementality testing, like a geo holdout, rather than counting signups as proof paid spend is working.
- Distinguishes activation and 30-day retention from raw signups as the real success measure for paid-acquired users.
- Names a concrete first experiment, a holdout region, that would actually falsify or confirm paid's incremental value.
Common mistakes
- Treating signup volume as success without checking whether paid users actually activate or retain.
- Running paid and organic growth simultaneously everywhere with no way to isolate incrementality.
Likely follow-up questions
- How long would you run the holdout experiment before making a budget call?
- What would you do if paid drives signups but organic drives better retention?
More metrics questions
- What metrics would you track for Replit Agent's success?Replit · Metrics · Medium
- How would you measure whether AI is actually helping users ship apps faster on Replit?Replit · Metrics · Medium
- How would you design a product-led land-and-expand motion for Replit enterprise so that a small group of internal builders grows into a company-wide rollout? What product milestones, leading indicators, and account-level metrics would you use to decide whether an account is ready to expand?Replit · Metrics · Hard
- A week after launch, CAC in one region spikes 35% while blended global CAC is flat. Walk me through how you’d detect it early, decompose the change by funnel stage and channel, separate mix-shift vs signal-quality vs creative-fatigue causes, and decide whether to cut spend, change targeting, refresh creative, or fix measurement.Replit · Metrics · Hard
- You are the single DRI for paid search, paid social, ASO, and lifecycle. Design the operating loop you would build in your first 90 days: what inputs you’d ingest, how you’d segment users and geos, how creative insights feed into new variants, how budget allocation decisions get made, and which parts must be automated vs human-reviewed. Be explicit about the metrics and decision rules that make the system improve over time.Replit · Metrics · Hard
- How would you measure the performance and health of a Netflix Original?Microsoft · Metrics · Hard
More questions from Replit
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop