Metrics question
Perplexity is launching a new agentic Computer feature across web, mobile, and desktop. How would you plan the rollout: which cohorts would you expose first, through which surfaces, and in what sequence? Explain how you would decide whether adoption of the feature is actually causing higher retention and paid usage, not just generating curiosity clicks.
- Perplexity
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can design a responsible staged rollout for a cross-surface agentic feature, and set up a causal test that separates real retention lift from curiosity-driven novelty.
How to approach it
- Expose first to a small cohort of highly engaged existing users on the surface with the most reliable agent execution today, likely web, before expanding to mobile and desktop.
- Sequence by surface based on technical readiness and usage patterns: web first for iteration speed, then desktop for power users doing longer sessions, then mobile last given tighter interaction constraints for agentic actions.
- Run the initial exposure as a randomized holdout within the target cohort, not just a phased rollout to everyone, since only a true control group lets you separate the feature's effect from concurrent product changes or seasonality.
- Define the causal test: compare retention and paid usage between exposed and holdout groups over a multi-week window, not just immediate click or session metrics, since curiosity clicks decay fast while real retention lift persists.
- Only expand to the next surface or cohort once the holdout comparison on the first shows sustained lift beyond an initial spike, avoiding a company-wide rollout based on early enthusiasm alone.
What a strong answer includes
- Uses a randomized holdout, not just a phased rollout, as the mechanism for actually proving causation rather than assuming a metric bump after launch means the feature worked.
- Sequences surfaces by technical readiness and usage pattern with clear reasoning, not an arbitrary order.
- Distinguishes a multi-week retention comparison from immediate curiosity-click metrics, directly addressing the novelty-spike risk in the question.
Common mistakes
- Rolls out to all cohorts and surfaces simultaneously with no holdout, making it impossible to prove causation later.
- Uses short-term engagement metrics alone to declare success, missing that curiosity spikes typically decay within weeks.
Likely follow-up questions
- How long would you wait before expanding from the holdout test to a broader rollout.
- What would you do if mobile shows weaker results than web, given the tighter interaction constraints.
More metrics questions
- What metrics would tell you whether Perplexity is winning against Google Search?Perplexity · Metrics · Hard
- Queries per user dropped 20% after a UI change. How do you investigate?Perplexity · Metrics · Medium
- You have one team and can materially advance only one of these areas in the next 6-12 months: Skills, collaboration, artifact generation, or Projects. How would you choose where to invest, and what leading indicators and longer-term outcome metrics would you use to defend that roadmap to leadership?Perplexity · Metrics · Hard
- Users can create Skills in Computer, but many run a Skill once and never use it again. How would you determine whether the main issue is setup friction, weak discoverability, inconsistent results, narrow applicability, or lack of trust, and what product changes would you test first to improve repeat use and sharing?Perplexity · Metrics · Hard
- Suppose Perplexity launches a new 'resume previous work' feature for research and productivity workflows. What metrics would you use to determine whether it improves retention, and how would you separate durable user value from short-term engagement spikes caused by novelty or accidental usage?Perplexity · Metrics · Hard
- Pick one Perplexity use case, research, investing, or shopping, and design a first prototype in Computer for a domain expert. What exact job-to-be-done would it solve, what would the user see and control step by step, and which early metrics would tell you the prototype is genuinely improving productivity?Perplexity · Metrics · Medium
More questions from Perplexity
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop