Metrics question
Perplexity Computer needs to get new users to a first successful outcome faster. Define a concrete 'first successful outcome' for a new user, including the event definition, denominator, and cohort window. Then propose one onboarding experiment across signup, intent routing, or first-session guidance, and explain how you would read the result without mistaking a short-term lift for a long-term retention loss.
- Perplexity
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can define a rigorous first-successful-outcome metric with real precision, and design an onboarding test whose result you can trust without mistaking a short-term activation lift for eroded long-term retention.
How to approach it
- Define the event precisely: a completed agentic task where the user takes the output as final, for example accepting or acting on the result without needing to redo it manually, not just opening the Computer interface.
- Define the denominator: new users who start any Computer session in their first week, not all signups, since some signups never attempt the feature at all and would dilute the metric.
- Define the cohort window: measured within the user's first 7 days, since a slower but eventually successful first outcome still counts as a different, weaker signal than a fast one.
- Propose one onboarding experiment: intent routing, asking a lightweight clarifying question at session start to route the user's request to the right workflow faster, reducing early failed attempts.
- Read the result cautiously: check week 4-8 retention for the experiment group against control, not just first-outcome speed, since a faster first success achieved by oversimplifying intent routing could produce shallow early wins that do not hold.
What a strong answer includes
- Defines the event, denominator, and window with real precision instead of a vague first success concept, exactly as the question requires.
- Picks a denominator that excludes users who never attempted the feature, avoiding a diluted and misleading metric.
- Explicitly checks longer-window retention before declaring the onboarding experiment a win, directly addressing the short-term-lift risk.
Common mistakes
- Defines first success loosely, like opening the feature, which does not reflect real value delivered.
- Declares the onboarding experiment a win based on faster first-outcome speed alone, without checking downstream retention.
Likely follow-up questions
- How would you handle a user whose first attempt fails but second attempt succeeds within the same session.
- What retention threshold would make you roll back the onboarding change despite a faster first outcome.
More metrics questions
- What metrics would tell you whether Perplexity is winning against Google Search?Perplexity · Metrics · Hard
- Queries per user dropped 20% after a UI change. How do you investigate?Perplexity · Metrics · Medium
- You have one team and can materially advance only one of these areas in the next 6-12 months: Skills, collaboration, artifact generation, or Projects. How would you choose where to invest, and what leading indicators and longer-term outcome metrics would you use to defend that roadmap to leadership?Perplexity · Metrics · Hard
- Users can create Skills in Computer, but many run a Skill once and never use it again. How would you determine whether the main issue is setup friction, weak discoverability, inconsistent results, narrow applicability, or lack of trust, and what product changes would you test first to improve repeat use and sharing?Perplexity · Metrics · Hard
- Suppose Perplexity launches a new 'resume previous work' feature for research and productivity workflows. What metrics would you use to determine whether it improves retention, and how would you separate durable user value from short-term engagement spikes caused by novelty or accidental usage?Perplexity · Metrics · Hard
- Pick one Perplexity use case, research, investing, or shopping, and design a first prototype in Computer for a domain expert. What exact job-to-be-done would it solve, what would the user see and control step by step, and which early metrics would tell you the prototype is genuinely improving productivity?Perplexity · Metrics · Medium
More questions from Perplexity
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop