Metrics question
A core responsibility is proving that Computer improves real SMB revenue outcomes and then turning those results into credible case studies. How would you measure impact honestly when performance is affected by seasonality, channel mix, and other tools the SMB is already using?
- Perplexity
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can design an honest measurement methodology for a product's revenue impact when many confounding factors exist outside the product's control.
How to approach it
- Define the outcome metric precisely, for example incremental revenue or conversion attributable to actions Computer took, not total SMB revenue which is affected by many other factors.
- Control for seasonality by comparing performance against the SMB's own historical baseline for the same period in prior cycles, rather than an unadjusted before-after comparison.
- Control for channel mix by segmenting results by channel and only claiming credit where Computer's actions directly touched that channel's spend or content.
- Account for other tools already in use by asking SMBs directly what else changed in the measurement window, and by preferring a matched cohort or holdout comparison (SMBs using Computer versus similar SMBs that are not) over a single-customer before-after story.
- Report a range and confidence level rather than a single precise number, being transparent about what the measurement can and cannot isolate.
- Build case studies only from customers where the causal story is clean, for example a clear before-after with minimal confounding, rather than cherry-picking the biggest raw number.
What a strong answer includes
- Proposes a holdout or matched-cohort comparison as the most credible way to isolate Computer's impact, rather than relying on single-customer before-after stories.
- Explicitly controls for seasonality using the SMB's own historical baseline, a concrete and honest method.
- Recommends reporting a range or confidence level instead of a single inflated number, directly addressing the 'measure honestly' framing of the question.
- Distinguishes case-study-worthy results (clean causal story) from typical results (more confounded), so case studies do not overstate the average impact.
Common mistakes
- Proposing a simple before-after comparison without controlling for seasonality or other tools in use.
- Cherry-picking the best-looking customer result for case studies without disclosing confounders.
- No mention of a control or holdout group to isolate causal impact.
Likely follow-up questions
- How would you handle an SMB that refuses to share data about other tools they use?
- What would you do if a holdout group is not feasible for a small customer base?
- How would you communicate honest, uncertain results to a customer who wants a clean success story?
More metrics questions
- What metrics would tell you whether Perplexity is winning against Google Search?Perplexity · Metrics · Hard
- Queries per user dropped 20% after a UI change. How do you investigate?Perplexity · Metrics · Medium
- You have one team and can materially advance only one of these areas in the next 6-12 months: Skills, collaboration, artifact generation, or Projects. How would you choose where to invest, and what leading indicators and longer-term outcome metrics would you use to defend that roadmap to leadership?Perplexity · Metrics · Hard
- Users can create Skills in Computer, but many run a Skill once and never use it again. How would you determine whether the main issue is setup friction, weak discoverability, inconsistent results, narrow applicability, or lack of trust, and what product changes would you test first to improve repeat use and sharing?Perplexity · Metrics · Hard
- Suppose Perplexity launches a new 'resume previous work' feature for research and productivity workflows. What metrics would you use to determine whether it improves retention, and how would you separate durable user value from short-term engagement spikes caused by novelty or accidental usage?Perplexity · Metrics · Hard
- Pick one Perplexity use case, research, investing, or shopping, and design a first prototype in Computer for a domain expert. What exact job-to-be-done would it solve, what would the user see and control step by step, and which early metrics would tell you the prototype is genuinely improving productivity?Perplexity · Metrics · Medium
More questions from Perplexity
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop