Metrics question
Queries per user dropped 20% after a UI change. How do you investigate?
- Perplexity
- Metrics
- Medium
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Structured metrics investigation isolating a UI-driven behavior change from other confounds.
How to approach it
- Confirm measurement validity: check the analytics implementation did not break coincident with the UI change, since a tracking bug can masquerade as a real drop.
- Segment the drop: break queries-per-user down by user cohort, new versus existing, and by platform, mobile versus desktop, since a UI change may affect these very differently.
- Form hypotheses: the new UI added friction to starting a query, changed default behavior in a way that discourages follow-up questions, or simply confused returning users used to the old layout.
- Check funnel-level detail: compare time-to-first-query after opening the app before and after the change, since increased friction should show up there specifically.
- Validate with qualitative signal: review support tickets, app store reviews, or a quick user survey mentioning the redesign directly.
What a strong answer includes
- Insists on ruling out a measurement or tracking issue first, since a coincident analytics bug is a common and easily overlooked explanation.
- Segments the drop by cohort and platform before concluding the UI change is the sole cause, since effects often concentrate in one group.
- Proposes a specific funnel metric, time-to-first-query, to pinpoint exactly where the new friction is occurring in the flow.
- Combines quantitative segmentation with qualitative signals like reviews or support tickets to build a fuller picture of what changed for users.
Common mistakes
- Assuming the UI change is the cause without checking for a coincident tracking or measurement issue first.
- Treating queries-per-user as one aggregate number instead of segmenting by cohort and platform.
Likely follow-up questions
- Which specific UI element would you suspect first, and how would you test that hypothesis?
- What would you recommend if the drop is concentrated only among long-time, habitual users?
More metrics questions
- What metrics would tell you whether Perplexity is winning against Google Search?Perplexity · Metrics · Hard
- You have one team and can materially advance only one of these areas in the next 6-12 months: Skills, collaboration, artifact generation, or Projects. How would you choose where to invest, and what leading indicators and longer-term outcome metrics would you use to defend that roadmap to leadership?Perplexity · Metrics · Hard
- Users can create Skills in Computer, but many run a Skill once and never use it again. How would you determine whether the main issue is setup friction, weak discoverability, inconsistent results, narrow applicability, or lack of trust, and what product changes would you test first to improve repeat use and sharing?Perplexity · Metrics · Hard
- Suppose Perplexity launches a new 'resume previous work' feature for research and productivity workflows. What metrics would you use to determine whether it improves retention, and how would you separate durable user value from short-term engagement spikes caused by novelty or accidental usage?Perplexity · Metrics · Hard
- Pick one Perplexity use case, research, investing, or shopping, and design a first prototype in Computer for a domain expert. What exact job-to-be-done would it solve, what would the user see and control step by step, and which early metrics would tell you the prototype is genuinely improving productivity?Perplexity · Metrics · Medium
- Perplexity cares about product-led growth and data-driven flywheels. For a new AI productivity feature inside Search or Computer, what user actions would you optimize first to drive activation, retention, and learning effects, and how would you know the feature is creating a real flywheel rather than a one-time novelty spike?Perplexity · Metrics · Hard
More questions from Perplexity
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop