Metrics question
Suppose Vercel launches a native tracing experience built on OpenTelemetry. What north-star, guardrail, and funnel metrics would you define from telemetry ingestion through successful debugging, and how would you instrument the product so you can distinguish collection health from actual customer value?
- Vercel
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can design a metrics framework for a telemetry product that separates pipeline health from actual customer value, since a tracing feature can look healthy on ingestion while never helping anyone debug faster.
How to approach it
- North star: median time from an incident starting to root cause identified using traces, since that is the actual value promise of tracing.
- Funnel metrics from ingestion to value: percentage of deployments with tracing enabled, spans successfully ingested without drops, traces actually opened during an incident, and traces that led to a resolved root cause.
- Guardrail metrics: ingestion latency and data loss rate, since a tracing product that drops spans under load is actively harmful to debugging trust.
- Instrument collection health, ingestion success rate and latency, as a separate dashboard from customer value metrics, so a healthy pipeline is never mistaken for customers actually benefiting.
- Track a leading indicator, traces opened per incident, against the lagging indicator, resolution time reduction, to catch cases where tracing is enabled and collecting data but nobody is actually using it during real incidents.
What a strong answer includes
- Explicitly separates collection health from customer value with different dashboards, directly answering what the question asks.
- Picks a north star tied to actual debugging speed rather than a data-volume metric like spans ingested.
- Names the specific failure mode to watch for: tracing enabled and healthy but unused during real incidents, which is the crux of the collection-versus-value distinction.
Common mistakes
- Uses ingestion volume or uptime as the north star, which measures pipeline health, not customer value.
- Never separates the two categories, risking a launch that looks successful on dashboards but does not actually help anyone debug.
Likely follow-up questions
- How would you attribute a faster resolution time specifically to tracing versus other debugging tools in use.
- What would you do if traces are opened frequently but resolution time does not improve.
More metrics questions
- What metrics would you use to measure v0's success (3.5M users, 50%+ revenue from Teams/Enterprise)?Vercel · Metrics · Medium
- How would you measure the performance and health of a Netflix Original?Microsoft · Metrics · Hard
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Facebook Ads dropped by 20%. Mark Zuckerberg calls you and asks you to fix it. What do you do?Meta · Metrics · Hard
- Imagine you're the product manager for Facebook Marketplace. Since many sellers don't mark items as sold, what existing functionality and metrics could you use to determine whether an item has likely sold?Meta · Metrics · Hard
More questions from Vercel
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop