Metrics question
Customers are tweeting about the incorrect wait times recommended by our waitlist feature. How would you investigate this?
- Yelp
- Metrics
- Medium
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Structured root-cause investigation skill when a live feature is producing bad predictions and generating public complaints.
How to approach it
- Quantify the problem first: pull wait-time prediction error, predicted versus actual, across restaurants and time windows, not just the tweets.
- Segment the error by restaurant type, party size, day and time, and city to find where the model breaks down most.
- Check recent changes: model updates, restaurant onboarding changes, or a data pipeline issue feeding stale seating data.
- Look at input data quality, since restaurants not updating table status promptly causes bad predictions regardless of the model.
- Talk to a sample of affected restaurants and diners to understand the specific failure pattern behind the complaints.
- Decide on a fix path: retrain the model, patch a specific segment, or show a range instead of a precise number.
What a strong answer includes
- Separates model error from data freshness issues, a common root cause interviewers want named explicitly.
- Proposes showing a range instead of a single number as a quick mitigation while the root cause is fixed.
- Uses the tweets as an early signal but insists on quantifying error before acting.
- Checks whether this is a new regression or a long-standing, never-noticed problem.
Common mistakes
- Reacting only to the tweets without pulling systematic error data.
- Assuming it is the model without checking input data freshness.
- Jumping to a fix before segmenting where the error concentrates.
Likely follow-up questions
- How would you design a long-term monitoring metric for wait-time accuracy?
- What would you tell restaurants while the fix is in progress?
- How do you decide when prediction error is acceptable versus a real bug?
More metrics questions
- How do you define success for Yelp reviews?Google · Metrics · Medium
- What is the north star metric for Yelp?Yelp · Metrics · Easy
- What metrics do you think are important to Yelp?Yelp · Metrics · Easy
- How do you find the top 10 brunch places in the USA, assuming you have all the Yelp ratings and review data?Yelp · Metrics · Medium
- How would you measure the success of Facebook Likes?Meta · Metrics · Medium
- Walmart's order return rate is increasing. As a product manager, what things would you look into to isolate the problem?Amazon · Metrics · Medium
More questions from Yelp
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop