Metrics question
Compare two search algorithms by defining the North Star KPI. Measuring that KPI through 3 metrics and identify success flags.
- Agoda
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Rigorous experiment measurement design for comparing two ranking algorithms, requiring a coherent metric hierarchy and clear success criteria.
How to approach it
- Define the North Star KPI: booking conversion rate from search results, the metric that most directly reflects whether search surfaces genuinely useful results.
- Choose three supporting metrics that causally feed the North Star: click-through rate on top results, average position of the booked result, and search abandonment rate.
- Define success flags for each: click-through should rise or hold steady, average booked position should decrease, and abandonment should fall.
- Run the comparison as a controlled A and B test between the two algorithms on matched traffic segments, ensuring enough volume for significance.
- Add a guardrail: revenue per search, to ensure a conversion-rate improvement isn't achieved by biasing toward lower-value bookings.
- Set the overall decision rule: the new algorithm wins if the North Star improves and at least two of three supporting metrics move rightly without hurting revenue.
What a strong answer includes
- Chooses a North Star, booking conversion rate, that directly reflects search quality rather than a vanity metric.
- Selects three metrics genuinely causally connected to the North Star, not just three unrelated numbers.
- Defines explicit, specific success flags for each metric rather than a vague should improve.
- Adds a revenue guardrail, catching a subtle risk where conversion improves but average booking value quietly falls.
Common mistakes
- Picking supporting metrics with no clear causal link to the chosen North Star.
- Not defining explicit success thresholds before running the comparison.
- Ignoring a revenue or value guardrail that could mask a metric being gamed.
Likely follow-up questions
- How would you size the test to detect a meaningful difference with confidence?
- What would you do if the North Star improved but two of the three supporting metrics didn't?
- How would you control for seasonality or demand differences between test groups?
More metrics questions
- How would you measure the performance and health of a Netflix Original?Microsoft · Metrics · Hard
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Facebook Ads dropped by 20%. Mark Zuckerberg calls you and asks you to fix it. What do you do?Meta · Metrics · Hard
- Imagine you're the product manager for Facebook Marketplace. Since many sellers don't mark items as sold, what existing functionality and metrics could you use to determine whether an item has likely sold?Meta · Metrics · Hard
- Uber has seen a dip in the ride confirmation rate. As a PM at Uber, you are responsible to investigate and fix the issue. How will you go about it?Uber · Metrics · Hard
More questions from Agoda
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop