Metrics question

What metrics would you use to evaluate a conversational shopping experience in ChatGPT? Define a north-star metric and supporting metrics for user value, trust, merchant ecosystem health, and business outcomes, and explain the tradeoffs among them.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests north star and supporting metric design for a three sided shopping marketplace balancing user, merchant, and business interests.

Choose a north star that reflects genuine user value delivered, such as confident completed purchases or decisions made without regret, rather than raw engagement or GMV.
Track user value supporting metrics: discovery to decision time, comparison depth used, and post purchase satisfaction or return rate.
Track trust supporting metrics: disclosure clarity ratings, recommendation accuracy against actual price and availability, and complaint rate.
Track merchant ecosystem health: merchant participation rate, fair exposure across merchants rather than concentration in a few large ones, and merchant reported conversion quality.
Track business outcomes: take rate or commission revenue, repeat shopping session rate, and overall GMV as a lagging validation, not the primary target.
Explain the tradeoff explicitly: optimizing purely for GMV risks pushing users toward more expensive or less suitable options, undermining trust and merchant ecosystem health long term.

Chooses a north star around decision confidence or completed value rather than raw GMV, avoiding the trap of optimizing for revenue at the cost of trust.
Includes merchant ecosystem health as its own category, recognizing this is a three sided marketplace, not just a user and business relationship.
States the GMV versus trust tradeoff explicitly, showing awareness that a purely revenue optimized recommender would undermine long term user and merchant trust.

Choosing GMV or conversion rate as the north star, which creates a direct incentive to push users toward suboptimal purchases.
Ignoring merchant ecosystem health entirely, treating this as a two sided user and business metric problem instead of three sided.

How would you detect if the recommendation system is quietly favoring merchants who pay more over what is actually best for the user?
Which one metric would you protect even if it meant sacrificing short term GMV growth?