Metrics question
Post-launch, ChatGPT shopping recommendations show high engagement but low purchase conversion and weak trust scores. How would you diagnose where trust is breaking down, what evidence would you gather, and how would you prioritize the fixes?
- OpenAI
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests diagnostic thinking for a trust gap between high engagement and low conversion, using evidence to prioritize fixes rather than assuming the cause.
How to approach it
- Review qualitative feedback and support signals first to hear in users' own words why they engage but do not purchase.
- Check whether recommendations disclose sourcing and any commercial relationships clearly, since undisclosed bias is a common driver of low trust scores.
- Check price and availability accuracy, since a recommendation that turns out wrong or outdated at the retailer erodes trust fast even if engagement stays high.
- Segment low conversion by category, since trust issues may concentrate in higher consideration purchases like electronics versus low risk categories like household goods.
- Test a transparency focused fix, such as clearer sourcing disclosure or showing comparison reasoning, before assuming a deeper personalization or ranking rebuild is needed.
- Prioritize fixes by evidence strength and speed to ship, starting with the most frequently cited trust issue in user feedback.
What a strong answer includes
- Starts diagnosis with direct user feedback rather than assuming the cause from behavioral data alone, which is the fastest way to find why trust specifically is low.
- Checks recommendation transparency and disclosure explicitly, since undisclosed commercial bias is a very plausible and fixable cause of a trust gap.
- Segments by purchase consideration level, recognizing that trust issues in electronics likely differ from trust issues in household goods.
Common mistakes
- Assuming low conversion means the recommendations are simply wrong, without checking whether disclosure or transparency is the real trust issue.
- Prioritizing a full personalization rebuild before testing cheaper transparency fixes that might address the trust gap faster.
Likely follow-up questions
- How would you measure whether a transparency fix actually improved the trust score?
- What would you do if trust scores stay low even after fixing disclosure and accuracy issues?
More metrics questions
- Weekly active users of Codex dropped 15% after a pricing change. How do you investigate?OpenAI · Metrics · Medium
- What metrics would you track to measure the success of ChatGPT Projects?OpenAI · Metrics · Medium
- OpenAI wants one topline safety metric for frontier model deployments. How would you define it so it is credible for leadership decisions, sensitive enough to detect meaningful changes in harm, and decomposable into drivers that research and engineering teams can act on?OpenAI · Metrics · Hard
- This team cares about measurable improvement in defensive outcomes per analyst-hour. For an AI-assisted threat investigation product, what metrics would you use across product quality, operational outcomes, and user behavior? Which would be leading vs. lagging indicators, and how would you handle tradeoffs if adoption is high but investigation accuracy or safety is weak?OpenAI · Metrics · Hard
- What metrics would you use to judge whether a legal AI product is working in a 5-customer pilot versus a scaled rollout? Be specific about user-value, trust/quality, operational, and business metrics, and explain which ones are leading indicators versus launch gates.OpenAI · Metrics · Medium
- What metrics would you use to determine whether the under-18 ChatGPT experience is both helpful and safe? Define a north-star metric, guardrails, and one metric that could be misleading, then explain how those metrics would change your roadmap.OpenAI · Metrics · Hard
More questions from OpenAI
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop