Metrics question

What metrics would you use to determine whether the under-18 ChatGPT experience is both helpful and safe? Define a north-star metric, guardrails, and one metric that could be misleading, then explain how those metrics would change your roadmap.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Metrics design for a dual-objective product where optimizing one goal, helpfulness, could actively work against the other, safety.

Propose a north-star metric that blends both goals conceptually, like the rate of sessions rated by review as both age-appropriate and genuinely helpful to the user's need.
Propose guardrail metrics separately from the north star: rate of confirmed policy-violating content served to under-18 accounts, held to a near-zero threshold regardless of engagement.
Name a metric that could be misleading: overall engagement or session length, since rising engagement could reflect either genuine helpfulness or unsafely permissive behavior, and cannot distinguish the two alone.
Explain how these metrics would change the roadmap: if the guardrail metric worsens even slightly, that should pause any feature expansion, prioritized above growth-oriented improvements to the north star.
Propose combining automated detection with periodic expert human review of a sampled set of teen conversations, since safety violations are often too subtle or context-dependent for automated flags alone.

Explicitly names engagement or session length as a metric that could mislead, correctly identifying that it cannot on its own separate genuine helpfulness from unsafe permissiveness.
Sets a near-zero threshold guardrail for confirmed safety violations, treating it as non-negotiable rather than something traded off against growth metrics.
States a clear roadmap rule: any guardrail regression pauses feature expansion, giving a concrete answer to how metrics would change priorities.
Proposes combining automated and human expert review, recognizing that safety in nuanced teen conversations cannot be fully captured by automated signals alone.

Proposing engagement or usage growth as the north star without flagging it as a potentially misleading signal for this specific product.
Failing to state a clear rule for how a guardrail regression should actually change the roadmap, leaving the tradeoff unresolved.

How would you set the sample size for human review to catch rare but serious violations?
What would you do if the guardrail metric and the north star metric moved in opposite directions?