Metrics question

This team cares about measurable improvement in defensive outcomes per analyst-hour. For an AI-assisted threat investigation product, what metrics would you use across product quality, operational outcomes, and user behavior? Which would be leading vs. lagging indicators, and how would you handle tradeoffs if adoption is high but investigation accuracy or safety is weak?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Metrics design that connects product usage to real defensive outcomes and forces an explicit tradeoff decision.

Propose product quality metrics: investigation accuracy, measured against expert-labeled ground truth cases, and false positive or false negative rate on flagged threats.
Propose operational outcome metrics: mean time to investigate an alert, and defensive outcomes per analyst-hour, the named north star, calculated as investigations completed per hour of analyst time spent.
Propose user behavior metrics: adoption rate among analysts, and reliance rate, meaning how often analysts accept the AI's investigation findings without significant manual rework.
Classify leading versus lagging: adoption and reliance rate are leading indicators of habit formation, while mean time to investigate and accuracy are lagging indicators of actual defensive value delivered.
Address the tradeoff explicitly: if adoption is high but accuracy or safety is weak, pause scaling and prioritize accuracy fixes first, since high adoption of an inaccurate tool actively increases risk rather than reducing it.

Organizes metrics into the three named categories explicitly, product quality, operational outcomes, user behavior, rather than one undifferentiated list.
Correctly classifies adoption and reliance as leading indicators and time-to-investigate or accuracy as lagging, showing understanding of how these metrics move at different speeds.
Makes the tradeoff decision explicit and directionally clear: high adoption with weak accuracy is a reason to slow down, not scale up, since it compounds risk.
Names a concrete north star aligned to the question's own framing, defensive outcomes per analyst-hour, and defines exactly how it would be calculated.

Listing metrics without classifying which are leading versus lagging, missing an explicit part of the question.
Treating high adoption as an unambiguous success signal without addressing the case where accuracy or safety is weak.

How would you build the expert-labeled ground truth needed to measure investigation accuracy?
What threshold of accuracy would you require before allowing continued scaling despite high adoption?