Metrics question

Suppose you're the PM of an API that can analyse and categorize user's spends. What are the key metrics you would track to measure the success of this feature?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Metrics for a feature, not a whole product, focused on model quality, coverage, and downstream usage.

Clarify the goal: helping users or downstream apps understand spending patterns automatically, likely powering budgeting or insight features.
Define the core quality metric: categorization accuracy, the percent of transactions correctly auto-categorized without user correction.
Add a coverage metric: percent of eligible transactions assigned a category, plus how often users manually recategorize.
Add a downstream business metric: usage of features built on top, budgeting alerts triggered, or partner API call volume if this is B2B.
Add a guardrail: API latency, since a slow API breaks the consuming product's experience.
Track manual correction rate over time as a proxy for model quality improving or degrading.

Separates model-quality metrics, accuracy and coverage, from product-adoption metrics, correction rate and downstream usage, a distinction many miss.
Gives a concrete bar, for example aiming for over 90% auto-categorization accuracy before it drives user-facing budgeting insights.
Adds API latency as a guardrail since this is infrastructure other products depend on.

Only listing accuracy and ignoring coverage, transactions the model can't confidently categorize at all.
Forgetting this is likely a B2B or internal API, so partner adoption matters as much as end-user metrics.

Chapters of the AI PM course that teach what this question tests.