MapleLoop · Metrics · Mid · 45 to 60 minutes

How should we measure our RAG assistant?

MapleLoop is a hypothetical enterprise software company whose internal knowledge assistant uses retrieval augmented generation to answer employee questions from company documents. Answer this product management case and get it graded line by line against a weighted rubric.

Metric design
Analytical reasoning
Trade-off judgment
Experimentation
Communication

The situation

You are the PM for MapleLoop's internal knowledge assistant. The head of product and the CTO have asked you for a clear success framework before the next quarterly review. Usage is growing, but leaders disagree on what that growth means. Some stakeholders want to track time spent, conversation length, and repeat usage as the main indicators. Others worry those numbers could rise when answers are unclear, users get stuck, or the assistant sends people on longer hunts through documents. The search team argues that retrieval quality should be measured directly, while operations leaders want proof that employees actually solve tasks faster and trust the answers more. Assume the assistant can log queries, retrieved documents, citations shown, answer text, clicks on cited sources, and a small amount of explicit feedback, but there is little manually labeled relevance data today. You need to recommend how MapleLoop should measure success now, which metrics should be primary versus diagnostic, and what instrumentation gaps matter most. The hard part is balancing technically sound evaluation with a metric system that leaders can understand and use.

What to produce

Write a 1 to 2 page metrics memo.

Your answer must

Define a metric hierarchy with 1 primary success metric, 3 to 5 supporting metrics, and at least 1 guardrail metric.
Explain why at least 2 common engagement metrics could be misleading for a retrieval augmented generation assistant.
Include at least 1 retrieval ranking metric, such as mean reciprocal rank or a defensible alternative, and explain what data it would use.
Show how the proposed technical metrics connect to user outcomes, such as task completion, faster resolution, or trust.
Recommend the first instrumentation or data collection improvements you would prioritize in the next quarter.

Constraints

Use a framework that can be reviewed monthly in a dashboard with no more than 6 total metrics.
Do not rely on a large new human labeling program before the next quarterly review.
Your approach must still work when many users do not click cited sources or leave explicit feedback.

Format

Use sections for goal, metric hierarchy, misleading metrics, instrumentation plan, and risks.

How MapleLoop actually played it

Explains how to measure success for a retrieval augmented generation assistant, warning that traditional engagement metrics can be misleading and introducing ranking measures such as mean reciprocal rank.

Time spent can mislead in AI answers
Use ranking quality metrics
Tie technical metrics to user outcomes

Built from How would you measure the success of RAG? by Technomanagers.