AI & Technical question

How would you measure word error rate so it reflects real user experience?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests AI evaluation design: making a technical accuracy metric actually reflect what matters to real users.

Start from the gap: raw word error rate treats every word mistake equally, but a missed critical word, like a name or number, matters far more to a real user than a missed filler word.
Propose a weighted error metric: assign higher weight to errors on content words, proper nouns, and numbers than to errors on function words or filler.
Segment by real usage context: measure error rate separately for the conditions users actually encounter, like noisy environments or specific accents, not just clean test audio.
Add a downstream task based metric: for example, in a voice agent context, measure whether the error caused the agent to misunderstand the user's actual intent, not just miscount words.

Proposes weighting errors by their real impact, like a missed name or number versus a missed filler word, directly addressing why raw word error rate can mislead.
Ties the metric to a downstream task outcome, like whether a voice agent still understood user intent despite a transcription error, connecting the metric to real usefulness.
Validates the proposed metric against human judgment of usability, ensuring the weighted score actually correlates with what users experience as good or bad.

Treating raw word error rate as sufficient without addressing that not all word errors matter equally to a real user.
Reporting only clean lab conditions, missing the noisy, real world conditions where users actually experience the product.

How would you decide which words deserve higher weight in the metric?
How would this metric differ for a transcription product versus a conversational voice agent?