AI & Technical question

A customer reports that Sierra’s agent performs well in English but has lower containment and CSAT in Brazilian Portuguese because it misses slang, tone, and regional phrasing. How would you diagnose whether the issue is coming from prompts, retrieval/content quality, evaluation coverage, workflow design, or the underlying model, and what would you ship first?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests structured diagnosis across the AI stack, separating prompt, retrieval, eval, workflow, and model causes for a language specific quality gap.

Check whether the eval set even has adequate Brazilian Portuguese coverage, since a coverage gap in evaluation can hide the real problem.
Review retrieval and knowledge base content for Portuguese language gaps or untranslated English only source documents.
Audit prompts and policies for English centric phrasing or examples that do not transfer well to Portuguese slang and tone.
Test the underlying model's Portuguese fluency directly, isolated from retrieval and prompts, to rule out a base model limitation.
Review workflow design for assumptions built around English conversational patterns that do not map to Brazilian communication norms.
Ship the fastest fix first, likely content and eval coverage gaps, since those are usually cheaper to fix than a model level limitation.

Checks eval coverage first, since a common root cause is that the team cannot even see the Portuguese failure modes clearly.
Isolates model quality from retrieval and prompt issues by testing the model directly, rather than assuming it is a model problem.
Sequences fixes by cost and speed, shipping content and prompt fixes before requesting a model level change.

Assuming it is a model limitation without first checking retrieval content and eval coverage in Portuguese.
Fixing prompts without validating whether the eval set can even detect the improvement.

How would you build an eval set that reliably catches Brazilian Portuguese specific failures?
What would you do if the root cause turns out to be the underlying model itself?