AI & Technical question

A pilot customer reports that an AI drafting workflow is 'unreliable,' but you do not know whether the failure is coming from model behavior, retrieval/data quality, UX/workflow design, trust/policy constraints, or deployment issues. How would you isolate the root cause, what instrumentation or evals would you inspect, and how would the next action differ by diagnosis?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Structured root-cause diagnosis for an AI product failure across the five plausible failure layers named in the question.

Start by reproducing the specific failure cases the customer reported, collecting exact examples rather than relying on a vague complaint of unreliable.
Check model behavior first: run the same inputs through eval suites to see if the model itself produces inconsistent or low-quality output independent of the product wrapper.
Check retrieval and data quality: verify whether the drafting workflow pulled the correct, current source documents, since stale or wrong retrieval looks identical to a model failure from the user's side.
Check UX and workflow design: review whether users are providing the context the model actually needs, since a confusing input flow can produce bad output even from a good model.
Check trust or policy constraints and deployment: confirm whether safety filters are over-triggering and silently degrading output, or whether a deployment issue, like stale caching or a config mismatch, is the real cause.

Insists on reproducing concrete failure examples first, rather than trying to diagnose from a vague unreliable label.
Walks through the five named layers systematically, showing each has a distinct diagnostic method, not one generic debugging approach.
Notes explicitly that retrieval failures and model failures can look identical to the end user, which is why isolating the layer matters before proposing a fix.
Ties the next action to the diagnosis: a model issue needs prompt or fine-tuning work, a retrieval issue needs data pipeline fixes, a UX issue needs interface changes, and each requires a different owning team.

Jumping to a fix, like re-prompting the model, before confirming which of the five layers is actually responsible.
Treating unreliable as a single diagnosis instead of decomposing it into distinct, testable failure hypotheses.

What instrumentation would you add so this diagnosis is faster for the next pilot customer?
How would you communicate the root cause and timeline back to the pilot customer while you investigate?