AI & Technical question

You discover Claude performs well on alert triage but poorly on attacker path analysis in complex enterprise environments. How would you diagnose whether the gap is caused by eval design, missing context/tooling, model capability limits, or safeguard/policy constraints, and how would you turn that into a cross-functional plan with research, engineering, safeguards, and policy?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Diagnostic rigor in separating an eval problem from a real capability gap, and the ability to turn that into a cross functional plan.

Start by auditing the eval itself: check whether attacker path analysis test cases reflect real world complexity or are artificially harder, and whether scoring criteria are fair.
Check for missing context or tooling next, for example whether the model lacks access to network topology data or prior alert history that a human analyst would have.
If the eval and context look sound, test raw model capability directly with a simplified version of the task to isolate whether reasoning over multi step attacker behavior is the actual limit.
Check safeguard or policy constraints last, since overly conservative guardrails could be suppressing otherwise capable outputs on this class of task.
Based on the most likely cause, form a cross functional plan: research if it is a capability gap, engineering if it is a tooling or context gap, safeguards and policy if it is a constraint issue.
Set a re test checkpoint to confirm which fix actually closed the gap, since more than one factor could be contributing.

Runs the diagnosis in a logical order, ruling out eval and context issues before concluding it is a fundamental capability gap.
Proposes a specific test, like giving the model missing context to isolate whether that alone closes the gap.
Builds a cross functional plan mapped precisely to the diagnosed cause, not a generic work with research answer.
Includes a re test step to confirm causation, not just correlation.

Jumping to the model just is not good enough without ruling out eval or context issues first.
Proposing a cross functional plan that involves every team regardless of what the diagnosis actually points to.