AI & Technical question

Lovable wants to catch agent regressions before launch. Design an evaluation framework for a tool-using, multi-step LLM agent: what offline and online evals would you run, how would you segment failures (for example reasoning, tool selection, tool execution, recovery, and communication), and what launch gates would determine whether a release is safe?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Whether you can build a launch-gating evaluation framework for a multi-step tool-using agent, covering both offline testing and live guardrails.

Build offline evals from a curated task suite covering common build patterns, scored automatically where possible (does the code compile, does the app run) and by human rubric where not.
Segment failures into stages: reasoning (wrong plan), tool selection (right plan, wrong tool), tool execution (right tool, bad call), recovery (fails to notice or fix an error), and communication (misrepresents what happened).
Run regression evals on every release candidate against the full task suite plus a held-out set of previously-found failure cases, so fixed bugs cannot silently reappear.
Add online guardrails: live monitoring of tool-call error rates and retry loops, with automatic alerts if a release shows an anomalous spike versus the prior version.
Set launch gates per failure category, for example reasoning and tool-selection errors must not regress beyond the current baseline, since those are the highest-trust-damaging categories.
Require a canary rollout with live monitoring before full release, with an automatic rollback trigger if guardrail metrics breach threshold.

Breaks failures into the five named stages, since each requires a different fix and a different eval method.
Uses a held-out regression set of past failures specifically to catch regressions, not just measure current quality.
Sets stage-specific launch gates instead of one blended pass or fail score, since reasoning failures are more damaging than minor communication issues.
Pairs offline evals with a canary and automatic rollback, since offline suites cannot catch every real-world failure mode.

Using a single blended pass rate instead of segmenting failures by stage.
No regression suite, so previously fixed bugs can silently reappear in a new release.
No online guardrail or rollback plan, relying only on pre-launch offline testing.

How would you build the initial curated task suite to be representative of real usage?
What would you do if the canary shows a regression only affecting a small user segment?
How would you decide which failure category to invest in fixing first?