Metrics question

How would you measure whether Lindy agents reliably complete their tasks?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests metrics design for evaluating whether autonomous agents actually complete tasks correctly, not just run without crashing.

Define task completion precisely: the agent achieved the user's actual intended outcome, not just executed without an error.
Build the core metric: task success rate, where success is verified against a clear definition per task type, like a correctly sent email versus a technically sent but wrong one.
Separate technical failure from logical failure: an agent that errors out is different from one that runs cleanly but does the wrong thing, and both need tracking.
Add a human correction signal: rate at which users edit, undo, or manually redo an agent's output, which reveals silent failures completion rate alone would miss.
Segment by task type and integration, since reliability on a simple calendar check likely differs from reliability on a complex multi step CRM update.
Confirm with the interviewer whether reliability should be measured per agent, per task type, or as one platform wide number.

Distinguishes technical failure from logical failure, since an agent that runs without error but produces the wrong output is a more dangerous and harder to detect failure mode.
Uses human correction or undo rate as a proxy for silent failures, catching cases a simple completion flag would miss entirely.
Segments reliability by task type and integration complexity, avoiding a misleading single blended number across very different task difficulties.
Proposes a sampled human audit of a subset of completed tasks, since fully automated verification cannot judge every kind of output correctness.

Measuring only whether the agent ran without a technical error, missing the more common failure of a wrong but well executed action.
Reporting one blended reliability number across very different task types and difficulties.
No signal for silent failures, where the agent appears successful but produced output the user had to quietly fix.

How would you verify task correctness for tasks with no clear right answer?
What would you do if reliability was high for simple tasks but low for complex multi step ones?
How would you use this reliability data to decide which agents get more autonomy?