Curriculum

Evals

Evals are 53 percent (62 companies), real but middling against the top themes, and the most over-taught topic in the field. The report's verdict is right-size: teach the PM-grade version (define a failure taxonomy, build or commission a gol

1. 08.01

   Read one hundred real traces before you buy an

   Specify what every trace must emit, then sit with a hundred real production traces and let the failure categories emerge before you spend a dollar on eval infrastructure.
2. 08.02

   Name the failure from one taxonomy

   Classify each failed run against a single taxonomy, then roll the counts into a frequency-and-severity table engineering can work off.
3. 08.03

   Turn one complaint into thirty golden examples

   Grow a single user complaint, or synthetic cases sampled from the spec when there is none yet, into a trustworthy golden set, and judge how many examples you need.
4. 08.04

   Pay for the cheapest evaluator that can see th

   You match every failure mode to the least expensive grader that can reliably catch it, from a code assertion up through an LLM judge to a human annotator, so eval spend goes only where cheap graders are blind.
5. 08.05

   Write the criterion

   You default every criterion to a binary pass or fail, reach for pairwise or an anchored scale only when a binary genuinely cannot say it, and force the judge to write its reasoning so its biases become visible.
6. 08.06

   Validate the judge

   You measure your LLM judge like a classifier against a frozen, human-labeled test set, tracking its true-positive and true-negative rates separately because rare failures make raw agreement a lie.
7. 08.07

   Eval-driven development

   You write the failing eval that reproduces a bug before you fix it, then gate the deploy on that suite in CI.
8. 08.08

   Evaluate an agent

   You score an agent across three altitudes, outcome, trajectory, and step, and separate pass^k reliability across a whole task from single-run pass@k capability.
9. 08.09

   INTEGRATION CASE

   Real failures let you produce a pass rate on held back cases, with pass or fail checks that point to specific fixes.