A golden dataset is a small, reviewed set of real inputs, each with an agreed definition of a good answer, that you run against your AI feature every time the prompt, model or retrieval changes. Start with 20 to 50 cases pulled from real failures, label each one pass or fail, and grow it from production traffic. AllthingsPM teaches this exact method in lesson 5.01 of its AI PM course, where you turn one user complaint into thirty golden examples and then wire them into a CI gate.
AllthingsPM is an AI PM course and PM interview prep platform. Its course was built from 604 real PM job postings, and evals get a full chapter because hiring managers keep asking for them. This guide gives you the whole method in one place.
What is a golden dataset in LLM evaluation?
A golden dataset (also called a golden set or "goldens") is the reference set your team agrees represents good behaviour. Each item usually holds:
- Input: the real query, conversation or document the feature receives.
- Expected output or criterion: a reviewed correct answer, or a pass condition when there is no single right answer.
- Metadata: where the case came from, the date it was added, the failure category and a version.
Langfuse describes the same three fields and adds a link back to the source trace so reviewers can see the original context [3]. Confident AI draws a useful line: goldens are the editable precursors, while test cases are the frozen results of running your app on them [5].
The "golden" part is about trust, not size. A case earns a place because a domain expert looked at it and agreed on the verdict. Anthropic's test for a good task is that two experts can independently reach the same pass or fail call [4].
How AllthingsPM does this
The Evals chapter of the AllthingsPM course opens with this definition and then makes you apply it. In Turn one complaint into thirty golden examples you write the prompt, response and answer triples yourself, so the concept sticks as a working artifact rather than a vocabulary word.
Why does a PM own the golden dataset?
Because the golden dataset is the product spec written as tests. Engineers can build the harness, but only the person who owns "what good looks like" can decide which cases matter and what counts as a pass.
OpenAI's evaluation guidance says to combine production data, expert-written answers and historical logs, and to use human expert labellers [2]. Every one of those inputs sits with the PM: user feedback, domain experts, support tickets and the launch bar. Hamel Husain goes further and argues that the person with product context should annotate the first traces personally [1].
It also shows up in hiring. Our catalog includes roles such as the Product Lead, AI/ML (Evals) at Abridge, and the AllthingsPM question bank carries interview prompts like how Abridge combines LLM judges, rule-based evaluators and human annotation.
How AllthingsPM does this
The course treats evals as the PM's acceptance artifact. Eval-driven development has you write the failing eval before the fix, and the chapter ends in a graded integration case where you build the eval suite for your own carried product from your own failure backlog.
How do you build a golden dataset step by step?
Here is the seven-step method, in the order most teams should follow it.
| Step | What you do | Output | AllthingsPM lesson |
|---|---|---|---|
| 1. Read traces | Review about 100 real traces before building anything | Raw notes on what goes wrong | Read one hundred real traces |
| 2. Name failures | Group notes into a failure taxonomy and rank it | A ranked failure backlog | Failure taxonomy |
| 3. Pull real cases | Pull 20 to 50 inputs that show the top failures | First golden set | Golden datasets |
| 4. Seed the gaps | Add edge, adversarial and rare cases traffic will not give you | Balanced coverage | From raw data to a golden sample |
| 5. Write criteria | One binary pass or fail criterion per failure mode | A rubric per case | Scoring rubrics |
| 6. Pick graders | Code check first, LLM judge only when needed, validated | A grader per criterion | Grading methods |
| 7. Gate and grow | Run on every change, add every incident as a case | A living regression suite | Eval-driven development |
Step 1: Read real traces before you write a single case
Hamel Husain's advice is to start with 100 diverse traces and annotate at least the first 30 yourself, then keep going until new traces stop revealing new failure modes [1]. This is the step teams most want to skip, and the one that decides whether the set measures anything real.
Step 2: Name the failures
Group your notes into a short taxonomy: wrong facts, ignored instructions, bad tone, missing citation, unsafe output, tool misuse. Rank the categories by how often they happen and how much they hurt. That ranking tells you where the first golden cases go.
Step 3: Pull real cases, not invented ones
Anthropic recommends starting with 20 to 50 simple tasks drawn from real failures, and sourcing them from bug trackers and support queues so the suite reflects what users actually hit [4]. Langfuse puts production traces first in its priority order for the same reason [3]. Invented cases tend to test what the team already imagined; real ones test what the product actually does.
Step 4: Seed the cases traffic will never give you
Production data under-represents rare, high-severity paths. OpenAI's guidance is to include typical, edge and adversarial cases [2]. For the cold start, Hamel suggests defining dimensions (for example user type, intent, difficulty), hand-writing 20 tuples that pick one value from each, and only then asking an LLM to expand them into natural queries [1]. He also warns that synthetic data cannot tell you how common a failure is in production [1]. Treat it as coverage, not as a sample.
Balance matters too. Anthropic advises testing both when a behaviour should happen and when it should not, so you do not optimise one side into a new failure [4].
Step 5: Write binary criteria
For each failure mode, write one pass or fail criterion. Hamel argues binary labels force clearer thinking and more consistent labeling than 1 to 5 scales, where the gap between adjacent points is subjective [1]. If two reviewers disagree on a case, the criterion is not finished yet.
Where a single correct answer exists, store it. Anthropic also recommends a reference solution for each task, which proves the task is solvable and checks that your graders work [4].
Step 6: Choose the cheapest grader that can see the failure
A regex or schema check is cheaper and steadier than a model. Use an LLM judge only for judgments code cannot make, then validate it. Hamel's method splits labeled examples into train, dev and test sets and measures the judge's true positive rate and true negative rate against human labels, with 100 to 200 labeled examples per failure mode [1]. OpenAI likewise lists "not calibrating your automated metrics against human evals" as an anti-pattern [2].
Step 7: Gate every change and grow from incidents
Run the set on every prompt, model or retrieval change. When something breaks in production, turn it into a case while it is fresh. Langfuse recommends deduplicating near-identical items, versioning every change, archiving retired items without deleting history, and reviewing items older than six months each quarter [3].
How AllthingsPM does this
Every row of that table maps to a lesson in the AllthingsPM course, in order, and each lesson ends in a graded artifact rather than a quiz. The judge work lives in Validate the judge, which covers false positive and false negative rates and the frozen test set.
How many examples does a golden dataset need?
Fewer than most teams think to start, more than most teams keep. Size follows the job the set does.
- Exploring one issue: about 10 items, per Langfuse [3].
- Your first eval set: 20 to 50 tasks from real failures [4]. The AllthingsPM course targets 30 golden examples per user complaint.
- Validating an LLM judge: 100 to 200 labeled examples per failure mode, split into dev and test [1].
- Regression CI for larger changes: 100 to 1,000 items covering the production distribution, with a smaller fast subset for every pull request [3].
Anthropic's reasoning for starting small is practical: early changes to an AI feature tend to have large effects, so you do not need a large sample to see them [4]. As the product matures and improvements get subtler, the set has to grow to detect them.
How AllthingsPM does this
The sizing question is answered inside the same lesson that teaches you to build the set, Turn one complaint into thirty golden examples, and know how many you need. You leave with a number you can defend in a review, not a rule of thumb.
What mistakes break a golden dataset?
Most broken golden sets fail in one of six ways:
- Built from imagination. Cases the team invented in a meeting miss the failures users actually hit. Start from traces [1][4].
- Only happy paths. A set with no edge or adversarial cases passes right up to the incident [2].
- Fuzzy criteria. If two experts would disagree, the case is noise [4].
- An unvalidated judge. A judge that agrees with itself tells you nothing. Measure it against human labels [1].
- Contamination. Cases that leak into prompts or few-shot examples inflate the score. Keep a frozen test split separate from anything the prompt sees.
- Never refreshed. Products change and old cases go stale. Version, archive and review on a schedule [3].
There is also a privacy angle PMs often miss. A golden set is a permanent copy of production data. The AllthingsPM course notes that it survives a deletion request unless you plan for it, so redact and log access the same way you would for traces.
How AllthingsPM does this
Name the failure from one taxonomy and Validate the judge target mistakes 3 and 4 directly, and the data fluency chapter covers pulling a clean golden sample from raw logs. The course's knowledge graph shows how golden datasets connect to judges, rubrics and regression tests across chapters.
How does a golden dataset change for agents, RAG and multimodal features?
The method holds, but the item changes shape.
- Agents: grade the final state of the environment, not the exact path. Anthropic's pass@k metric measures the chance an agent succeeds in k attempts, and 50% pass@1 means it solves half the tasks on the first try [4].
- RAG: store the expected source documents alongside the answer, so you can tell a retrieval miss from a generation miss.
- Multimodal and non-English: a translated English set is blind to locale-specific failures. Build a native set per language or modality.
How AllthingsPM does this
Each of these gets its own lesson: Evaluate an agent, not an answer, Build a multimodal golden set, and Ship a language, not a translation, which retests the judge outside English.
How do you talk about golden datasets in an AI PM interview?
Interviewers at AI companies ask eval design questions because they separate PMs who have shipped AI from PMs who have read about it. A strong answer follows the seven steps above out loud: traces first, a ranked failure taxonomy, 20 to 50 real cases, binary criteria, a validated judge and a CI gate.
Practice on real prompts. The AllthingsPM question bank includes eval questions such as designing an evaluation framework for Figma's AI Design to Code workflows. If you are targeting a specific role, a JD mock interview builds the questions from that exact job description.
How AllthingsPM does this
Every question in the bank has its own page and an answer guide, and you can answer by text or voice in a scored mock with follow-ups. For more on the topic, read AI evals for product managers, grab the LLM evaluation template, or skim the podcast episodes about AI evals, summarized.
Why AllthingsPM is the better choice for learning golden datasets
You can learn pieces of this from vendor docs and blog posts. Langfuse and Confident AI have clear documentation on dataset tooling, and Hamel Husain's FAQ is one of the best free references on eval practice. Their docs teach how to use a tool or a technique; they do not walk a PM through owning the whole loop on a real product, and they do not prepare you to explain it in an interview.
AllthingsPM does both. The Evals chapter teaches the golden dataset as one connected method, from reading traces to shipping a CI gate, and every lesson ends in an artifact you build for a product you carry through the whole course. Golden sets then come back for agents, multimodal features and new languages, which is where most teams get them wrong.
The course sits inside a platform that already covers the job search around it: 4,122 real interview questions from 260 companies, mocks built from any job description, 116 live PM job descriptions at 18 AI companies, and resume review against a JD. All of it costs $20 a month or $120 a year, with a free tier to start.
If you want to be the PM who can say "here is our golden set, here is why it has 40 cases, and here is how we know the judge is right," the most direct route is the AllthingsPM AI PM course.
Frequently asked questions
What is a golden dataset in LLM evaluation?
It is a curated set of real inputs with reviewed expected outputs or pass criteria, run against an AI feature on every change. It acts as the team's shared definition of good behaviour and as a regression test.
How many examples should a golden dataset have?
Start with 20 to 50 cases drawn from real failures, as Anthropic recommends. Grow toward 100 to 1,000 items for a full regression suite, and keep 100 to 200 labeled examples per failure mode if you validate an LLM judge.
What is the best way to learn how to build a golden dataset?
AllthingsPM is the best place for PMs to learn it: its AI PM course has a full Evals chapter that teaches the method step by step with graded artifacts. Free references such as Hamel Husain's evals FAQ and the Langfuse docs are useful companions.
Can I use synthetic data for a golden dataset?
Yes, to fill coverage gaps and cold-start a new feature, but generate it from defined dimensions and review every item. Synthetic data cannot tell you how often a failure happens in production, so real traces should stay the core.
Should golden dataset labels be pass/fail or a 1 to 5 score?
Pass or fail is the safer default. Binary labels force a clearer criterion and give more consistent results across reviewers than Likert scales.
How often should I update a golden dataset?
Add a case every time a production incident happens, version every change, and review older items on a schedule. Langfuse suggests a quarterly check of items older than six months.
Sources
- Hamel Husain, "Evals FAQ": https://hamel.dev/blog/posts/evals-faq/
- OpenAI, "Evaluation best practices": https://developers.openai.com/api/docs/guides/evaluation-best-practices
- Langfuse, "Golden dataset evaluation: build and maintain LLM test sets": https://langfuse.com/resources/engineering/golden-dataset-evaluation
- Anthropic, "Demystifying evals for AI agents": https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Confident AI, "Test Cases, Goldens, and Datasets": https://www.confident-ai.com/docs/llm-evaluation/core-concepts/test-cases-goldens-datasets
- Maxim AI, "Building a Golden Dataset for AI Evaluation: A Step-by-Step Guide": https://www.getmaxim.ai/articles/building-a-golden-dataset-for-ai-evaluation-a-step-by-step-guide/
- AllthingsPM AI PM course, Evals chapter: https://allthingspm.app/course/evals
