A System for Testing New AI Models

Context

This Operator's Cut is about how to decide whether a new AI model actually deserves a place in your workflow. Nufar Gaspar argues that published benchmarks and launch-week social tests are poor substitutes for testing models on your own recurring tasks, especially when new frontier models are arriving about every 11 days, as Nathaniel Whittemore notes. For PMs, the relevance is straightforward: model choice now affects research, writing, planning, and stakeholder communication, but switching tools has real cost. The episode gives a repeatable way to benchmark models, tools, or full harnesses against the work you really do.

The Big Idea

Do not choose AI models from vendor benchmarks or launch hype. Build a small, repeatable benchmark from your own tasks, score the outputs blind, and make decisions using quality, speed, cost, and switching friction together.

The important shift is from asking "which model is best?" to asking "which model or setup is best for this kind of work, in my environment, at an acceptable cost?" That produces more useful answers, and the episode shows that the winners can differ sharply by task.

Key Insights

Public benchmarks miss reality

• Vendor benchmarks are limited for two reasons:
• many older benchmarks are already in training data, so they say less about real capability than they used to
• even a perfect benchmark may still fail to predict fit for your actual work
• Nathaniel Whittemore's point is that model selection is now often about fit and feel, not just leaderboard rank. A model that is state of the art on a public chart can still be worse for your writing style, negotiation tone, or planning workflow.
• PM takeaway: do not treat a benchmark win as evidence that a model will improve customer research synthesis, roadmap writing, or stakeholder communication inside your team.

Task choice sets the bar

• The most important step is choosing the tasks in your personal benchmark.
• Nufar Gaspar recommends 4 to 6 tasks that:
• reflect work you do often
• cover different kinds of work
• are important enough that better output would change your behavior
• are easy enough to judge that you can tell good from bad
• She also recommends adding at least one wishlist task. This is a task AI does poorly today, but one that could become newly viable after a model jump. That is how you notice when a new model unlocks work that used to sit on the sidelines.
• Her own benchmark included tasks like LinkedIn writing, pricing negotiation pushback, feature research, course content, and weekly planning. The point was not that those tasks are universal. The point was that they were decision-worthy for her.

Benchmark setups, not just models

• Nufar Gaspar explicitly broadens the benchmark target beyond raw models.
• Sometimes the thing you are testing is:
• a new model
• a new tool
• a model plus tool combination
• a harness or agent setup that includes memory, connectors, or web access
• This matters for PMs because many real workflows depend on the surrounding system, not only the model. A research assistant connected to internal docs is a different product experience from the same model in a blank chat window.
• She makes a specific caution for operations-heavy tasks. If a task depends on your full setup, an API-only benchmark may be misleading because it strips away the context and connectors that make the workflow useful.

Blind testing reduces bias

• Nufar Gaspar strongly recommends hiding model names before scoring.
• The reason is simple: people are biased toward familiar or favored tools, and that bias can overwhelm actual output quality.
• The process she used:
• run the same request against each candidate
• use a fresh chat for each run
• anonymize results as A, B, C, and so on
• score each output before revealing which model produced it
• She also recommends a one-line note alongside each score. A quick phrase like "too generic" or "best negotiation tone" is often more useful later than a bare number because it captures the reason behind the judgment.

One winner rarely exists

• Her own results did not produce one clear universal champion.
• Different models won different tasks:
• GPT models did best on pricing negotiation pushback
• Grok won weekly planning
• Opus 5.5 won the website design task
• This is the practical reason to stop asking for a single best model. The answer may be a roster, not a replacement. A PM may want one setup for writing, another for analysis, and another for planning or design work.
• Nufar Gaspar's personal takeaway was that she may use GPT more often, even though she came in biased toward Claude and Cursor plus Grok.

Speed and cost affect decisions

• Output quality is only one part of the decision.
• Nufar Gaspar frames the adoption decision as:
• switch
• split
• stay
• A model can win on quality and still lose the overall decision if:
• the cost is much higher
• latency is much worse
• company policy blocks it
• the gain is not big enough to justify changing habits
• Her example was Fable 5.1. It scored very well, but it was also far more expensive. GPT Sol had a similar average score for her, at materially lower cost. That is the kind of trade-off that should shape an adoption decision.

AI judges need supervision

• Nufar Gaspar uses an AI model as a secondary judge, not as the source of truth.
• Her cautions are important:
• use a judge from a different model family when possible, to reduce family bias
• check whether the judge's preferences actually match your own
• be careful with visual tasks, because an API judge may evaluate HTML or code rather than the rendered experience
• In her own run, the AI judge and her human ratings disagreed often. Her conclusion was not "AI judging is useless." It was "judge the judge first." If the judge is not correlated with your taste, discard it.

Mental Models & Frameworks

Five-step benchmark loop

• 1. Pick tasks: choose representative work, not generic demos.
• 2. Pick candidates: compare the current baseline against new models, tools, or harnesses.
• 3. Run blind tests: freeze the prompt, use fresh chats, and hide names.
• 4. Score outputs: combine a numeric score with a short explanation.
• 5. Decide adoption: choose switch, split, or stay based on quality, speed, cost, and constraints.

Use this when a new model release creates real adoption pressure. The episode's argument is that the loop matters more than any single benchmark result.

Balanced task portfolio

A good benchmark portfolio is not a random pile of prompts. It should mix frequent work, high-stakes work, and at least one aspirational use case. That gives you a benchmark that is stable enough to reuse, but still sensitive to model improvements that unlock new behavior.

Switch, split, stay

This is Nufar Gaspar's adoption framework after scoring:

• Switch: a new setup is clearly better and the gains justify the change.
• Split: different models deserve different roles in the workflow.
• Stay: the new option is not better enough to overcome cost, policy, speed, or habit friction.

This framework is useful because it stops PMs from turning every model release into a forced migration decision.

Judge the judge

Treat AI evaluation the same way you would treat any metric system.

• Check whether the evaluator matches the outcome you care about.
• If the evaluator disagrees with obvious human preference, fix or replace it.
• Do not trust visual evaluation if the judge only sees code rather than the rendered artifact.

This is a reusable model for any AI-assisted evaluation, not just model benchmarking.

Decision Principles

Principle: Start from real tasks

• When: choosing what to benchmark.
• Why: a benchmark only helps if it represents work that would actually change your day-to-day decisions. Generic prompts create interesting demos, but weak adoption guidance.

Principle: Keep a current baseline

• When: comparing a new model against launch hype.
• Why: without the model or setup you already use as a baseline, you cannot tell whether the new option is truly better or just different.

Principle: Test the full workflow

• When: evaluating operations, research, or agentic tasks that depend on memory, connectors, or internal context.
• Why: a stripped-down API comparison can understate the value of a harness, or overstate a raw model that only looks good in isolation.

Principle: Retest after big shifts

• When: a major model release changes speed, cost, or quality enough to reopen old use cases.
• Why: some tasks move from "not worth it" to "now viable" very quickly. Nufar Gaspar's weekly planning example is exactly the kind of task that belongs in a periodic retest.

Trade-offs & Nuance

Frontier gains versus stability

The frontier may improve often, but changing tools too often creates confusion and habit cost. Nufar Gaspar's answer is not "benchmark weekly no matter what." It is to run the process when there is a meaningful shift or on a cadence that matches your bandwidth.

Realism versus clean comparison

A clean benchmark wants identical prompts and isolated runs. Real work often depends on memory, tools, and context. The closer you move toward realism, the harder it is to maintain strict apples-to-apples comparison. PMs need to decide which matters more for the decision at hand.

Open source versus frontier

Nufar Gaspar does not frame this as ideology. The right answer depends on the task and the constraint.

• For table-stakes work like basic research or email drafting, cheaper or open models may be fully good enough.
• For more demanding tasks, frontier commercial models may still show a gap.
• Security, procurement, and company policy can matter as much as raw quality.

Speed changes perceived quality

After seeing ultra-fast demos, Nufar Gaspar noticed she cared more about latency than before. That is a useful reminder for PMs: speed is not only a technical metric. It changes whether a workflow feels usable enough to become a habit.

Better model, harder switching

Even if a model wins a benchmark, you may still stay put.

• Team habits have value.
• Existing subscriptions may hide the true token cost.
• Company rules may block the winner.
• A small gain may not justify the coordination cost of retraining people or rewriting prompts.

Common Mistakes

Mistake: Copy vendor benchmarks

Public benchmarks can help you notice candidates, but they are a poor substitute for testing against your own work. Using them as the final decision tool creates false confidence.

Mistake: Test generic prompts

If the tasks are too generic, the results will not be decision-worthy. Nufar Gaspar repeatedly comes back to this point: the benchmark should be strong enough that you would actually change behavior based on it.

Mistake: Reveal model names

If you score output after seeing the label, brand preference contaminates the test. That is especially risky when teams already have a favored vendor.

Mistake: Trust one run

Model outputs vary even with the same prompt and settings. For any important adoption decision, Nufar Gaspar recommends multiple runs per model so you are not overreacting to one lucky or unlucky sample.

Mistake: Ignore setup effects

A task that depends on web search, internal memory, or other tooling should not be judged as if the raw model did all the work. If the value lives in the harness, benchmark the harness.

Practical Application

Audit recurring AI tasks

List the tasks where AI already affects PM leverage. Good starting categories include writing, research synthesis, stakeholder communication, analysis, planning, and any design or prototyping work you do regularly. Pick the few where better output would actually change your workflow.

Write scoring rubrics first

For each benchmark task, define what good looks like before you compare outputs.

• name the audience or outcome
• specify obvious failure modes
• set measurable constraints when possible
• add a short scoring guide if you want to use an AI judge later

This makes the scoring less impressionistic and makes later reruns more comparable.

Run blinded comparisons locally

Run the same prompt across your baseline and a small set of challengers. Keep each run in a fresh chat or clean API call. Label outputs anonymously, then review them side by side before revealing the model names.

Log cost and latency

Create a simple table for each task with:

• human score
• one-line rationale
• response time
• estimated cost
• any policy or security blocker

This is the minimum data you need for a real adoption decision rather than a vibes-based one.

Keep a mixed model roster

Do not force one winner if the results do not support it. If one model consistently handles negotiation, another handles planning, and a third handles design or prototyping, keep that specialization explicit instead of pretending the stack should collapse to one tool.

Questions to Consider

• Which 4 to 6 recurring PM tasks would be important enough that better AI output would actually change how your team works?
• Where is your team currently trusting vendor benchmarks or launch-week social demos instead of testing models on stakeholder emails, research synthesis, planning, or other real product work?
• Which AI-assisted tasks in your workflow depend so heavily on internal context, memory, or connectors that a raw model comparison would miss the real source of value?
• If a new model were only 10 percent better on output quality but much faster, would that speed gain make it more likely to become part of daily PM habit?
• What is one wishlist task your team has mostly given up on because today's models are unreliable, and how would you notice if a new model made that task viable?

Bottom Line

The episode's core lesson is that model evaluation should look like product evaluation. Test on real tasks, blind the comparison, and decide with quality, speed, cost, and switching friction together.

A PM does not need the internet's best model. A PM needs the model or setup that reliably improves real work enough to justify changing behavior.

Tools & Products

People to Follow

Nufar Gaspar

Nufar Gaspar is the guest leading this Operator's Cut. Her contribution here is practical rather than theoretical: a reusable process for building a personal benchmark, plus a script and workflow for blind side-by-side comparison.

Nathaniel Whittemore

Nathaniel Whittemore, the host of The AI Daily Brief, frames the problem clearly. His main contribution in this episode is the warning that public benchmarks increasingly fail to predict personal fit, especially as models become good enough that differences show up in workflow and feel rather than only on public scoreboards.

Experiments to Try

Experiment: Blind benchmark sprint

• Hypothesis: a blinded comparison will produce different winners than your team's current vendor preferences.
• Change: benchmark one recurring PM task, such as customer research synthesis or stakeholder email drafting, across your baseline and two challengers.
• Test: anonymize outputs and have two or three teammates score them independently.
• Measure: compare blind winner, revealed winner, and score spread across raters.

Experiment: Wishlist task retest

• Hypothesis: a task that failed six months ago may now be viable with a new model or faster mode.
• Change: pick one abandoned AI use case, such as weekly planning or more nuanced negotiation drafting.
• Test: run the old prompt against today's baseline plus one frontier challenger.
• Measure: record whether the task now works in one shot, how many prompt repairs are needed, and whether the latency is low enough to make it habitual.