See pricing
Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test?
How I AIAI

Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test?

Claire Vo scraps her planned Opus 5.5 review after Anthropic and OpenAI both ship new models the same morning, then blind scores them live across emails, PRDs, frontend, backend, agents, SVGs and video.

September 22, 2026 · 39 min listen · 8 min read
0:00
–:––

Context

Claire Vo runs How I AI and had already recorded a review of Anthropic's Opus 5.5. Then OpenAI shipped GPT-6 Sol and GPT-6 Luna the same morning, so she scrapped the plan and ran her benchmark live instead. She scored outputs blind, without knowing which model made them, across a much wider set of tasks than her earlier model episodes: email and productivity, frontend prototypes, backend coding, agent personality, long running research, SVG illustration, and video editing. The episode matters to PMs because it is a live look at how someone who evaluates AI models for a living actually builds and runs a comparison bench, not just a verdict on which model won.

The Big Idea

No single model won across every task. Opus 5.5 scored highest most consistently across the widest range of work, GPT-6 Astra and Sol were the ones Claire personally enjoyed using, and the split between "best on average" and "most loved" is itself the lesson.

Picking a model is not one decision. It depends on the task, and a PM's own taste for how a model communicates matters as much as raw output quality.

Key Insights

A blind bench beats gut feeling

Claire scores every output without knowing which model produced it, specifically because she admits her own bias: "I am always inclined to think that Astra or Soul will win because I love the Open AI models. It's just the heart wants what it wants." She writes her predictions down before the reveal, then compares them to the blind scores afterward. The point is to catch her own favoritism rather than assume it isn't there.

The bench grew to eight categories

Older How I AI model episodes focused mainly on PRDs and prototypes. This time Claire expanded the bench to messy notes to PRD, personal productivity and inbox triage in her own voice, frontend coding, backend coding and auditing, agentic voice and personality, long running research, SVG illustration, and video editing. Her reasoning: models got smarter and more agentic, so a two-task bench no longer reflects real usage.

Em dashes are a score killer

Across the email, agent personality, and productivity tests, Claire repeatedly penalized outputs for using em dashes, in one case giving a response a flat one specifically because "every single message has an M dash in it." This shows up as a real, repeated scoring criterion in her bench, not a one-off complaint, and it cost multiple models points across different task categories.

Narration changes how fast it feels

Claire found Opus 5.5 was not as fast as GPT-6 Sol in raw latency, but part of the gap was something else: Opus 5.5 stopped narrating its own work step by step, which made her unsure whether it was actually running. "When it doesn't narrate its work, then you get into this loop where you're like, are you actually working?" Sol, by contrast, narrated at what she felt was the right level, which made it feel faster even when the underlying speed difference was smaller.

Claude gets denser on harder tasks

On frontend prototypes, Claire noticed a pattern specific to Claude-family outputs: given a genuinely complex spec, like a container-ship scheduling platform, the model produced something "super dense and complex and really detailed," which she said is good for backend code but hard to use for prototyping because she couldn't tell what the interface was actually supposed to do. Calibrating a benchmark's task difficulty, she notes, changes which model looks best.

Claude won't skip the review gate

In the agent personality test, Claire noticed one model consistently told her it could not push straight to production without review, and identified it as a Claude model on that basis alone. When she checked whether the other models would let her skip review and deploy directly, none of them did either, so she didn't penalize the Claude model for it, but the moment shows a real, testable difference in how models around this generation talk about safety gates versus simply enforcing them.

OpenAI models won on character SVGs

Claire predicted Claude would win creative SVG illustration. The blind results reversed that: Astra and Sol produced the character illustrations she scored highest, including a plant-tracking app illustration that got a full five and "gave me a little spark of joy." She called this the one surprise that fully overturned her prediction going into the reveal.

The LLM judge disagreed with her

Claire also ran an LLM as judge over parts of the bench and compared its ranking to her own. The two disagreed outright: she personally liked Astra best, while the LLM judge ranked Fable highest and ranked Sol much lower than she did. She flags this as funny but real: an automated judge and a human evaluator can reward different things from the exact same outputs, so which one you trust changes your answer.

Barbie Bench still hasn't been beaten

Claire's running benchmark for 3D character rendering, which she calls Barbie Bench, asks a model to build a 3D Barbie fashion designer game. No frontier model has cleared it yet. Opus 5.5's attempt was "way better than it used to be" but still failed on hands, feet, and hair. She uses it as a standing reminder that visual and physical reasoning for the human form lags far behind text and code output, regardless of which model currently leads elsewhere.

Mental Models & Frameworks

Predict first, then blind-test

Claire's actual method: write down a prediction for which model will win each task category before testing, run every output blind so you don't know the source model, score each one on its own terms (a vibe score plus a short note), then reveal the source models and compare the blind results to your prediction. Use this whenever you are comparing AI tools or outputs for your own team, since it separates what a model is actually good at from which brand you already prefer.

Calibrate task difficulty first

Claire's frontend tests ranged from a simple wireframe to a "cuckoo bananas" complex logistics platform, and the model rankings shifted depending on which difficulty tier she was looking at. A model that looks best on an easy, one-shot prompt may not be the one that handles a genuinely complex, detailed spec well. Before trusting any model comparison, check whether the test tasks match the difficulty of the real work you intend to hand the model.

Practical Application

Build your own blind comparison bench

Before picking a default model for a task type, generate the same output from two or three models on that exact task, strip out any labels showing which model made which, and score them without knowing the source. This removes the brand preference Claire admits she has and forces a judgment based on the actual output.

Score for boring, high-frequency failures first

Add em dashes, verbosity, and slop tells (like decorative circles or literal filler phrases) as scoring criteria before you evaluate creative quality. Claire's bench repeatedly penalizes these small, easy-to-spot issues, and they turn out to separate models more consistently than harder-to-judge creative differences.

Match the model to the task

Claire's own results split by task: she preferred GPT-6 Sol and Astra for personal writing and productivity, Opus 5.5 for the broadest range of agentic and frontend work, and OpenAI's models unexpectedly for character SVGs. Before defaulting your team to one model for everything, test your actual task mix and let each task keep its own winner.

Spot check any LLM judge

If you use an LLM as judge to score AI outputs at scale, run a parallel human review on a sample and compare rankings, the way Claire did. A mismatch, like her judge favoring Fable while she favored Astra, tells you the automated judge is rewarding something different from what your actual users or reviewers care about.

Questions to Consider

  • If you compared two AI tools blind, without knowing which output came from which model, would your preference change from what you currently assume?
  • Does your team's evaluation of AI-generated work check for small tells like excessive em dashes or filler phrases, or only for whether the output looks impressive?
  • Are you using one AI model as a default for every task, or have you actually tested whether a different model wins on your specific task types, like writing versus frontend code versus agent-style back and forth?
  • If you use an automated judge to score AI outputs, have you checked whether its rankings match what a human reviewer on your team would actually prefer?

Bottom Line

Claire Vo's blind, live comparison of Opus 5.5, GPT-6 Sol, and GPT-6 Astra found no universal winner: Opus 5.5 scored most consistently well across the broadest range of tasks, while Astra and Sol were the models she personally enjoyed using most, and an LLM judge disagreed with her rankings entirely. The real takeaway is that model choice should be tested per task, not decided by brand loyalty or a single averaged score.

Tools & Products

Tool / ProductWhat it doesWhy it was mentioned
Claude Opus 5.5Anthropic's updated frontier-adjacent modelThe main subject of the planned review that got folded into the live blind bench.
GPT-6 Sol and LunaOpenAI's newly released models, cheaper and faster tiersLaunched the same morning as Opus 5.5, prompting the live comparison.
GPT-6 AstraAn OpenAI model Claire already uses dailyScored highest on personal preference for writing, productivity, and character SVGs.
CodexOpenAI's coding agent harnessThe harness Claire says she moved to full time before Opus 5.5 won her back.

Notable Quotes

"When it doesn't narrate its work, then you get into this loop where you're like, are you actually working?" (Claire Vo)

"It's just the heart wants what it wants." (Claire Vo)

"I am always inclined to think that Astra or Soul will win because I love the Open AI models." (Claire Vo)

AI PM course

Everyone hears the same episodes.
Few can do what they describe.

Start for free