See pricing
Grok vs. Dot vs. Muse vs. Claude Duet: which one crushes the competition?
How I AIAI

Grok vs. Dot vs. Muse vs. Claude Duet: which one crushes the competition?

Claire Vo runs four personal agents across nine real tasks and shows that the winner was shaped by far more than model quality. For PMs, the episode is a practical lesson in how to evaluate AI products: connectors, speed, interface design, and safety defaults changed outcomes as much as raw intelligence.

October 28, 2026 · 35 min listen · 13 min read
0:00
–:––

Context

Claire Vo runs a nine-task benchmark across Grok Bot, Dot, Muse, and Claude Duet, using real work and life tasks instead of one-off prompts. The setup matters because it tests agents as products, not just as models: each one has different connectors, memory, permission defaults, and ways of presenting results. For PMs evaluating AI tools, that is the useful lesson. The benchmark is less about crowning a universal winner and more about showing what actually changes outcomes in day-to-day use.

The Big Idea

The benchmark shows that a personal agent is a product, not just a model. Connector coverage, latency, permissioning, and output format changed results as much as raw intelligence did.

That is why the overall winner, Grok Bot, was not also the clear winner on every important task, and why Claire still preferred OpenAI and Anthropic tools for direct coding work. The episode is most useful as a PM lesson in evaluation: judge the whole workflow, not the brand or the base model alone.

Key Insights

Product fit beat raw models

Claire's final recommendation was segmented by job, not by vendor:

  • For work breadth, she would pick Grok Bot.
  • For life and personal tasks, she would pick Muse.
  • For coding, she still trusted OpenAI and Anthropic tools most, but not necessarily their personal-agent shells.

That distinction matters. A strong underlying model did not guarantee the best end-user agent experience. Claude Duet, for example, did the best launch video and one of the best bug-fix evaluations, but still finished last overall because it was slow, cautious, and weaker across the full task set.

Connectors decided the leaderboard

A large share of the benchmark came down to tool access, not pure reasoning.

  • Grok and Dot completed all nine tasks.
  • Muse and Claude Duet completed only seven of nine tasks.
  • Muse missed some work tasks because it was not connected to Slack, GitHub, and other business tools.
  • Claude Duet failed the short-form video task because it would not download the video.

This is a core PM takeaway for any internal AI evaluation. If one tool cannot reach the systems where work happens, the user experiences that failure as a product failure, even if the model underneath is strong.

Speed changed the winner

Claire repeatedly treated speed as part of product quality, not a cosmetic extra.

  • Muse was the fastest on four of the nine tasks.
  • Grok was the fastest on drafting a Slack reply and fixing a bug.
  • Claude Duet was consistently the slowest and often felt stalled.

The homework example makes the point clearly. All agents got the physics answer right, but Muse won because it returned the answer almost instantly. For lightweight, high-frequency tasks, fast enough often beats slightly better.

Presentation changed trust

The same underlying answer became more or less useful depending on how the agent packaged it.

  • Grok's inbox triage stood out because it returned a clean document with checkboxes and structure, not just a chat blob.
  • Muse's family newspaper and short-form video felt polished and easy to use, even without much steering.
  • Dot often completed tasks but presented them in a flat or awkward format, which made the result feel worse than its raw capability suggested.

For PMs, this is a reminder that interface design is part of intelligence. A good artifact reduces follow-up work. A bad artifact makes a capable system feel dumb.

Safety defaults shaped utility

Claude Duet asked for permission four times, while the other agents asked only once each. Claire read that as a product choice, not just a model quirk.

For a personal agent, there is a real balance to strike:

  • Too little caution creates risk.
  • Too much caution turns the agent into a bottleneck.

Claire's reaction was that Claude Duet landed too far on the conservative side. The result was not just slower execution. It also made the agent feel less helpful in routine tasks like shopping or general assistant work.

Benchmarks need real humility

Claire was explicit that this was not a perfectly controlled study.

  • The judging was not blind.
  • She used different operator agents: Claude ran Dot, while Codex ran the other agents.
  • She gave human nudges when computer use broke.
  • Prompts were intentionally short, which reduced prompt-engineering effects but also meant some tasks depended more on default behavior.
  • Muse was under-connected for business workflows, which likely hurt its score.

That does not make the benchmark useless. It means the results are best read as directional product insight, not as a universal ranking.

Mental Models & Frameworks

Separate model from agent

When judging an AI product, split the stack into distinct layers:

  • Model quality: can it reason, write, code, and plan well?
  • Tool access: can it reach Slack, GitHub, Dropbox, email, Figma, or shopping sites?
  • Product shell: is the interface fast, clear, and easy to inspect?
  • Permissioning: does it act at the right level of initiative?
  • Output design: does it return a useful artifact or just text?

This episode is a strong example of why this split matters. Claude and Dot looked strong on coding-related tasks, but that did not make them the best overall personal agents.

Use a task portfolio

Claire did not benchmark agents with trivia questions. She used a portfolio of real tasks that covered different modes of work:

  • communication
  • media creation
  • coding
  • design
  • triage
  • family planning
  • explanation
  • shopping
  • launch marketing

That is a better benchmark shape for PMs because it tests breadth, not just one benchmark-friendly capability. A product that looks great on one class of task can still fail in the workflow mix users actually need.

Equalize access before judging

Claire tried to baseline all agents to roughly the same connectors and skills. She even used browser automation to check access parity. The fact that Muse still lacked some work tools became one of the clearest lessons in the episode.

Use this model whenever you compare AI products internally:

  • First, audit connector parity.
  • Then, run the benchmark.
  • Finally, separate capability failures from access failures.

Otherwise, the comparison mixes product setup mistakes with true product differences.

Judge artifacts, not completion

A binary pass-fail score hides too much. Claire's live scoring implicitly used a richer rubric:

  • Did the agent finish?
  • How fast was it?
  • How much steering did it need?
  • Did the result match her voice or brand?
  • Was the format pleasant to review and act on?
  • Did the agent surface anything extra, like another bug?

That is a more realistic PM evaluation model than asking only whether the agent produced an answer.

Decision Principles

Principle: Test full workflows

When: evaluating an AI tool for real adoption rather than demo value.

Why: single prompts hide the product work that matters most. This benchmark surfaced differences in tool use, downloading files, navigating apps, formatting outputs, and handling permissions. Those are exactly the parts that determine whether a PM or operator can trust the system day to day.

Principle: Match agent to job

When: choosing a default AI tool for a team or a personal stack.

Why: the best overall agent was not the best at every kind of work. Claire's own conclusion was segmented: Grok for broad work tasks, Muse for personal life tasks, and OpenAI or Anthropic tools for direct coding workflows.

Principle: Separate shell from model

When: a product team is deciding whether to buy an agent product, use a model directly, or build its own workflow.

Why: this benchmark showed that an excellent model can sit inside a weak agent experience. If the shell is slow, over-cautious, or poorly connected, users will still judge the product as bad.

Trade-offs & Nuance

Breadth versus specialization

Grok Bot won because it was consistently good across many categories. Muse, however, often felt better on personal and creative tasks.

That is a classic product trade-off:

  • A broad agent can become the default for more workflows.
  • A specialized agent can deliver a much more delightful experience in narrower domains.

A PM choosing a team standard may value breadth. A PM choosing a personal stack may prefer specialization.

Speed versus caution

Claude Duet's slower pace and higher permission count likely reflect a safer product stance. The downside was a much weaker feeling of momentum.

This only becomes more important as agents take action instead of just answering questions. More safety can reduce risk, but it can also reduce utility enough that users switch tools.

Fairness versus realism

Claire tried to make the benchmark fair by normalizing prompts and tool access. At the same time, real users do not experience products in a lab. They experience them with uneven connectors, built-up memories, and idiosyncratic setup.

So there are two valid ways to benchmark:

  • a fair comparison of capability under similar conditions
  • a realistic comparison of the product as users actually have it configured

This episode sits somewhere in between, which is why the caveats matter.

Model strength versus product quality

Claude Duet and Dot were still Claire's preferred options for direct coding work, yet neither emerged as the best personal agent. That is a useful warning for PMs evaluating AI vendors.

A strong model can still lose if:

  • it is blocked from tools users need
  • it asks for permission too often
  • it formats results poorly
  • it moves too slowly for lightweight tasks

Common Mistakes

Mistake: Uneven connector access

It is easy to think you are benchmarking intelligence when you are actually benchmarking setup quality. Muse looked weaker on some business tasks because it lacked Slack and GitHub access, not because it necessarily lacked the reasoning ability.

The better approach is to log missing connectors before the test and call out which failures are product failures versus configuration failures.

Mistake: Scoring only answers

Several tasks showed that correct output was not enough. The shopping task, inbox task, and homework task all depended heavily on speed, interface quality, and how actionable the result felt.

If a PM scores only final correctness, they will miss the user-facing qualities that drive adoption.

Mistake: Confusing agent and model

Claire's clearest nuance was that she still wanted OpenAI and Anthropic for coding, but not Dot or Claude Duet as her default personal agents. Teams often collapse those into one decision.

The better question is two-part:

  • Which model or toolchain is best for the core capability?
  • Which agent product is best for the workflow around that capability?

Practical Application

Build a task deck

Create a benchmark with 6 to 10 tasks your team actually repeats. Mix different task types so the comparison reveals where each agent breaks.

A PM-friendly task deck could include:

  • drafting a stakeholder reply in Slack or email
  • triaging a support or bug report
  • summarizing customer calls into product themes
  • editing a design file or slide
  • turning notes into a launch asset
  • organizing inbox or calendar follow-ups

Keep prompts short so you are testing default product behavior, not prompt-writing skill.

Audit connectors before tests

Before you compare agents, make a simple checklist of access to the systems that matter: Slack, calendar, email, docs, GitHub, Linear, Figma, and any internal knowledge base.

Then mark three things for each agent:

  • connected and working
  • connected but limited
  • unavailable

This prevents a weak setup from being mistaken for a weak product.

Score speed and format

When you run an evaluation, capture more than a yes or no result.

Track at least these fields:

  • time to completion
  • number of permission asks
  • number of human nudges
  • quality of final artifact
  • ease of acting on the output

This mirrors the way Claire actually judged the benchmark and gives you a much more useful decision record.

Set defaults by domain

Do not force one AI tool to be the answer for every use case. Claire's own stack was segmented by domain, and many PM teams should do the same.

For example:

  • one default for broad work assistant tasks
  • one default for personal or consumer-style tasks
  • direct coding tools for engineering-heavy workflows

That approach reduces frustration and stops teams from judging a tool on jobs it was never built to do well.

Test permission posture

Include at least one task in your benchmark where the agent has to take action, not just answer a question. Count how often it pauses for approval and ask whether those interruptions feel appropriately safe or simply obstructive.

This is especially important if your team is evaluating agents for execution, not just brainstorming.

Questions to Consider

  • Which recurring PM workflows in your team would expose meaningful differences in connector coverage, such as bug triage in GitHub, design edits in Figma, or inbox follow-up from customer calls?
  • Are you currently evaluating AI tools on isolated prompts when your real work depends on multi-step workflows across Slack, docs, code, and calendar?
  • In your own context, what matters more for adoption: maximum correctness, lower latency, or fewer permission interruptions?
  • If a coding model produces the best fix but its agent shell is slow or awkward, should your team adopt the agent, use the model directly, or build a thinner internal wrapper?
  • Which of your AI product evaluations are being distorted by uneven setup, such as one tool having richer memory or better connector access than another?

Bottom Line

The headline is not just that Grok Bot won. The real lesson is that AI agents should be evaluated as full products, where connectors, speed, safety defaults, and output design can matter as much as the underlying model. For PMs, the right move is to benchmark agents on your own recurring workflows and choose them by job type, not by brand reputation alone.

Tools & Products

Tool / ProductWhat it doesWhy it was mentioned
Grok BotA personal and work-oriented AI agent from xAIIt won the benchmark overall and stood out on Figma work, inbox triage, and shopping.
DotOpenAI's personal-agent productIt performed reasonably well on voice-matching and coding-adjacent tasks but was weaker on creative outputs and artifact quality.
MuseMeta's personal agentIt was the fastest on many tasks and strongest on personal or creative work, but lacked some business connectors in Claire's setup.
Claude DuetAnthropic's personal-agent productIt produced the best launch video and a strong bug fix, but was slow and asked for permission much more often than the others.
CodexOpenAI's coding and computer-use toolClaire used Codex to operate most of the benchmark and still preferred OpenAI and Anthropic tools directly for coding workflows.
FigmaDesign and collaboration softwareThe thumbnail task highlighted that Grok was especially strong at operating inside Figma.
SentryError monitoring and bug-reporting softwareThe bug-fix task tested how well each agent could investigate and resolve a real engineering issue.

Experiments to Try

Experiment: Permission interrupt audit

  • Hypothesis: a more cautious permission model lowers perceived usefulness even when raw output quality is high.
  • Change: run the same 5 to 8 action-oriented tasks across two or three agents.
  • Test: count every approval request, every manual nudge, and total time to completion.
  • Measure: compare completion rate, trust rating from reviewers, and how often users abandon the workflow before the agent finishes.

Experiment: Access normalization retest

  • Hypothesis: connector parity explains more performance variance on internal workflows than model quality alone.
  • Change: fully connect two competing agents to the same Slack, email, calendar, GitHub, and design systems.
  • Test: rerun the exact same workflow benchmark before and after connector parity.
  • Measure: track the change in task completion rate, number of dead ends, and reviewer quality scores.

Experiment: Output format comparison

  • Hypothesis: users prefer an agent that returns a structured artifact over an agent that returns similar content in plain chat.
  • Change: take one repeated workflow, such as inbox triage or customer-feedback synthesis, and compare agents that return checklists, documents, or templates against agents that return free-form text.
  • Test: ask reviewers to act on each output without editing it first.
  • Measure: time to action, amount of manual cleanup, and reviewer preference.
AI PM course

Everyone hears the same episodes.
Few can do what they describe.

Start for free