Context

Claire Vo talks with Zach Lloyd, cofounder and CEO of Warp, an AI-native terminal that has expanded into "software factories," a cloud system that takes a request from Slack all the way through implementation, review, and a merged pull request. Lloyd was a principal engineer on Google Sheets before founding Warp, and he demos Warp's own internal factory (nicknamed Wilson) live, including the dashboard he wishes he'd had as an engineering leader. The episode matters to PMs and engineering leaders because it's a concrete look at what changes, and what doesn't, once an org moves past individual coding agents into a centralized, measured, agent-run development pipeline.

The Big Idea

A "software factory" is not a faster coding agent, it's a full, centralized, measured pipeline (Slack to Linear to GitHub to QA to merge) that makes AI-assisted development visible and improvable at the team level, and once you build that visibility, the data shows humans, not the agents, are now the throughput bottleneck.

Warp's own numbers make the point concretely: the time from kicking off a task to opening a pull request is 35 minutes, but the time from that PR to first human review is three and a half hours. The factory's real value isn't that agents write code faster, it's that centralizing and measuring the whole pipeline lets a team actually see and fix bottlenecks like that one.

Key Insights

A factory is a defined system, not a single agent

Lloyd defines a software factory as a genuine product concept: a set of repos, MCP servers, configuration, and multiple purpose-built agents (a code review agent, a triage agent, and so on), all defined in code rather than living in one person's individual coding-agent session. At Warp, a request tagged to their factory ("Wilson") in a public Slack channel triggers a full sequence: triage the request, open a Linear issue, implement the change, open a GitHub PR, run computer-use verification (the agent records a video of the feature actually working), and only then merge. That's meaningfully more than what a single coding agent session like Claude Code or Codex does on its own, which stops at writing and testing the code locally.

Work moves from private laptops to a public record

Before factories, an engineer would pull up a local coding agent, make a change, test it locally, then push to GitHub, all inside their own silo. Warp's factory instead starts most work in a public Slack channel, so anyone on the team can watch a task unfold, jump into the thread, or contribute, and the resulting ticket, PR, code review, and verification video are all visible together rather than scattered across separate private sessions. Work can also start automatically, not just from a person: Warp wires crash reports from Sentry directly into the factory so certain bugs get triaged and fixed without a human kicking anything off.

Human interactions per PR measures agent independence

Lloyd's team tracks a metric he calls human interactions per PR: every re-prompt in Slack, every comment left on a Linear ticket, and every correction made during code review, all counted together as one score. The instinct behind it is that humans are the bottleneck on raw production throughput even though they're also the creative force, so the fewer times a person has to steer or correct an agent's work, the higher the effective throughput. Lloyd compares it to a Tesla plant: the metric that matters is how often you have to stop the line, and Warp is trying to give engineering leaders exactly that view of how automated their pipeline really is.

PR-to-review lag is the real bottleneck, not the agent

Vo presses Lloyd on Warp's own numbers: task kickoff to opening a PR takes 35 minutes, but PR to first human review takes three and a half hours, a gap Lloyd concedes is "still the bottleneck." Warp still requires human code review on every PR, though they loosened one rule: it used to require a different person to review an agent's code than the one who prompted it, and now the same person can do both. Lloyd expects some share of PRs will eventually skip human review entirely once the team trusts specific categories of change enough, but that isn't happening yet.

Cost per PR is driven mainly by model choice

Warp centrally tracks cost per PR across the team and has already driven it down after a spike, and Lloyd's clear read is that model selection is the single biggest lever on that cost, more than context-management tuning, which he calls a secondary factor. He recommends actually testing different model configurations against your own real tasks rather than guessing, since the cost-quality tradeoff varies by task type.

Redundant tests reveal a common agent failure mode

Lloyd walks through a concrete failure pattern the factory's scoring system catches: agents write far more tests than a task needs, not to prevent regressions but just to exercise behavior along the way, leaving a bloated, unnecessary test suite. Because every factory task gets recorded, Warp built an LLM-as-judge scorer that samples completed runs and flags when a different agent than the one that did the work judges it to have added redundant tests, turning what used to be an invisible pattern into a trackable one.

Self-improvement lets the factory fix its own failure modes

Beyond scoring, Warp runs a second loop it calls self-improvement: an observer agent reviews a meaningful sample of failed runs (Lloyd says you need something like 20 to 25 to avoid overcorrecting on one-off cases), identifies a recurring failure pattern, and proposes a specific change to the factory's own agent definitions to prevent it going forward. Because the whole factory is defined in code, an agent can literally edit the instructions of another factory agent, and because that state is versioned, the team can freeze a configuration, rerun past tasks against a proposed change, and see whether it actually helped before adopting it.

Mental Models & Frameworks

The two factory personas: builder and manager

Lloyd frames factory adoption around two different needs. From the builder's seat, using the factory doesn't feel wildly different from using any local coding agent, except the work happens in public and gets automatically ticketed, reviewed, and QA'd rather than staying in one person's local setup. From the manager's or CTO's seat, the real payoff is centralized visibility across the whole team: automation rate, velocity, cost, and quality all become measurable in one place instead of being scattered across everyone's individual tool choices. Use this split to evaluate any "AI factory" pitch: ask whether it's actually solving the manager's visibility problem, or just repackaging a single builder's coding-agent workflow with extra branding.

Replay real tasks to choose a model configuration

Rather than guessing which model to route different work to, Warp replays real past factory tasks under different model configurations and scores the outcomes with the same LLM-as-judge scoring infrastructure it uses for live runs, producing a cost-versus-quality tradeoff chart specific to their own tasks (not a generic public benchmark like SWE-bench). Use it whenever you're deciding which model to route a category of task to: curate a batch of your own representative tasks, replay them under each candidate model, and let the resulting Pareto chart, not intuition, drive the routing decision.

Practical Application

Build session-level telemetry before optimizing anything

Lloyd and Vo both stress that without session-level, tool-call-level, and computer-use-level observability into every coding session (not just final PR outcomes), a team is guessing at where cost, quality, and speed actually break down. If your team is using local coding agents without this, the practical first step is aggregating that telemetry (even by shipping local session logs to storage and scoring them yourselves) before trying to tune models or workflows.

Track PR-to-review lag as your real bottleneck metric

Instead of only measuring how fast agents produce code, measure the gap between a PR opening and its first human review, the way Warp's own 35-minutes-to-3.5-hours numbers exposed their actual constraint. If that number is large relative to kickoff-to-PR time, the fix is a human-process change (faster review triage, risk-based auto-approval for low-risk PRs), not a faster or cheaper model.

Risk-score PRs to protect senior review time

Vo describes a pattern from her own "merge mommy" agent build: automatically risk-score every PR, auto-approve the low- and extra-low-risk tranche, and route medium-, high-, or outlier-risk PRs to mandatory human review. This clears the bottom of the review queue quickly while preserving human attention for the PRs that actually need it, reframing code review as an exercise in risk management rather than uniform inspection.

Sample enough failed runs before letting an agent self-correct

If you build a self-improvement loop like Warp's, don't let the observer agent act on a single failed run or a handful of them; Lloyd's rule of thumb is a real sample, on the order of 20 to 25 failed runs, before trusting a proposed fix to your agent definitions. A small sample risks overcorrecting on one unusual failure rather than fixing a genuine recurring pattern.

Bottom Line

The value of an AI software factory isn't that it writes code faster than a single coding agent, it's that centralizing the whole pipeline in code and measuring every run turns "are we getting more efficient?" into an answerable question, and at Warp that measurement has already pointed the finger back at human review time as the actual constraint on throughput.