Context
Matt Garman is the CEO of AWS. He was the first general manager of EC2 and interned on the early AWS project in 2005. In this conversation, a16z's Raghu Raghuram asks how AI is changing the cloud itself. Topics include what AI-native startups need, how scarce GPU capacity gets allocated, and what changes when agents, not people, write code and run infrastructure. It matters to PMs because it shows how a very large platform decides who to serve first, and how it redesigns products for a new kind of user.
The Big Idea
When agents become the main users of a platform, you do not just add agent features. You rethink the building blocks (accounts, permissions, databases, sandboxes) around how an agent works, while keeping the trust that enterprises need.
Garman says most of AWS's core pieces were already in good shape. The work is making them easy for agents to use, adding a few brand-new building blocks, and solving the trust gap that stops enterprises from letting agents run on their own.
Key Insights
Startups are the learning engine
- What: AWS has aimed at startups since its first internal analysis in 2005. Garman estimates that 30 to 40% of AWS revenue comes from companies that were once startups.
- Why it matters: startups push the platform harder than banks, healthcare firms or governments do. They ask for capabilities that larger customers will want about five years later.
- Change over time: startups now begin far bigger. A team with an idea can be valued at a billion dollars with $200 million raised, where $10 million used to be typical.
What startups still need
Even with a huge head start in funding, startups still ask the same questions: how will this scale, how do I handle security, how do I set up identity permissions that work past three employees. Garman says this is why many prefer AWS to a newer GPU-only cloud. They need the security and capabilities around the compute, not just the chips.
Agents want a different cloud
- Agents get blocked by tail latency, which people rarely notice. Garman says agents care about the 99.9th percentile S3 latency in a way people do not.
- Agents need a well-defined API they can easily work across.
- Garman says AWS has been working on things like starting a database in three seconds.
- AWS mostly tuned existing services such as S3 for both people and agents. It also built agent-specific services, including AgentCore and Bedrock, and a beta called AWS Context that builds a layer so agents can find data across Aurora, S3 and other stores.
Cut new account setup cost
Setting up an AWS account used to mean defining VPCs and IAM roles, which matter at scale but block a quick start. Garman says a new account can now be created with a Gmail sign-up and no credit card, and is running in under 30 seconds. Those settings are handled by default behind the scenes. The key product choice is that it is a real account, so a team can add an organization and fine-tune later with no migration.
Agents break old durability assumptions
Many agents create a database, do a little work, and delete it. Five nines of durability is overbuilt for that. But AWS does not want a non-durable option either, because nobody knows up front whether a database will live for hours or years. So the team is trying to build something that is cheap to create and discard, yet can grow into a durable production database.
New building blocks for agents
Garman listed new pieces that agents need, not just old ones used differently:
- Compute sandboxes: isolated spaces for agent work. AWS's Firecracker micro VMs, invented about 10 years ago for another purpose, suit this well because they start fast and have a strong security boundary with little overhead.
- Gateways.
- Agent permissions: short, time-boxed access for a single task, not "Raghu's permissions" to do anything he can do. Permissions can be fine grained down to part of a tool.
Capacity is split on purpose
- Demand is so high that AWS could sell every GPU and accelerator to the biggest frontier labs. Garman says it chooses not to, to keep growing the whole ecosystem.
- Large customers such as Anthropic, OpenAI and Meta get a large share, and big enterprises like Salesforce and JPMorgan are also supported.
- Startups get reserved capacity too. AWS says yes in some form to about 60% of startup requests, though sometimes later, in another region, or in a different configuration.
- AWS recently announced it will buy 2 million NVIDIA GPUs over the next couple of years.
There is no single bottleneck
Garman recalled reading the book The Goal in undergrad. Its lesson is that there is never one constraint, only the latest one. Today AWS tracks power, memory, TSMC capacity, HBM, networking parts, disk drives, SSDs, construction labor, and capital. Location matters too: there may be plenty of power in Indonesia and not enough in Germany. He says capacity is not fully fungible across regions.
Planning now spans decades
- Fifteen years ago, a power company would just add a few megawatts. Now AWS brings its own power and pays for solar, nuclear and other projects.
- Server planning used to cover a few quarters. Now AWS plans multiple years out, thinking about what it needs for 2026, 2027 and 2028.
- Garman frames this as value AWS gives customers, since a customer cannot plan its own power or memory supply in 2028.
Mental Models & Frameworks
Greenfield, not replicate
Garman's advice to enterprises building agents. The first instinct is to copy a human workflow: Bob does steps one to five, the agent does steps one to five, then Bob checks. He says that is not the model to want. Step back and ask how an agent would solve the problem differently. It can run in a massively parallel way and try 50 different things, so rethink how a computer would solve it versus a human.
Build, then prove, then scale
Garman's account of how AWS makes silicon bets. Each step proved value before the next bigger one:
- Network offload card: moved network virtualization off the server for near bare metal performance.
- Storage next: a company with ARM cores on an offload card helped move storage virtualization too. AWS acquired that team, Annapurna.
- Bare metal servers: a bigger card took all virtualization off, which improved performance, utilization and security isolation.
- Graviton: the ARM cores became a server chip. AWS saw the slope of ARM performance and power and drove the ecosystem.
- Trainium: about five or six years ago AWS built its first AI chip.
Decision Principles
Principle: Don't ration to the biggest buyer
- When: capacity is scarce and a few large customers could take all of it.
- Why: Garman says a healthy ecosystem is better for AWS long term, and today's startups are the enterprises of tomorrow. He also notes AWS has single-digit customer concentration at most, where some neoclouds have 30 to 60% in one or two customers.
Principle: Protect customer data first
- When: deciding how fast to ship AI services.
- Why: Garman says AWS took heat roughly three years ago for being slow on AI. It chose to build Bedrock so prompts and data never go back to the model provider. Now, he says, as companies move from proofs of concept to production, most land on Bedrock.
Trade-offs & Nuance
Easy start versus deep control
Letting a user skip VPC and IAM setup helps speed, but those controls matter at scale. AWS's answer is to hide them by default in a real account rather than offer a simplified account to migrate out of later.
Autonomy versus trust
Enterprises want agents that act on their own, but Garman says nervousness about agents deleting a production database may be holding them back "maybe appropriately". Most enterprise agents today are simple and still have people in the loop. The path to autonomy is guardrails, permissions, sandboxing and clear rules on where a human stays in the loop.
Frontier models versus open weights
Garman agrees enterprise data is its owner's most valuable asset. Some customers want to post-train an open weights model on their own data and distill it into a better, cheaper model. That depends on having a good eval to prove it works, and he notes most customers do this on SageMaker today. Bedrock offers both open and proprietary models.
Spend for a possible bubble
Raghu asked about bubble risk. Garman's reply: enterprises he talks to are seeing positive ROI, so workloads like core compute, storage and inference will not vanish. He compares it to the internet, where many companies failed but durable ones like Google and Amazon did well. He admits not every billion-dollar startup will make it.
Common Mistakes
Mistake: Copying human steps into agents
Many enterprises first build an agent that repeats the same workflow a person did, then have a person check it. This captures little value. The better approach is to rethink the whole job around what an agent can do, such as parallel attempts.
Mistake: No evals before autonomy
Garman says enterprises need help with evals, labeled data, constant testing loops, goal-seeking, and production measurement to catch drift. He says he does not know if anyone is great at this yet. Without them, there is no basis to trust an agent with a decision.
Practical Application
Make your product agent-ready
- Do: check whether an agent can sign up, authenticate and use your product through a clear API with no human setup steps.
- Then: measure how fast an agent can create and use a resource, and watch tail latency, since agents get blocked by slow outliers.
- Why it works: this is what Garman says AWS did, from three-second databases to 30-second account setup.
Scope agent permissions narrowly
Write down what a single agent task needs and give only that, for a limited time. Garman's contrast is giving an agent a person's full permissions versus fine-grained, time-boxed ones.
Redo a workflow from scratch
Pick one human workflow your team wants to automate. Instead of listing the steps for an agent to copy, ask what you would do if you could try 50 approaches in parallel and pick the best. Design the agent's version from the outcome back.
Teach customers to own evals
Garman's forward-deployed engineer model is to work with a ready customer for about 45 days. In that time, show them how to write an eval and label data, then leave. If you sell to enterprises, copy that goal: leave customers able to run it themselves, not dependent on consultants.
Try smaller agent-first pods
Garman describes AWS experiments with teams of three or four, not ten, because agents let them build fast and then move on. Test a smaller pod on one feature and see how well it can also maintain what it built.
Questions to Consider
- If an AI agent, not a person, were the main user of your product, which sign-up step would fail first?
- Which permission does your product give by default that an agent should only get for one short task?
- What part of your agent workflows just copies the steps a person used to do, and what would a from-scratch design look like?
- Which of your agent features would your customers trust enough to run with no human check, and what evidence (evals, guardrails) would they need to see first?
Bottom Line
Agents are becoming the main users of cloud platforms, so AWS is tuning old services, adding new building blocks like sandboxes and agent permissions, and making sign-up near instant. The harder problem is trust: enterprises will not let agents run fully on their own until guardrails and evals make that safe.
Resources Mentioned
| Resource | Type | Why it was mentioned |
|---|---|---|
| The Goal | Book | Garman read it in undergrad and uses its point that there is always a next bottleneck to explain why AWS tracks every part of its supply chain. |
Tools & Products
| Tool / Product | What it does | Why it was mentioned |
|---|---|---|
| Bedrock | AWS service for running AI models | Keeps customer data inside their environment so model providers never see prompts. |
| AgentCore | AWS building blocks for building agents | Example of a service made for agents, usable with any model. |
| AWS Context | Beta layer that helps agents find data across stores | Lets agents work across Aurora, S3 and other data. |
| Firecracker | Micro VM technology | Fast to start with a strong security boundary, so many sandbox startups use it. |
| Graviton | AWS ARM server chip | Garman says it is about 20% cheaper with 20% better performance, and is the easiest way to lower a bill. |
| Trainium | AWS AI chip | Used for training and a lot of Bedrock inference, with a third generation in market. |
| SageMaker | AWS model building platform | Where customers post-train and host open weights models. |
| Continuum | AWS security service | Uses powerful models to find and prioritize vulnerabilities in a customer's setup. |
| Amazon Quick | Agent tool rolled out to Amazon employees | HR and finance teams built agents that save weeks of work. |
People to Follow
Matt Garman
CEO of AWS and a long-time leader there, starting with EC2. He talks here about agents, chips and capacity from the vantage point of running the largest cloud.
Raghu Raghuram
a16z partner who hosts the conversation and asks about GPUs, startups and enterprise adoption.
Notable Quotes
"The startups today are the enterprises of tomorrow." (Matt Garman)
"There's never one constraint. There's always just the latest constraint." (Matt Garman)
"You don't just want to give it Raghu's permissions and let it go do whatever you can do." (Matt Garman)
