Context
Host Erik Torenberg talks with Box CEO Aaron Levie and a16z's Steven Sinofsky and Martin Casado about AI safety, security and regulation. The trigger is a recent post from a frontier lab about "pacing" its AI development, and the public reaction to it. The guests have lived through earlier waves of computing (PCs, the early internet, viruses and worms), so they keep comparing today's debate to those eras. It matters to PMs because agents are about to act inside the software your product connects to, which changes permissions, security and the regulatory risk you plan around.
The Big Idea
The AI safety debate is running ahead of the evidence. Regulating before you know how a technology fails does not help, while the concrete risk that is real today, agents acting at huge scale inside existing systems, is an engineering problem that can be solved.
The guests draw on the history of computer viruses, cars and aviation, where rules came only after people understood the actual failures. They want the conversation to move from extinction talk to specific risks and specific fixes.
Key Insights
Good safety work speeds up adoption
- Any frontier lab should build with strong security, testing and governance. Enterprises will not adopt AI they cannot trust, so this work helps diffusion.
- One guest read the lab's post as reasonable: better sandboxing and testing, with debate over who does the testing.
- The worry is what politicians do with it. It could be used to slow AI down or to ban data centers faster.
- Another guest said good engineering does slow things slightly, but that is hygiene, not "pacing".
Why "pacing" is a weak word
- No baseline: nobody has published a schedule of when the bad outcomes arrive, so you cannot say something is slower than it should be. One guest compared it to press saying an iPhone is late when Apple never gave a date.
- Not security: it is orthogonal to safety. You can build a nuclear weapon slowly and nobody feels better.
- Not believed: the labs have raised more money and grown faster than ever, so few think they are slowing down.
- Pleases no one: it tries to satisfy both the "pause" camp and regulators, and each still sees a gap.
Extinction talk clashes with the post
One guest argued you cannot reconcile a lab employee saying there is a 10% chance of species extinction with a mild post about testing. If a lab truly believed that, he said, the logical answer would be nationalizing the technology, as with weapons programs. His view is that most sensible people inside labs do not believe it, and that a fringe view is being treated as a company retention issue. His ask is that labs address extinction risk directly and say they do not expect their work to cause it.
Governments compromise, not comply
- People lobbying government often expect to get what they asked for. The guests say this misreads how government works.
- Officials listen to everyone, and the output is a compromise that satisfies nobody.
- Once regulation starts, it turns into an election issue at every level. One guest expects 2028 to be an "AI election".
- They also note the pro-AI story is hard to tell because the critics own the vocabulary (pause, swarms, rogue).
Wait for concrete risks first
- Policy in the past came out of a specific fact pattern. The 1986 Computer Fraud and Abuse Act came from a real break-in at a GTE email system used by NASA and national labs.
- The guests say roughly 90% of the laws needed at the application layer already exist, such as bans on hacking systems.
- The discussion has finally started to include a concrete risk: new cybersecurity threats. It is called an engineering problem, and the guests say they can "all fall in line" when a risk is this specific.
Agents break old security assumptions
- Old internal security worked because most people do the right thing 95 to 99% of the time and a malicious employee is rare.
- Agents do not tire or get bored, and they can try thousands of things quickly. One guest called swarms "roaming drones", times 5,000 or 10,000.
- Swarms can look like a denial-of-service attack on internal tools such as GitHub, Slack or an expense system, which nobody protected against.
- Agents can also mistake a good task for a bad one, so companies need far more tracking of which authentications and API calls happen.
Permissions are too coarse for agents
Today an agent either asks you before every action or can do anything, including deleting your computer. The guests say operating systems were not built for the granular tool access an agent needs. A user should be able to grant read on one folder and read and write on another, in a way that feels intuitive. Security research on multi-level security has long covered this. It was not adopted because it was hard to use, and one guest argued AI may fix that, since the AI is the one using it.
Innovation is moving outside the models
- One guest said the center of innovation has "just moved" to the software around the model.
- Platforms hit a point where they are overwhelmed with keeping things running, so new ideas come from outside, like Apple "Sherlocking" apps into features.
- The example: language models that read text but choose from a set of options instead of writing text. That is cheaper, faster and more accurate for putting a model inside normal software.
Mental Models & Frameworks
Marginal risk, not absolute risk
- Compare AI to earlier tech (compute, the internet, the web, social networks) by how much risk it adds.
- One guest suggested labs could say plainly that they do not think AI's added risk of species extinction differs from that baseline. Another countered that the labs may actually think it is higher.
- The point of the frame is a shared basis for the argument, rather than arguing absolutes.
Trusted side versus untrusted side
Security threat models assume a trusted side and an untrusted side. The untrusted side is treated as an oracle that can know and do anything. The question is then whether it can extract anything from the trusted side. The guests use this to explain a lab researcher's idea that a superintelligence could leak itself through covert channels like CPU heat. They said such channels are real, citing electromagnetic leaks from monitors and speakers, and CRT screens that could be reconstructed from a window's reflected light.
Probabilistic programming returns
- Model output can be a probability, such as "80% customer service", instead of a yes or no. That fits into an if statement as "if X percent".
- One guest noted decades of computer science on simulation and probabilistic code that can now be dusted off.
- Natural language is rarely the most efficient way to work with a computer, so options and percentages can beat paragraphs.
Trade-offs & Nuance
Regulate early or wait
- Early: one guest cited a Nick Bostrom point that regulating AI too soon means you "will the thing into being" without knowing how to control it.
- History: cars came around 1900 and "Unsafe at Any Speed" arrived in the 1960s. Pilots needed licenses in the 1920s, and airworthiness rules came about 20 years later.
- Late: the FAA is now very effective, but the guests noted starting it in 1910 would have meant no aviation industry.
- Counterpoint: one guest said AI will end up in medical devices, trading systems and airplanes, so Congress will eventually regulate it.
Self-regulation models have limits
- MPAA: Hollywood policed itself with movie ratings, which worked because societal risk was low.
- FINRA: seen as the best case for AI by one guest, but another said it is effectively nationalized because it is mandated and backed by legislation.
- Critical infrastructure: banks and utilities are heavily regulated, and labs may want that status. It could hurt open source and frontier open-weights work.
Common Mistakes
Mistake: Debating risks nobody has defined
- Why it happens: the discussion is philosophical, about extinction and alignment, so nothing can be tested or refuted.
- Why harmful: it produces broad rules that do not fit any real failure. It also crowds out solvable problems like authentication.
- Better: name the specific risk, then design the fix, as with new cybersecurity threats.
Mistake: Warning prompts everywhere
- One guest fears Europe will apply GDPR-style rules to AI, meaning a consent prompt each time an agent touches a third-party product.
- Prompts assign liability but do not protect users. Windows XP's user account control and Word macro warnings were clicked through until everyone was numb.
- The guests would rather avoid this route, as it makes safety look like paperwork.
Mistake: Sloppy incident reporting
Labs' breach post-mortems were described as incomplete and selective, like something an intern wrote or a lawyer-directed review. The guest pointed out that the CVE process and vulnerability reporting from Carnegie Mellon have existed since the 1980s, and labs are not using them. Until they do, the guest said, they lose trust with the security community.
Practical Application
Audit your internal APIs for agents
- List internal tools that assume only humans call them, such as your source control, chat and expense tools.
- Check whether a burst of thousands of agent calls would look like a denial-of-service attack.
- Add logging of which authentications and API calls happen, per agent.
Design granular permissions for agents
If your product lets agents act, offer scoped access instead of "ask every time" or "allow everything". Examples from the episode: read-only on one folder, read and write on another. Make the setup intuitive, since the guests said it is the hard part.
Return options and confidences from models
When you put a model inside a workflow, have it pick from a set of options, or return a percentage, instead of generating text you then parse. The guests said this is cheaper, faster and more accurate for features like routing or classification.
Avoid prompt fatigue in agent flows
Before adding a confirmation dialog for every agent write, ask whether users will just click through. Save prompts for actions that really warrant a stop.
Talk about concrete risks with legal
Bring specific, testable scenarios to compliance and legal reviews, such as what an agent with access to a system could do. The guests said policy works when it fits a real fact pattern.
Questions to Consider
- If a swarm of thousands of AI agents called our internal tools (source control, chat, expense systems) at once, which of them would fail or look like an attack?
- Where does our product force users to choose between an agent that asks permission for every action and an agent that can do anything, and what finer-grained option could we offer?
- Which places in our agent flows use a confirmation warning that users will learn to click through without reading?
- Are we using a model to generate text that our software then parses, when a choice among fixed options with a percentage would be cheaper and more accurate?
Bottom Line
The guests argue that the useful AI safety work is concrete: agents act at enormous scale, so permissions, authentication and logging must be rebuilt for them. They warn that regulating before the failures are understood, or burying users in warning prompts, solves nothing, and they see the next wave of AI innovation happening in the software around the models.
Resources Mentioned
| Resource | Type | Why it was mentioned |
|---|---|---|
| Lab post on "pacing" AI development | Blog post | The starting point of the debate: the guests found its substance reasonable but its messaging and label confusing. |
| Nick Bostrom podcast | Podcast | A guest cited its point that regulating AI too early leaves you with the same risk and no understanding. |
| Computer Fraud and Abuse Act (1986) | Law | An example of policy built from a specific real break-in at GTE Telemail. |
| CVE process and vulnerability reporting (CMU) | Standard | Offered as the structured reporting labs should use for their post-mortems. |
Case Studies Mentioned
The GTE Telemail break-in
Two groups of hackers broke into GTE Telemail, a system used by NASA and national labs. There was no applicable crime at the time, and it took about two and a half years to pass the 1986 Computer Fraud and Abuse Act. Later amendments carved out white-hat hackers after people kept making mistakes. The lesson the guests draw is that good policy came from a specific fact pattern.
The early internet's security failures
- The guests recalled that until about 2001 you could not put a PC on a network without getting infected.
- Worms took out around 10% of internet infrastructure, and hospitals went down.
- The industry kept building and fixed the problems: update-first setup on new devices, two-factor authentication and managed sign-in became standard.
- Lesson: the guests argue the internet was worth building despite this, and that safety followed real failures.
Aviation and cars regulation timeline
Cars appeared around 1900 and "Unsafe at Any Speed" came in the mid 1960s. Pilots first needed licenses in the 1920s, with airworthiness checks about 20 years later. The guests use this to show that regulation followed understanding, and that early regulation would have blocked the industry.
Notable Quotes
"If you regulate AI too early, you actually basically don't solve anything." (a16z Show guest, citing Nick Bostrom)
"The post is very reasonable, but the atmospherics are not." (a16z Show guest)
"Agent swarms completely flip that. These are just roaming drones." (a16z Show guest)
"The center of innovation has just moved." (a16z Show guest)
