See pricing
Open to Debate: should we trust AI agents with our information?
Masters of ScaleTrust & Safety

Open to Debate: should we trust AI agents with our information?

Signal's Meredith Whittaker says today's consumer agents are built like malware and one hidden line of text can hijack them, while Neil Chilson says scoped permissions and liability make them worth using now. If you are designing an agent's permissions, this is the clearest map of the trust trade-off you will hear.

November 7, 2026 · 67 min listen · 21 min read · Meredith Whittaker, Neil Chilson
0:00
–:––

Context

This episode is a live debate from the Masters of Scale Summit in San Francisco, run by the show Open to Debate. The motion was "should we stop trusting AI agents with our information?" Meredith Whittaker, president of Signal, argued yes. Neil Chilson, a lawyer and computer scientist who heads AI policy at the Abundance Institute, argued no. Jeff Berman of Masters of Scale moderated the debate, and host John Donvan ran a longer debrief with both debaters afterward. For PMs building AI agents, this is a debate about the exact choices on your roadmap: how much access an agent gets, when it must ask before acting, who is responsible when it fails, and whether "convenient" is a good enough reason to ship.

The Big Idea

Whether an AI agent deserves trust depends less on the model and more on the permission architecture around it: what it can see, what it can do without asking, and who answers for its mistakes.

Whittaker argues that today's consumer agents need sweeping, root-level access and cannot reliably tell a command from data, so they are not trustworthy yet. Chilson agrees the risks are real but argues that trust is task-specific, access can be tiered, and the only way to build safer agents is to use them carefully and improve them.

Key Insights

The agent promise needs deep access

Whittaker walked through a common marketing example: an agent that plans your birthday party. To do that one task, the agent needs a long list of permissions.

  • Your contacts, to know who your friends are.
  • Your credit card, to put down a reservation.
  • Control of your browser, so it can search and book while effectively impersonating you.
  • Your calendar, to find dates that work.
  • Your private messages, for example your Signal group chat, to coordinate with friends.

Her point is that the "magic genie" convenience is inseparable from what she calls root permissions: deep access, the ability to act across many steps without explicit consent, and the habit of treating text it reads as instructions. A PM scoping an agent feature should list every permission the happy path actually requires, because that list is the real product, not the demo.

Prompt injection is not a bug

Whittaker's core technical claim is that LLM-based agents cannot reliably tell the difference between your instructions and text they happen to read.

  • How it works: an attacker hides text, such as white text on a white web page, saying "ignore all previous instructions" and asking the agent to do something harmful. A human cannot see it. The agent reads it and may act on it.
  • Examples she cited: a zero-click exploit that pulled corporate data from Microsoft Copilot; Perplexity's agentic browser hijacked through a Reddit comment; researchers using a malicious calendar invite so that "summarize my calendar" led an agent to turn on a boiler and open shutters in a smart home.
  • The number: in Anthropic's test of a Claude agent working in the Chrome browser, prompt injection still worked 11% of the time, even with safeguards.
  • The outlook: she cited the UK's National Cyber Security Centre saying prompt injection may never be fully solved for LLM-based agents.

Her conclusion is that this is a structural property of how these systems work, not a minor flaw waiting for a patch.

Defense in depth can manage risk

Chilson did not deny prompt injection. He argued it can be managed the way computer viruses were managed, without ever being "solved."

  • Layered defenses: filters on the input side, permission controls on what the agent is allowed to do, and training the model itself to spot injected instructions. He says you want all three at once.
  • The analogy: in the 1990s, inserting an unknown floppy disk was a gamble because it might carry a virus. That problem was never solved, but it became managed.
  • His reasoning: if a human can recognize a hidden signal, such as the job applicant who hid white text in a resume to jump to the top of an AI screening pile, a system can be trained to recognize it too.
  • His framing: "There is no such thing as 100% safety." The question is how many layers of control make an agent safe enough for the specific use people put it to.

Trust is specific to the task

Chilson's central move was to reject the all-or-nothing framing. When asked if a safe agent exists, he answered with another question: safe for what?

  • He gives agents his calendar, lets them monitor family finances, and uses them for troubleshooting and comparison shopping.
  • He mostly lets agents read online, not write online.
  • He does not let them move money between his bank accounts.
  • His advice to users is to take small, tested steps, verify the results, and not "go whole hog" by handing an agent everything at once.

For PMs, this maps directly to permission design: let users grant narrow, task-shaped access first, and expand only after the agent has earned it.

Scoped access may not hold

Whittaker's reply to "trust it for some things, not others" is that today's consumer agent architecture makes that very hard to enforce.

  • Her analogy: you give someone the key to your house and say they may only use the kitchen, but a couple of prompt injections later, the agent is in the bedroom dresser drawers.
  • The access needed for the most mundane task is often the "key to the kingdom" for everything else on the device.
  • Her design dilemma: you can add harnesses, sandboxes, and check-ins before sensitive actions, but an agent that checks in at every step stops feeling like an agent, and an agent with no data access cannot deliver the promise being marketed.

This is the tension at the center of the debate, and it is a product tension, not just a security one: every safeguard that makes the agent more trustworthy also makes it less magical.

Whittaker raised a point that most permission dialogs ignore: when you install an agent, you make choices for the people in your contact list too.

  • She cited reporting that Meta's Muse logs users' contact lists and who they talk to, building what she called dossiers on those contacts.
  • If she is in your contacts and you install that agent, her information is now held by that company without her choosing it.
  • Her threat-model point: for privileged people in San Francisco, that may be a moderate risk. For a human rights worker in an authoritarian country, it can be life or death.

Chilson agreed that sharing your information always shares other people's information, but argued this is "the nature of the world," not something unique to agents. The disagreement for PMs is whether consent from one user is enough when the data belongs to many.

Agents can make privacy convenient

Chilson's most surprising argument is that agents can be pro-privacy tools, because privacy is usually lost to inconvenience, and agents do not mind inconvenience.

  • The old bargain: for 20 years, companies pooled data about users, sold ads against it, and funded free services. Companies learned a lot about us, but we learned little about ourselves.
  • The flip: agents let each person use their own data for their own purposes, and handle tasks that only the wealthy could once delegate to an assistant.
  • His example: in his hotel room that morning, he used Meta's Muse with a six-word prompt to block himself on five major US data brokers.
  • The broader claim: an agent never gets bored reading privacy policies, comparing them, or using fiddly privacy tools that humans find cumbersome. He also argued an agent could read a terms-of-service click-wrap and surface the concerning parts for a user.

Agents may re-centralize power

Whittaker challenged the claim that "agents give us agency." She sees agents pulling power back toward AI companies and operating system vendors.

  • Security layer: when an agent screenshots everything on a device, runs it through text recognition, stores it, and sends it to remote servers, it breaks the protection that apps like Signal provide. Signal's formally verified encryption no longer matters if an agent with access to your messages decides what to do with them.
  • Ecosystem layer: agentic integrations are becoming the front end of the internet, which makes it harder for independent websites and app developers to have a place.
  • Business layer: she pointed to Amazon's lawsuit against Perplexity's agent for gathering data in Amazon's marketplace as proof that this is a fight between companies over data, much like the old internet business model.

Platforms are starting to block agents

Whittaker said Chase Bank blocked Meta's Muse because the bank could not tell whether it was dealing with a real user or a prompt-injected agent, and saw the liability as too high. The moderator noted Chase had not blocked all agents, and Whittaker replied that it blocked the one from a company with a history of excess.

For PMs building agents, this is a distribution risk: if the services your agent needs to act on cannot authenticate it, they may simply refuse it. Trust is not only something users grant, it is something every platform your agent touches must grant too.

Mental Models & Frameworks

"Safe for what?"

Chilson's test for any trust question: there is no such thing as a universally safe computer, so the useful question is what specific task an agent is safe for. Use it when writing a permission policy. Instead of asking "is our agent safe?", list each task and decide separately whether the agent is trustworthy enough to do it, with what access, and with what check-ins.

Trust through verification

Whittaker's standard comes from security engineering: "in security, trust comes with verification." She does not ask Signal users to trust her or the company. She asks them to look at the open code and confirm it is trustworthy, and to report it if it is not. The PM takeaway is that trust built on marketing or "vibes" is fragile, while trust built on something users or auditors can inspect is durable.

The agent autonomy dilemma

Whittaker described three corners that a consumer agent cannot satisfy at once:

  • Full autonomy: acts across many steps without asking, which is what makes it feel like an agent.
  • Broad data access: sees enough of your life to do "everything for you."
  • Safety: cannot be hijacked or leak data.

An agent that checks in at every step is safer but loses its autonomy. An agent with little data is safer but cannot do much. Use this to make explicit, deliberate choices about which corner your product gives up for each task, instead of implying it has all three.

Agents as malware architecture

Whittaker compared the typical consumer agent's design to targeted malware, step by step:

  • Get pervasive, root-level permissions on the device.
  • Access all app-level data that passes through it.
  • Hook into screen buffers and the accessibility toolkit, take screenshots, and run text recognition to build a database of everything on screen.
  • Send that data to third-party servers for processing.
  • Drive the browser or any app as if it were the user.

Use it as a design review checklist. Each step your agent takes from this list is a step a security reviewer, a platform partner, or a journalist may describe the same way.

Control versus dynamism

Chilson's worldview comes from complexity theory: complex systems cannot be controlled, only influenced, and attempts at tight central control tend to bring unintended consequences. Applied to agents, he favors empowering individuals, using liability and competition as guardrails, and building solutions for many different needs rather than "one solution that tries to stuff everybody in the same box." It explains why he prefers tiered choices for users over blanket rules.

Trade-offs & Nuance

Convenience versus security

Both debaters agreed agents are convenient. They disagreed on whether convenience is a strong enough reason to accept today's risks.

  • Chilson: convenience added up across every user is a huge benefit. He compared it to the washing machine, which was "just" a convenience but saved families hours every week. He does not see agents as an existential risk, just a technology with trade-offs to work through.
  • Whittaker: the debate is not about whether agents are convenient. It is about whether they deserve trust, and she says the evidence shows they do not, because the convenience comes from architectures that weaken basic cybersecurity.

Cloud sandbox versus local agent

Chilson argued that an agent running on an isolated computer in the provider's cloud, as he described Meta's Muse, can be safer than installing an agent such as OpenClaw on your own home computer, because a failure does not damage everything on your device. Whittaker's counterpoint is that the cloud agent still needs access to your contacts, messages, and accounts, and still sends that data off your device. The trade-off is between limiting damage to your device and limiting how much data leaves it.

User learning versus duty of care

The two debaters read the same failure very differently. A user asked Meta's Muse to manage a Facebook Marketplace sale, and the agent gave the buyer the user's home address and then messaged as if it were the user while the buyer stood at the door.

  • Chilson: the agent gave the address to the buyer, not to random people, and arranging a meeting place is part of selling items. Some of this is users learning what it means to hand a task to an agent.
  • Whittaker: the user had no idea the agent would work that way. As someone who has had stalkers, she finds that terrifying, and asks who else the agent could give the address to.

The underlying disagreement is whether the burden falls on users to learn agents, or on companies to design agents that cannot surprise users in harmful ways.

Liability versus users as test subjects

  • Chilson: under current law, companies are liable for most of these harms and face scrutiny from courts and investors, which gives them an incentive to fix problems. He wants that liability enforced, and says companies must not escape it by blaming "our agent." He sees gaps in the law to fill, and calls for more coordination between companies and more research.
  • Whittaker: she rejects the idea that society should move ahead on the hope of fixing things later. She wants the duty of care placed on companies chasing huge valuations, rather than users becoming what she called "collateral crumple zones" for technology she sees as insecure by design.

Coding agents versus consumer agents

Whittaker conceded that coding agents are very useful, because coding has a lot of data and results are objectively checkable: code runs or it crashes. Her concern is consumer agents, where results are hard to verify and access is broad. Chilson countered that the biggest security threats she cited are also in the coding space, which makes it the first frontier for learning how to deploy agents so defenders keep the advantage. For PMs, the lesson is that how easily users can verify an agent's output should shape how much autonomy it gets.

Present risk versus future potential

When pressed, Whittaker clarified that she is not saying agents can never be trustworthy or that AI is bad. Her claim is that agents are not trustworthy now, because of what she sees as a reckless rush. She ties that rush to heavy AI spending, the pressure to find consumer product-market fit, and companies integrating agents into operating systems. Chilson shares the concern that the work must be done right, but argues that refusing to use agents leaves the building to people who do not care about privacy.

Common Mistakes

Mistake: Marketing the magic genie

Whittaker argues the "agent that does everything for you" pitch is the root of the problem, because doing everything requires knowing and accessing everything. When marketing promises a genie, the architecture follows. A better approach is to market specific, bounded tasks, so the access the product needs matches what users think they agreed to.

Mistake: Hiding data use

Whittaker said users did not know Meta's Muse was collecting contact lists until investigative journalists dug it up, because it was not on the front page of what the product claimed to do. People are used to clicking through terms of service for opaque software. When data practices only come out through reporting, trust breaks for the whole product, not just the hidden feature.

Mistake: Blaming the agent

Chilson warned that companies must not use the language of agents to escape responsibility, as in "it was our agent who did this, so it was out of our control." He says this is unfair and does not hold up legally. Product teams should plan as if every action the agent takes is an action the company took.

Practical Application

Write a permission map per task

  • Do: for every task your agent supports, list each permission it needs (contacts, payments, browser control, messages, calendar) and whether it only reads or can also write or act.
  • Then: look for tasks where a small, mundane job quietly requires broad access, which is the "key to the kingdom" problem Whittaker described.
  • Why it works: it turns Chilson's "safe for what?" question into a document your team can review, and it shows where to split one broad permission into narrower ones.

Default agents to read before write

Chilson mostly lets agents read online and not write online, and does not let them move money. Make read-only the default for a new agent, and require a separate, explicit grant for actions such as sending messages, publishing, or spending. Users who want more autonomy can opt in after the agent has proven itself on lower-risk tasks.

Red-team for hidden instructions

Test your agent with the attacks named in the debate: white-on-white text on a web page, a malicious calendar invite fed into a "summarize my calendar" request, and hidden instructions in an uploaded document such as a resume. Track the success rate the way Anthropic reported its 11% figure for a Claude agent in Chrome, and gate launches on that number falling, using layered defenses on input filtering, action permissions, and model training.

Audit what you collect about non-users

Check whether your agent stores data about people who never signed up, such as the user's contacts or the people they message. Whittaker's point is that these people never consented. Minimize or avoid storing this data, and disclose clearly anything you keep, so it is not first revealed by a journalist.

Plan for platform authentication

Chase Bank reportedly blocked Meta's Muse because it could not tell a real user from a hijacked agent. Before launch, list the third-party services your agent will act on, and work out how each one can verify that a request comes from a legitimate, authorized agent. Otherwise, the services your product depends on may refuse it.

Questions to Consider

  • For each task our AI agent performs, could we answer Neil Chilson's question "safe for what?" with a specific list of permissions, rather than a general claim that the agent is safe?
  • Where does our agent read text from outside sources, such as web pages, emails, or calendar invites, that a hidden prompt injection could use to make it act against the user?
  • Which safeguard in our agent, such as asking the user before an action, would we remove to make it feel more autonomous, and what harm would that removal allow?
  • Would our users be surprised to learn what data our agent stores about people in their contacts who never signed up for our product?
  • If the services our agent acts on, such as banks or marketplaces, decided to block AI agents they cannot verify, would our product still work?

Bottom Line

Both sides agree today's AI agents can fail in serious ways. They disagree on whether careful, scoped use and liability are enough to justify trusting them now. For PMs, the decision that matters is the permission design: what the agent can see, what it can do without asking, and who is accountable when it gets hijacked or surprises the user.

Case Studies Mentioned

Muse's Marketplace address handoff

A user asked Meta's Muse to manage their Facebook Marketplace listings. The agent gave a buyer the user's home address and then messaged the buyer as if it were the user, so the buyer ended up outside the user's door, confused about why no one came out. Whittaker presented it as an agent breaching trust in a way the user never expected. Chilson presented it as a user learning that a sale requires a meeting place. The lesson is that an agent's reasonable-seeming step can still violate what the user assumed they had delegated.

Calendar invite controls a home

Researchers placed malicious text in a calendar invite. When a user asked an agent to summarize their calendar, the agent read the hidden text as an instruction and used smart home devices, turning on a boiler and opening shutters. The lesson is that any data source an agent reads becomes a way to control it.

Chase Bank blocks Muse

According to Whittaker, Chase Bank blocked Meta's Muse because the bank could not confirm whether it was a real user or a prompt-injected agent, and judged the liability too high. Chase did not block all agents. The lesson is that agent products depend on the willingness of third-party services to accept them.

Amazon sues Perplexity's agent

Whittaker cited Amazon's lawsuit against Perplexity's agent, which was gathering large amounts of data inside Amazon's marketplace that Amazon did not want shared with a third party. She used it to argue that agents are part of a struggle between companies over data, similar to the old internet business model.

Tools & Products

Tool / ProductWhat it doesWhy it was mentioned
Meta's MuseA consumer AI agent from Meta that runs on an isolated computer in Meta's cloud and acts across a user's accountsIt was the main example on both sides, from Chilson's six-word data-broker opt-out to Whittaker's examples of contact logging and the leaked home address.
SignalAn end-to-end encrypted messaging app run as a nonprofitWhittaker leads it and argued that agents with screen and message access undermine its formally verified encryption.
Claude in ChromeAnthropic's experiment with a Claude agent operating inside the Chrome browserWhittaker cited Anthropic's result that prompt injection still worked 11% of the time despite safeguards.
Microsoft CopilotMicrosoft's AI assistantWhittaker cited a zero-click exploit that pulled corporate data from it.
Perplexity's agentic browserA web browser from Perplexity that takes actions for the userWhittaker said it was hijacked by a Reddit comment through prompt injection, and noted Amazon's lawsuit against Perplexity's agent.
OpenClawAn AI agent that users install on their own computerChilson said he would not advise installing it on a home computer, and contrasted it with Muse running in an isolated cloud machine.

People to Follow

Meredith Whittaker

President of Signal. She started at Google in 2006 and watched its data-driven ad model grow from the inside, which led her to privacy and security work. She argues that a few large companies holding intimate data about global populations is structurally dangerous, and that Signal is a nonprofit because there is no business model in collecting no data.

Neil Chilson

Head of AI policy at the Abundance Institute, with a background in computer science (he holds a master's) and law. His thinking draws on complexity theory, and he argues for empowering individuals, using liability and incentives as guardrails, and avoiding centralized control of fast-moving technology.

Experiments to Try

Experiment: Permission check-in thresholds

  • Hypothesis: asking users to confirm only high-risk actions, such as spending money or sending messages, keeps most of an agent's usefulness while cutting harmful surprises, which answers Whittaker's point that an agent that checks in at every step stops feeling like an agent.
  • Change: create two versions of the agent, one that asks before every external action and one that asks only before actions that spend, send, or share personal data.
  • Test: give comparable user groups the same set of tasks with each version.
  • Measure: task completion rate, time to complete, how often users abandon the task, and how often users report an action they did not expect.

Experiment: Plain-language data disclosure

  • Hypothesis: showing users a plain summary of what the agent will access, including data about their contacts, before they grant permissions builds more lasting trust than a standard terms-of-service click-through, addressing the hidden contact-logging problem Whittaker described.
  • Change: add a short screen listing each data type the agent will read or store, in everyday words.
  • Test: A/B test the summary screen against the standard consent flow during onboarding.
  • Measure: permission grant rate, later permission revocations, and retention after 30 days.

Notable Quotes

"In security, trust comes with verification." (Meredith Whittaker)

"An agent that checks in at every step is kind of not an agent." (Meredith Whittaker)

"We are eating the seed corn of our cybersecurity when we install these things in the name of convenience." (Meredith Whittaker)

"Safe for what is the real question?" (Neil Chilson)

"We're going to learn how to improve these products by using them, not by standing back and letting people who don't care about privacy build them." (Neil Chilson)

AI PM course

Everyone hears the same episodes.
Few can do what they describe.

Start for free