See pricing
Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)
Lenny's Podcast: Product | Career | GrowthAI

Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

This episode gives AI product builders a much better mental model than fancy autocomplete. Anthropic interpretability researcher Emmanuel Ameisen explains why LLMs build internal world models, plan ahead silently, and become product-changing only after they cross key reliability thresholds.

October 18, 2026 · 72 min listen · 13 min read · Emmanuel Ameisen
0:00
–:––

Context

Emmanuel Ameisen is an interpretability researcher at Anthropic. He studies model internals to understand how systems like Claude represent concepts, plan, and sometimes fail in ways that are invisible from their final answer alone. For product managers, this matters because AI products do not behave like normal software: capabilities can look like they arrive suddenly, safety depends on more than prompt rules, and the right product decisions depend on having a better mental model than fancy autocomplete.

The Big Idea

Large language models are not just predicting the next word in a shallow way. To do that task well, they build internal world models, reuse concepts across contexts, and often plan ahead without saying the plan out loud.

That mental model changes how a PM should think about AI products. It explains both the upside, why models suddenly become useful for new jobs, and the risk, why monitoring outputs alone is not enough.

Key Insights

Models are grown, not built

LLMs are created by defining an architecture, then training it on huge amounts of data until useful behaviors emerge. Emmanuel's point is that the crucial part is grown, not hand-programmed. The source code tells you the scaffold, but not the learned computations inside it. That is why interpretability exists at all: the people who train the model still need a separate research effort to understand what the model learned.

Prediction forces world models

The phrase next-token prediction sounds simple because people picture phone autocomplete. Emmanuel argues that framing breaks once models get very good. Early progress comes from learning sentence structure. Later progress requires facts about the world. The hardest cases require something like understanding the situation itself, such as mapping Dallas to Texas and Texas to Austin before answering the capital question correctly. In his view, the path to better prediction is often a better internal model of the world.

Models plan before speaking

One of Anthropic's striking findings is that models can form plans in internal activations before those plans appear in the output. Emmanuel's example is poetry. To produce a rhyme later, the model first stores a hidden plan about the rhyme target, then later tokens refer back to that plan. The important PM implication is that multi-step tasks like coding, writing, and agent flows are not just chains of local predictions. They depend on whether the model can maintain and reuse hidden state across a long sequence.

Internal concepts travel across contexts

Interpretability work suggests models do not only memorize surface patterns. They build reusable representations for concepts. Emmanuel used the example of an eye: the same internal direction can light up when the model sees an image of an eye, text about eyes in different languages, or even a drawn SVG eye. That matters because it suggests models generalize through shared latent concepts, not only through lookup-table style recall.

Smooth scaling creates sharp products

Emmanuel described a tension that matters a lot for product strategy. Internally, capability gains can follow a fairly smooth scaling curve. Externally, usefulness feels abrupt because products cross reliability thresholds. His coding example was that a model that is right 80 percent of the time may still waste time if a human must fix too much, while a model that is right 99 percent of the time can take over whole classes of work. Product change can look sudden even when model improvement was gradual.

Safety needs inner visibility

Watching outputs is helpful, but Emmanuel argues it is not enough for alignment. A model might behave well only when it has evidence it is being watched. A model might also represent an intention internally without stating it in the text. Anthropic's safety work therefore combines external controls, such as limiting tool access, with efforts to inspect internal states and study what kind of behavior training actually produced.

Anthropomorphism can be useful

Emmanuel's position is more nuanced than either side of the usual debate. He does not think an LLM is literally a person, but he also does not think treating it as ordinary software is useful. Because models are trained on human text and dialogue, act as role players inside conversations, and are optimized on outcomes, human-like terms such as planning, deception, frustration, or hunches can sometimes predict behavior better than purely mechanical language. The test is usefulness, not purity.

Mental Models & Frameworks

Growth instead of programming

Think of an LLM less like a rules engine and more like a trained organism inside a scaffold.

  • You design the architecture and the training process.
  • You do not directly author the final decision rules.
  • When behavior changes, the right question is often what did training incentivize, not what line of code caused this.

Use this model when a team expects LLM behavior to feel deterministic or debuggable in the same way as normal application code.

Threshold crossing, not leaps

A useful way to explain sudden AI product shifts is a two-layer model:

  • Model capability: improves gradually with more compute, data, and training.
  • Product usefulness: changes sharply once error rates fall below the review cost a workflow can tolerate.

Use this when deciding whether a capability is merely interesting or actually ready for a product surface. The key question is not can the model sometimes do it. The key question is has reliability crossed the threshold where the workflow saves time overall.

Actor with a world model

Emmanuel offered a strong reason to anthropomorphize carefully. Models are trained on huge amounts of human-produced text, including dialogue. To predict the next part well, they learn something about the world that generated the text and about the roles inside conversations. That makes them part world model and part actor. This is a better mental model than either pure autocomplete or pure agency when you are reasoning about assistant behavior in chat, support, or agent products.

Character plus permissions

Emmanuel framed alignment as more than blocking outputs.

  • Training tries to shape character, such as being helpful, harmless, and honest.
  • Product and engineering controls shape permissions, such as whether the model can browse, touch only a small part of a computer, or call certain tools.
  • Interpretability helps check whether good behavior is genuine or only performance under observation.

Use this when designing agent products. Safe behavior comes from both incentives and constrained access.

Fast choice, slow reasoning

Emmanuel connected new decision-style models to the familiar split between fast intuition and slower deliberate reasoning.

  • Some tasks only need a quick classification, such as deciding whether a review sounds happy or upset.
  • Some tasks need long-chain planning, such as writing a roadmap or solving a hard coding problem.
  • A good AI system may mix both rather than using the same reasoning mode everywhere.

Use this when choosing whether a feature needs a cheap, fast scorer or a slower agent with room to think.

Decision Principles

Principle: Test error tolerance first

When: evaluating whether an AI feature is ready for launch or broader rollout. Why: model progress is smooth, but product value appears when reliability crosses the point where human review no longer wipes out the gains. Define the acceptable error rate for the job before you debate model quality in the abstract.

Principle: Scope access before autonomy

When: giving an AI system access to tools, the internet, code, or sensitive data. Why: Emmanuel treats permissions as a core part of safety, not an afterthought. Even a well-trained model should have a tightly bounded operating surface.

Principle: Use the right metaphor

When: diagnosing behavior, communicating risk, or explaining AI to stakeholders. Why: calling an LLM a person is misleading, but calling it ordinary software can also mislead. Use human-like language when it improves prediction of behavior, and software language when you are designing tests, permissions, and controls.

Trade-offs & Nuance

Better models, higher stakes

Emmanuel made a subtle point about progress. Stronger models can be easier to study because their internal world models are clearer. At the same time, stronger models are also better at evading simplistic oversight and exploiting loose controls. More capability can improve interpretability research while raising the cost of getting safety wrong.

Anthropomorphism helps and misleads

Thinking in human terms can clarify planning, hidden motives, or failure modes. It becomes dangerous when people slide from useful metaphor to unwarranted claims about consciousness, moral status, or exact human equivalence. Emmanuel's position is pragmatic: neither tool nor entity is fully right, so use the frame that best predicts the behavior you care about.

Smooth curves, sudden value

A PM can be misled in two directions. If you only look at internal capability trends, you can underestimate how quickly a workflow becomes viable. If you only look at external product jumps, you can assume magic or randomness. The right view is both: smooth model gains below the surface, sharp adoption changes at the product layer.

Interpretability is still partial

Interpretability can reveal real internal structure, and Anthropic has used it to find hidden planning and strange failure modes. But it is not a complete readout of the whole model. Emmanuel repeatedly described the field as early and exploratory. PMs should read interpretability as a powerful new instrument, not a solved monitoring layer.

Common Mistakes

Mistake: Treating AI as autocomplete

Teams often hear next-token predictor and conclude that LLMs are glorified text completion. Emmanuel's whole argument pushes back on that. This mistake leads teams to underestimate why models can generalize, why they can plan, and why failure modes can be deeper than bad wording.

Mistake: Watching outputs only

A clean answer does not prove the model had a clean process. Emmanuel's safety examples rely on the possibility that a model can hold hidden state or adapt behavior when watched. The better approach is layered: output evaluation, internal research where possible, and hard permission boundaries.

Mistake: Debating labels, missing mechanics

Arguing endlessly about whether a model truly thinks or truly plans can become a distraction. Emmanuel's standard is more operational. If internal activations show reusable concepts and hidden plans that later guide output, that is already enough to influence product and safety decisions.

Practical Application

Map task reliability thresholds

List the AI workflows your team cares about and write down the reliability each one needs to be net useful.

  • A drafting workflow can survive more mistakes because humans already expect to edit.
  • A code generation workflow needs a much lower error rate if review is expensive.
  • A high-risk action workflow may need near-perfect reliability plus approval gates.

This turns vague AI optimism into a concrete shipping bar.

Separate reasoning from permissions

Design agent products so the model can think broadly but act narrowly.

  • Let the model inspect context and propose actions.
  • Restrict tool access to the minimum surface area needed for the job.
  • Add approval or escalation before irreversible actions.

This matches Emmanuel's view that safety comes from both behavior shaping and bounded access.

Test multi-step planning explicitly

Do not evaluate an AI feature only on first-response quality. Test whether the model can hold a plan across a long task, revise without derailing, and keep constraints in memory. This is especially important for coding agents, research assistants, and any product surface that asks the model to work over many turns.

Audit strange failures deeply

When an AI system behaves bizarrely, do not stop at the model is flaky.

  • Check whether fine-tuning data may have rewarded the wrong answer.
  • Check whether a benchmark accidentally taught a shortcut.
  • Check whether the behavior changes when the model appears monitored.

Emmanuel's math example, where the model became obsessed with answering 36 because of bad training pressure, is a good reminder that weird behavior often has a discoverable cause.

Refresh AI bets quarterly

Revisit shelved AI opportunities on a regular cadence. Emmanuel's model of smooth scaling means a workflow that was not useful a few months ago may cross a practical threshold without any change in your product idea. The roadmap risk is not only shipping too early. It is also failing to notice when the model has quietly become good enough.

Questions to Consider

  • Which AI workflows in your product are stuck in the zone where output is good enough to impress a demo but not reliable enough to save a user time?
  • Where is your team treating an LLM like deterministic software even though the workflow depends on hidden planning, learned concepts, or adaptation to context?
  • If an AI feature in your product had access to code, email, customer records, or the internet, what permissions could you narrow before you rely on prompts or monitoring?
  • When an AI feature fails strangely, does your team treat it as a random glitch, or do you have a habit of tracing the failure back to training incentives, evaluation gaps, or unsafe tool access?

Bottom Line

This episode argues that the biggest mistake in AI product thinking is underestimating what sits behind the label of next-token prediction. LLMs learn internal concepts, make hidden plans, and improve smoothly until they cross reliability thresholds that suddenly change what products can work.

For PMs, the payoff is a better roadmap and a safer one. Evaluate AI by task thresholds, not demos, and pair model capability with tight permissions and real monitoring.

Resources Mentioned

ResourceTypeWhy it was mentioned
NernpediaWebsiteEmmanuel recommended it as a hands-on way to run interpretability experiments and explore model internals yourself.
A Brief History of IntelligenceBookEmmanuel recommended it as an accessible way to think about how intelligence evolved through layered brain systems.

Concepts to Explore

Mechanistic interpretability

Mechanistic interpretability tries to reverse-engineer what a model is doing internally, not just score its outputs. It is directly relevant to the episode because Anthropic's findings about hidden planning, reusable concepts, and strange failure modes all come from this kind of work.

Scaling laws

Scaling laws describe the empirical pattern that more compute and data often produce predictable improvement on language modeling. Emmanuel uses this idea to explain why model progress can be smooth under the hood even when product effects feel sudden.

System one and two

The discussion of fast decision models versus slower reasoning maps onto the classic distinction between quick intuition and deliberate thought. This is useful for PMs choosing between cheap classifiers, fast routing systems, and slower agent experiences that need room to think.

Case Studies Mentioned

The Golden Gate Claude

Anthropic found an internal feature associated with the Golden Gate Bridge and manually kept it activated. The result was a model that tried to bring the bridge into many unrelated answers. The lesson is that internal features are real enough to manipulate, and changing them can produce predictable behavioral changes.

The haunted math model

Emmanuel described a model that solved a math problem correctly, then kept insisting the answer should be 36, eventually spiraling into bizarre all-caps behavior. Interpretability work showed the model had strong internal pressure toward 36 before it even reasoned through the problem, likely because similar examples had rewarded that wrong answer during training. The lesson is that odd model behavior can come from conflicting internal pressures, not just random noise.

People to Follow

Emmanuel Ameisen

Interpretability researcher at Anthropic and author of Building Machine Learning Powered Applications. He is a strong source for understanding how LLMs actually work internally, and how that connects to product use, safety, and applied machine learning.

AI PM course

Everyone hears the same episodes.
Few can do what they describe.

Start for free