Short answer: an AI agent is a language model running in a loop. On each turn it looks at the goal and what it knows so far, picks a tool, reads the result, and decides whether to act again, stop, or hand off to a human. That gives five parts every PM should be able to name: tools, planning, state, termination and escalation. AllthingsPM is an AI PM course and PM interview prep platform, and its course has a full lesson on exactly this anatomy, The anatomy of an agent, inside a chapter built from real PM job postings.
This guide explains each part in plain terms, shows what can go wrong in each one, and tells you what a PM actually owns.
What is an AI agent, in one sentence?
The two labs that ship the most agents define it almost the same way. Anthropic calls agents "systems where LLMs dynamically direct their own processes and tool usage," and contrasts them with workflows, where "LLMs and tools are orchestrated through predefined code paths" [1]. OpenAI's guide says agents "are systems that independently accomplish tasks on your behalf" [3].
The key word in both is direct. In a workflow, an engineer wrote the path. In an agent, the model chooses the next step at run time. That single difference is why agents need the extra parts below: once the model picks the path, you need rules for what it can touch, how it remembers, when it stops, and when it asks for help.
If you are still deciding whether your feature should be an agent at all, read our guide to agents vs workflows first. The course rule is simple: if you can write the path, it is a workflow.
What are the five parts of an agent architecture?
Here is the anatomy at a glance, with what each part does, the failure it causes when it is weak, and what the PM owns.
| Part | What it does | Typical failure | What the PM owns |
|---|---|---|---|
| Tools | Lets the model read data and take actions outside itself | Wrong tool picked, huge outputs flood the context, cryptic errors | The tool list, each tool's contract and its risk level |
| Planning | Breaks the goal into steps and picks the next one | Loops, skipped steps, confident wrong plans | Task scope and the success definition |
| State (memory) | Carries what happened so far, in context and in storage | Forgets earlier facts, repeats work, leaks data across users | What is remembered, for how long, and for whom |
| Termination | Decides when the run is done or must stop | Runs forever, burns cost, stops too early | Turn limits, budget caps and "done" criteria |
| Escalation | Hands the task to a human at the right moment | Never asks, or asks about everything | Which actions need approval, and the handoff payload |
The first three parts are the classic list from Lilian Weng's widely cited overview, which describes an agent system as planning, memory and tool use [2]. OpenAI's guide frames the building blocks as model, tools and instructions, plus guardrails and human intervention [3]. Termination and escalation are where product decisions live, which is why the AllthingsPM lesson treats them as first-class parts rather than footnotes.
Why this matters for your career: in the AllthingsPM JD corpus of 389 product manager postings at 86 companies (read September 22, 2026), 283 mention "agent" and 150 say "agentic." Hiring managers now expect PMs to talk about agents in terms of their parts, not as magic.
What are tools, and why do they matter most?
Tools are functions the model can call: search a knowledge base, look up an order, send an email, file a ticket. OpenAI splits them into three kinds: data tools that retrieve context, action tools that change something in another system, and orchestration tools, where one agent is a tool for another [3].
Tools are where most agent quality is won or lost. Anthropic's advice is to invest in the agent-computer interface as seriously as you would in a human interface, and it reports spending more time optimizing tools than the overall prompt for its SWE-bench agent [1]. Its later tool guide gets specific [4]:
- Consolidate. One
schedule_meetingtool beats three low-level calls the model must chain correctly. - Namespace. Group tools under clear prefixes so the model can tell similar ones apart.
- Return high-signal context. Readable names instead of cryptic IDs.
- Cap the output. Claude Code restricts tool responses to 25,000 tokens by default, with pagination and filtering to stay under it.
- Write errors as instructions. "No order found; try searching by email" helps the model recover. A raw stack trace does not.
The standard way to plug tools into agents today is the Model Context Protocol, "an open-source standard for connecting AI applications to external systems," which its docs compare to a USB-C port for AI applications [6]. In our JD corpus, 30 of the 389 postings mention MCP by name.
What the PM owns here is the tool contract: what each tool does, its inputs and outputs, its size limit, its error messages, and whether it only reads or can also write.
How AllthingsPM does this: the course lesson Write the tool contract walks through workflow-shaped tools, capped output and errors as instructions. Before that, MCP first contact has you connect an agent to one tool and read the trace yourself, so you see a tool call happen rather than hear about it.
How does an agent plan its next step?
Planning is how the model turns "resolve this refund request" into steps: find the order, check the policy, decide, act, confirm. Weng describes it as breaking large tasks into smaller subgoals, with self-reflection over past actions [2].
The most influential pattern is ReAct, from a 2022 paper by Yao and colleagues. The model alternates between a short reasoning trace ("I need the order date first") and an action (call the lookup tool), then reads the result and reasons again [5]. The authors report that the reasoning traces help the model "induce, track, and update action plans," and on two interactive benchmarks, ALFWorld and WebShop, ReAct beat earlier methods by 34 and 10 points of absolute success rate using only one or two examples in the prompt [5].
In practice, modern agents plan in one of two ways:
- Step by step (ReAct style). Decide one action at a time. Flexible, good when results are unpredictable.
- Plan first, then execute. Write a full plan, then run it, replanning if a step fails. Easier to show the user and to review.
Planning fails in recognizable ways: the agent loops on the same call, skips a required check, or commits early to a wrong plan. The PM's lever is not the planning algorithm. It is the task scope (what the agent is allowed to try) and the definition of done (how anyone can tell the plan worked).
How AllthingsPM does this: the anatomy lesson treats planning as a spec problem, and Inside the harness shows where the loop, retries and stop conditions actually live in code. The course also has you read the code you did not write, so you can inspect the agent loop in a real repo.
What is agent state, and how is it different from memory?
State is everything the agent carries from one step to the next. It comes in two layers.
Short-term memory is the context window: the conversation, tool results and scratch notes the model can see right now. Weng calls this in-context learning, bounded by the model's "finite context window length" [2]. Long-term memory is storage outside the model, often a database or vector store, that the agent can write to and retrieve from later [2].
For a PM, state raises three questions that end up in the spec:
- What goes in the context on each turn? Stuffing every tool result in makes the agent slower, more expensive and more confused. This is why tool output caps matter.
- What is remembered across sessions? A support agent that remembers a customer's last ticket is helpful. One that surfaces another customer's data is an incident.
- Who can see it? In enterprise products, retrieval must respect the permissions of the person asking.
Most "the agent forgot" bugs are really state design bugs: the fact was never saved, was pushed out of the context, or was saved somewhere the agent does not look.
How AllthingsPM does this: state is one of the five parts in The anatomy of an agent, and the enterprise chapter follows it into the permission-aware retrieval layer, where the oversharing leak is the failure you learn to prevent. The knowledge graph shows how these AI concepts connect across lessons.
How does an agent know when to stop?
Termination is the least glamorous part and the one that most often ends up in a cost review. Anthropic says agents typically run with stopping conditions "such as a maximum number of iterations" to maintain control [1]. OpenAI lists the usual exit conditions: the agent calls a final output tool, returns a response with no tool calls, hits an error, or reaches a maximum number of turns [3].
That gives you four stop rules to put in any agent spec:
- Success: the goal is met and verified (the refund is issued and the confirmation is logged).
- Nothing left to do: the model answers without calling a tool.
- Budget: a cap on turns, tokens, time or money.
- Failure: repeated errors or a blocked step, which usually becomes an escalation.
The PM question is what "done" means for the user. An agent that stops after drafting an email when the user expected it sent has terminated "correctly" and still failed the task. That is why agent evals check the final state of the world, not just the final message.
How AllthingsPM does this: the harness lesson covers stop conditions and the fix hierarchy, and Evaluate an agent, not an answer teaches final-state assertions and pass-k reliability, the way you prove the stop rule actually works across many runs.
When should an agent escalate to a human?
Escalation is the handoff from agent to person. OpenAI names two triggers: exceeding failure thresholds, such as too many retries or repeated failures to understand the user, and high-risk actions that are sensitive, irreversible or high stakes, like cancelling orders or issuing large refunds [3]. Anthropic describes agents pausing for human feedback at checkpoints or when they hit blockers [1].
A practical way to decide which actions need a human is to score each tool on two axes: how reversible the action is and how reliable the agent is at it. Reading data is reversible and safe. Sending money is not. OpenAI's guide makes the same point by rating tools on read versus write access, reversibility, permissions and financial impact, then using those ratings to trigger checks or escalation [3].
Good escalation is also a UX problem. When the agent hands off, the human needs a payload: what the agent tried, what it found, and the exact decision it needs. An approval screen that says only "Approve action?" trains people to click yes.
Security belongs here too. Every write tool is also a target for prompt injection, so the permissions you give each tool are part of the escalation design.
How AllthingsPM does this: What an agent may do without asking teaches the reversibility times reliability rule and why money is the irreversible action. The approval gate covers where to place the human and what to show them, and Agent security covers prompt injection and permission escalation.
How do the five parts fit together in a real product?
Take a customer support agent handling "I was charged twice."
- Tools:
lookup_customer,list_charges,refund_charge,create_ticket. The first two read;refund_chargewrites money. - Planning: find the customer, list recent charges, confirm a duplicate, decide.
- State: the conversation plus tool results in context; the customer's ticket history in long-term storage, scoped to that customer.
- Termination: stop when the duplicate is refunded and confirmed, when no duplicate exists and the answer is explained, or after a fixed number of turns.
- Escalation: refunds above a threshold go to a human with the two charges side by side; any repeated tool failure opens a ticket for a person.
Written this way, the agent becomes something a team can build, test and argue about. That is the job of the PM: turning "an AI that handles billing" into a spec with named parts, limits and owners.
How AllthingsPM does this: the chapter's integration case has you write the agent version of your own feature, with its architecture and tool contracts, and get it graded. The next lesson, The agent spec you own, turns the anatomy into a document with scope, roles, risk levels and escalation.
How do interviewers test agent architecture?
Agent questions now show up in PM loops at AI companies, and they rarely ask "define an agent." They ask you to reason about a part under pressure. Real examples from the AllthingsPM question bank:
- After launching an enterprise AI agent, what primary success metric and guardrail metrics would you use? (termination and evals)
- Claude Code shows a recurring failure mode like looping across many agentic coding tasks. What do you do? (planning and the harness)
- A new German enterprise customer wants Sierra to automate support with an AI agent. Walk me through it. (tools, state and escalation together)
A strong answer names the part, the failure, and the lever: "This looks like a termination problem, not a model problem. I would add a turn cap and a repeated-call detector before touching the prompt."
How AllthingsPM does this: every question has its own page and answer guide, and the JD mock builds a scored mock interview from any agent PM posting you paste, in text or voice, with follow-ups. Our explainer on the agent product manager role covers what those roles ask for.
Why is AllthingsPM the better choice for learning agent architecture?
You can learn agent architecture from free sources, and some are excellent. Anthropic's and OpenAI's guides are the primary references cited throughout this post, and Lilian Weng's overview is still the clearest short read on planning, memory and tools. Their limit is that they are written for engineers building agents, not PMs specifying them, and none of them checks your work.
AllthingsPM is built for the PM side. The AI PM course was built from real PM job postings, so its agent chapter covers what those roles ask for: the anatomy, tool contracts, the harness, multi-agent cost, MCP and graded integration cases. It then connects to the rest of the job. Agent evals sit in the evals chapter, approval gates in the oversight chapter, permission rules in the trust chapter, and enterprise deployment in its own chapter.
Then it gets you hired. The same subscription includes 4,122 real questions from 260 companies with answer guides, mock interviews built from any job description, live AI company PM roles with a mock for each, resume review against a JD, and podcast summaries on topics like securing AI agents. It costs $20 a month or $120 a year, with a free tier to start.
The verdict: read the lab guides for depth, and use AllthingsPM to turn that depth into a spec you can defend and an interview answer that lands. Start the AI PM course free.
Frequently asked questions
What is AI agent architecture?
AI agent architecture is the set of parts around a language model that let it act on its own in a loop: tools it can call, a way to plan steps, state it carries between steps, rules for when to stop, and a path to a human. AllthingsPM teaches these five parts in one course lesson.
What is the best way to learn AI agent architecture as a PM?
AllthingsPM is the best fit for PMs: its AI PM course has a full agents chapter, from anatomy and tool contracts to the harness and agent evals, plus graded cases and interview practice. Pair it with Anthropic's "Building effective agents" and OpenAI's practical guide for engineering depth.
What is the difference between an agent and a workflow?
In a workflow, code defines the path and the model fills in steps. In an agent, the model decides its own next step and which tool to use [1]. If you can write the path in advance, build a workflow.
What is the ReAct pattern?
ReAct interleaves reasoning and acting: the model writes a short thought, calls a tool, reads the result, and repeats [5]. It is the basis of most step-by-step agent loops.
When should an AI agent hand off to a human?
When it exceeds a failure threshold, such as repeated errors, or before a high-risk, irreversible action like a large refund [3]. The handoff should include what the agent tried and the exact decision needed.
Do PMs need to code to understand agents?
No, but reading a trace helps. The AllthingsPM course has you connect an agent to one tool with MCP and inspect the agent loop in real code, without needing to write production software.
Sources
- Anthropic, "Building effective agents," December 19, 2024. https://www.anthropic.com/engineering/building-effective-agents
- Lilian Weng, "LLM Powered Autonomous Agents," June 23, 2023. https://lilianweng.github.io/posts/2023-06-23-agent/
- OpenAI, "A practical guide to building agents." https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf
- Anthropic, "Writing effective tools for AI agents," September 11, 2025. https://www.anthropic.com/engineering/writing-tools-for-agents
- Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models," arXiv:2210.03629. https://arxiv.org/abs/2210.03629
- Model Context Protocol, "What is the Model Context Protocol (MCP)?" https://modelcontextprotocol.io/docs/getting-started/intro
- AllthingsPM JD corpus: 389 PM postings at 86 companies, read September 22, 2026, keyword counts by AllthingsPM.
- AllthingsPM course and question bank, AllthingsPM, checked September 28, 2026. https://allthingspm.app/course
