The word “agent” is now attached to almost every AI product on the market, which makes it nearly useless as a description. The fix is to stop treating “agent” as a single magic thing and see the ladder underneath it. Almost everything in this space is built from a handful of simple parts, stacked one on top of another, and each rung adds exactly one new capability. Climb it slowly and the buzzwords resolve into something you can actually reason about — and buy, and govern.
The bottom rungs: a call, then a chain
The base of the ladder is a single model call: you send text, you get text back. Crucially the model is stateless — it remembers nothing between calls, so anything you want it to “remember” you have to send again each time, within its context window (its working memory, measured in tokens). The next rung up is prompt chaining: you break a task into a fixed sequence of calls, each feeding the next — draft an outline, then write from it. You decided the steps and their order, in code. This is already, technically, the simplest kind of “workflow.”
The hinge: tool use
Up to here the model can only talk. The rung that changes everything is tool use (also called function calling): you give the model a set of tools — a search, a calculator, a database, the ability to run code or call an API — and when a request needs one, the model doesn’t run it itself; it asks your system to, by emitting a structured request, and your code does the real work and hands back the result. This is the hinge of the whole ladder, because it’s the moment the model can act on the world rather than merely describe it. Every rung above this exists only because of it. A model equipped with tools, retrieval and memory is sometimes called an “augmented” model — the basic building block from which agents are made.
A note on plugs: what MCP actually is
Once models could use tools, every team wired up its own one-off connections — a bespoke adapter for each tool, each model, re-built by the next team along. A standard emerged to stop that sprawl: the Model Context Protocol (MCP), a common shape for the plug between a model’s “hands” and the tools and data it reaches for. The honest one-liner is that MCP is USB-C for tools. It’s worth being precise, because it has become one of the most name-dropped and least understood acronyms in the field: MCP is not a kind of AI, not a “brain”, and not a foundation the rest of the stack sits on. It is plumbing — an integration layer that sits on top, and an optional one at that. It belongs to this tool-use rung; it is not a new rung above the agent, and a system is no more “agentic” for using it. When you next see a diagram that places MCP alongside “LLM” and “agent” as if they were the same kind of thing, that’s the tell: one is a model, one is a pattern of control, and MCP is the connector between the hands and the tools.
A clarification: RAG is knowledge, not agency
One rung is constantly mistaken for agency: retrieval, often called RAG. It fetches relevant documents — from your wiki, a database, a search index — and pastes them into the prompt before the model answers, so the answer is grounded in your data rather than the model’s frozen training. This is enormously useful, and it is the backbone of “chat with your documents.” But it adds knowledge, not will: in classic RAG your code decides what to fetch and when; the model doesn’t choose. A chatbot with RAG is still a chatbot. Agency only appears when the model itself controls the loop.
The heart: workflow versus agent
This is the distinction everything hinges on, and it comes down to a single question: who decides the next step — your code, or the model? In a workflow, you designed the path; the steps, the order and the branches are fixed in code, and the model fills in the intelligent bits at each station but can’t go off-script. It’s predictable, controllable, cheaper, and easy to debug. In an agent, the model decides the path: you give it a goal and some tools, and it works out which tools to use, in what order, and when it’s finished. It’s flexible and powerful, but less predictable, more expensive, and harder to debug. The cleanest analogy: a workflow is a train on rails; an agent is a taxi driver you tell “get me to the airport.” Most products marketed as “agents” are, on inspection, workflows — and that is usually the right and responsible choice.
The loop (and how much leash)
What actually makes an agent an agent is that it runs in a loop: it looks at the situation, decides a next step, acts through a tool, observes the result, and repeats — until the job is done. The most important part of that loop is the stopping condition: a loop with no exit is a runaway that burns money in circles, so real systems cap it (a maximum number of steps, a budget, a check that the goal is met). Two more dials sit on top. Multiple agents can be coordinated for genuinely parallel work, but at a steep premium — multi-agent setups can use on the order of fifteen times the tokens of a single chat and are harder to debug, so “more agents” is rarely “smarter.” And autonomy is itself a dial, not a switch: you choose how much leash, from “asks permission for everything” to “acts on its own,” and you keep a human at the checkpoints that matter — anything consequential or hard to reverse.
| Rung | What it adds | Use it when |
|---|---|---|
| Single model call | A one-shot answer (no memory, no action) | Summarise, classify, rewrite — one shot |
| Prompt chaining | A fixed sequence of steps | The task splits into clean, known steps |
| Tool use | The model can ACT, not just talk | You need live data or a real action |
| Retrieval (RAG) | Grounding in your data (knowledge, not agency) | Answers must come from your documents |
| Workflow | A path YOU designed, with the model in it | You can draw the decision tree in advance |
| Agent (a loop) | The MODEL chooses the path, until done | The task is open-ended; the path can’t be known ahead |
The discipline is choosing the lowest rung that solves the problem, not the most impressive one. If you can draw the decision tree in advance, build a workflow — it’s cheaper, more controllable and easier to trust. Reach for a real agent only when the path genuinely can’t be known ahead of time.
So what
When a vendor says “agent,” you now have the questions that cut through it: does the model decide its own steps, or follow a path someone wrote? Does it act through tools, or only generate text? Does it run in a loop with a sensible stopping condition, and how much can it do without a human? The honest answer is usually that the thing in front of you is somewhere low on the ladder — and that’s fine, because the goal is never to be as “agentic” as possible. It’s to use the simplest rung that does the job well, safely, and at a cost you can defend.
Frequently asked
- What’s the difference between a workflow and an agent?
- Who decides the next step. In a workflow your code decides — the steps and order are fixed in advance and the model fills in the intelligent parts. In an agent the model decides — you give it a goal and tools and it chooses its own path in a loop. Workflows are predictable, cheaper and easier to debug; agents are flexible but less predictable and costlier. Most “agents” on the market are actually workflows.
- What is the “agent loop”?
- The cycle that makes an agent an agent: it assesses the situation, decides a next step, acts through a tool, observes the result, and repeats until the task is done. The critical part is the stopping condition (a step limit, a budget, a goal check) — without one, a stuck agent loops forever and burns money.
- Does using RAG make something an agent?
- No. Retrieval (RAG) adds knowledge by fetching your documents into the prompt before the model answers, but in classic RAG your code decides what to fetch — the model has no control of the process. Agency means the model drives a loop. A chatbot with RAG is still a chatbot.
- Is a more “agentic” system always better?
- No. Higher on the ladder means more capable but also more expensive, less predictable and harder to control. The discipline is to use the simplest rung that solves the problem — often a workflow rather than a full agent — and to add autonomy only where the task genuinely requires it.
- Where does MCP fit — is it a type of AI?
- No. The Model Context Protocol (MCP) is a connection standard — a common way to plug a model into tools and data instead of wiring each integration by hand. Think “USB-C for tools.” It lives at the tool-use rung, not above the agent: it’s plumbing that makes tools easier to connect, not a new capability and not a brain. A system is no more “agentic” for using it. Treat any diagram that lists MCP next to “LLM” and “agent” as peers with suspicion — they’re different kinds of thing.