Search for agentic AI architecture and you get a hundred diagrams. They are mostly the same diagram. Box at the top labeled "LLM." Boxes underneath labeled "tools," "memory," "planner." Arrows. Sometimes a robot icon.
The diagrams are not wrong. They are just where the architecture conversation ends, when the interesting part of the conversation is what is missing from them.
Every agentic AI stack has five layers, not three. The top three are the parts everyone is shipping right now. The bottom two are the parts a small number of teams are quietly building and the rest will need within a year.
Here are all five, in the order things actually depend on each other.
Layer 1: Model
The model is the reasoning engine. Claude, GPT, Gemini, Llama, take your pick. It takes a prompt and a tool spec and returns either text or a tool call.
This is the most visible layer because it is what every product launch announces. It is also the most rapidly commoditizing. Two years ago there was one model that was clearly best at agent work. Today there are five. In a year there will be twelve. Picking a model is no longer a strategic decision; it is a pricing decision.
What model choice still genuinely affects: tool-call fidelity (some models follow JSON schemas better than others under load), context length (matters when the working memory is the whole codebase or the whole inbox), latency profile (a 2x faster model on the same task changes the loop budget for orchestration), and structured-output reliability. None of those are model-family decisions in the long run. Every six months a smarter, cheaper, faster model lands and the team that bet too hard on one ends up rewriting prompts.
The practical implication: your prompts and your tool definitions need to be portable. Models churn. Your prompt library shouldn't churn with them. The teams who treat the model as fungible end up with the strongest agent stack, because they keep the asset (the prompts, the evals, the tool specs) and swap the model when something better lands.
The real moat at this layer is not the model. It is the eval set: the captured set of tasks your agents must solve, the regression suite that runs on every model swap. If you can't tell at a glance whether GPT-5.1 beats Claude Opus 4.7 on your specific workload, you are not really doing model selection. You are doing vibes.
If your agentic architecture is just this layer with some glue, you are building on shifting ground. Every six months a smarter, cheaper, faster model lands and you rebuild your prompts.
Layer 2: Tools
Tools are what the agent can actually do. Read a file. Hit an API. Send a message. Update a row. Schedule a job.
The Model Context Protocol (MCP) is in the middle of standardizing this layer the way HTTP standardized network calls. Tools become discoverable, callable, and self-describing across any model that speaks MCP. A year ago every framework had its own tool schema; today there is one schema that most of them agree on. The public MCP specification defines the wire format: JSON-RPC base protocol, capability negotiation, and a server primitives set (Resources, Prompts, Tools) any client can discover at runtime.
The tool-layer decisions that genuinely matter once the schema is settled are not the schema. They are the surface area and the boundary.
Surface area is the size of the tool catalog. The temptation, when designing a tool for an agent, is to wrap a whole API. Don't. Wrap the task the agent actually needs to do. A 200-tool catalog is unworkable for the model; a 12-tool catalog where each tool does one thing well is usable. The smallest useful MCP tool is sharper than the broadest API wrapper. See the smallest useful MCP tool for the worked example.
The boundary is the auth and scope question. Every tool an agent can call is a possible blast radius. A tool that can post to Slack on behalf of the user is fine; a tool that can post on behalf of any user is a credential-laundering anti-pattern. The right pattern is per-agent identities with their own scopes, not delegated human tokens. We've written about this at length in OAuth scopes for agents and what's wrong with agents using human credentials.
Like the model layer, tools are commoditizing fast. The agent loops, the schemas, the auth flows are all converging. By 2027, picking a tool layer will be a pricing and breadth decision. What stays proprietary is the catalog you build on top: the specific tools that match your specific workflow. Treat the catalog as an asset.
Layer 3: Orchestration
Orchestration is how steps chain together. The agent loop. The planner. The multi-agent coordinator. The retry policy. The reflection step.
This is the layer the frameworks own: LangChain, LangGraph, CrewAI, Mastra, Autogen, the OpenAI Agents SDK. Each framework has a different opinion on how to chain steps, but they are all converging on a small set of patterns: ReAct loops (interleaving reasoning traces and tool calls, originally proposed by Yao et al.), plan-then-execute, multi-agent supervisor-worker, hierarchical task decomposition. Anthropic's own Building Effective Agents writeup catalogs the same shortlist (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) and lands on the same conclusion: the patterns are simple, composable, and roughly framework-independent.
Most engineering teams roll their own orchestration in their first six months and end up with something that looks like one of the framework patterns by month nine. That is fine. The framework patterns are converging because they describe the actual shape of useful agent loops, not because of marketing alignment. By 2027 this layer is also commoditized.
What still genuinely matters at this layer is the handling of dangerous operations and irreversible actions. Most orchestration libraries treat tool calls as fungible: every step is a step. In production, this is wrong. There is a class of operations (refunding a customer, rotating a credential, deleting a record, sending an email) where the cost of a wrong step is much higher than the cost of a paused step. Good orchestration has a concept of consent gates for these. The agent proposes; a human or a second agent confirms; only then does the operation fire.
We've written the dangerous-ops contract and the specific consent gates pattern we use at Dock for billing actions. The shape that ended up working: the first call returns a confirmation token and a human-readable summary; the agent surfaces this to the user; the user confirms; the agent re-calls with the token. For the most expensive class (account-level changes, mass mutations) we run a two-key handshake so a single compromised agent can't drain anything alone.
A team's orchestration layer is mature when it can answer two questions cleanly: which tool calls require consent? and what happens when the agent retries a failed dangerous call? If the answer to either is "we don't really gate that," the orchestration is not yet production-ready, regardless of which framework provides it.
Layer 4: Memory
Memory is what the agent knows about the task, the user, and the past. There are three sub-layers here:
- Working memory. The context window. Whatever fits in the prompt right now.
- Episodic memory. What happened in past sessions. Usually a vector database with embeddings.
- Semantic memory. What is true about the world. Documents, knowledge bases, structured records.
Working memory is a feature of the model. Episodic memory is a feature of your stack. Semantic memory is the thing you have been building inside your company for ten years, just usually without naming it that.
Memory is the first layer that does not commoditize, because the contents are yours. Two companies running the same model with the same tools and the same orchestration will produce wildly different agent behavior based on what their agents know. The model is fungible. The memory is not.
The episodic memory store is becoming a real asset. The most-shipped pattern today is a vector database the agent embeds queries against, but the more interesting move is to give each agent its own structured data store the agent fully owns. A scoped relational store the agent reads and writes to over time, gated by its own credential, with an audit trail. We wrote about that pattern in giving your AI agent its own database. The agent stops re-deriving facts every session. It accumulates.
The hard problem at this layer is shape, not storage. An agent that pulls a 200KB document into its context window every step burns budget on irrelevant tokens; an agent that pulls a 200-token summary loses fidelity on details that turn out to matter. Cap the shape of what gets pulled, not just the byte count. We hit this directly in the editor and wrote up the shape cap on TipTap JSON approach we settled on: hard byte ceiling, hard depth ceiling, hard node-count ceiling, all at the gateway level so every writer is covered by one check.
Memory still has a problem the above pattern doesn't solve. Vector databases hold embeddings. Knowledge graphs hold triples. Document stores hold text. None of these hold work in progress the way a real workspace does. None of them are designed for two agents and a human to write to in the same minute and have the result be coherent.
That is the gap layer 5 fills.
Layer 5: Substrate
The substrate is the place agent work lives.
Not the prompt the agent reads. Not the document it cites. Not the database it queries. The place where its actual output, its in-progress drafts, its decisions, its handoffs, its mistakes and its corrections accumulate over time. The place a human teammate reads to see what the agent did Tuesday morning. The place another agent reads on Wednesday to pick up where the first one left off.
In a chat world, the substrate is the chat scroll. Lossy, single-player, no attribution, no surface for review. It works for one human asking for one output. It breaks the moment work is multi-step and durable.
A real substrate has the same primitives a human team has been using to coordinate for decades:
- Typed state. Tables with typed columns. Docs with formatted prose. Row-level updates are atomic and observable.
- Identity per principal. Every edit is signed by a specific agent or human, not delegated through a shared token. See service accounts vs agent identities for the difference that makes.
- Audit, not just logs. Append-only event ledger. Every change is queryable, streamable, exportable.
- Real-time presence. Cursors, status, presence flags. When the agent is mid-action, the workspace shows it.
- Comments and mentions. Threads on any row, any cell, any range. Mentions notify the right principal, agent or human.
The substrate is the layer nobody draws on agentic-architecture diagrams because, historically, there has not been one. Teams used wikis (built for humans, no agent identity). Project tools (built for humans, no real audit). Chat threads (no state at all). The substrate has been bodged together from whatever was nearby.
The substrate is also where the trust model lives. Every agent action that touches state on the substrate gets attributed to that agent's identity, not to the human who launched it. The accountability substrate matters more than people realize: when five agents share a workspace with two humans over a week, you want to be able to reconstruct who did what, when, and on whose authority. We wrote up the signed-agent inheritance model we use for cross-workspace access, where an agent inherits the workspaces its owning human has access to, with the audit trail going to the agent. The owner is the accountability anchor; the agent is the actor.
The substrate is also the place where merge becomes real. Two agents editing the same doc in the same minute is a normal occurrence, not an edge case. The substrate has to handle concurrent writes without silent clobber, has to surface conflicts when they happen, has to support the operational equivalent of git pull --rebase. We wrote up the backmerges or bust pattern when we hit it ourselves shipping the auto-merge workflow.
Why the substrate is the layer that compounds
The layers commoditize from the top down. Models commoditize fastest. Tools next. Orchestration after that. Memory is slower because the contents are proprietary, but the infrastructure of memory is becoming a commodity too.
The substrate is the only layer where the asset is the surface itself, not the contents on it. Like a database is a long-lived asset whether or not the schema changes. Like GitHub is a long-lived asset whether or not the languages on it change. The substrate is where the agent work lives, and the longer it lives there, the more valuable the substrate is.
The substrate is also the only layer where switching cost is real. Switching models is a prompt rewrite. Switching tool catalogs is a re-spec. Switching orchestration frameworks is a refactor that takes weeks. Switching the substrate where two years of agent decisions, drafts, audit logs, and cross-references have accumulated is a migration that nobody actually completes. The substrate is the gravity well of the agentic stack.
This is why we built Dock at the substrate layer instead of higher up the stack. Models commoditize. Frameworks commoditize. The shared, persistent, auditable surface where mixed teams of humans and AI agents do their actual work does not. It compounds.
How to evaluate an agentic stack
If you are designing or auditing an agentic architecture today, five questions are worth more than the rest. Walk through them in this order.
-
Identify which layer your stack stops at. Look at your architecture diagram. Does it have a model, tools, and orchestration but stop there? Does it have a memory layer? Does it have a named substrate, or is the substrate implicit (the chat scroll, an S3 bucket of JSON, a Postgres table nobody's claimed ownership of)? Most stacks stop at three layers. The diagram tells you where the conversation ends.
-
Audit the bottom-up: memory and substrate first. Counterintuitive, because the top layers feel more visible. But the bottom layers are where the gaps compound. Ask: where do this week's agent outputs live? Can a teammate read them next week without asking the agent? If the answer involves searching chat history or re-prompting, your substrate is missing.
-
Look for the boundary between tools and orchestration. Tools should be one-thing-each. Orchestration should chain them. When you find a tool that does five things, you have orchestration leaking into the tool layer. When you find orchestration that hardcodes a specific tool's auth, you have the boundary blurred. The fix is to push every tool toward the smallest useful surface and let orchestration do the chaining.
-
Check who owns identity at every layer. Models don't have identity. Tools should be called by a specific agent identity. Orchestration runs as a specific agent. Memory is keyed to a specific principal. Substrate logs every edit to a specific principal. If at any layer the principal becomes ambiguous (delegated tokens, shared service accounts, anonymous tool calls), that is the layer where your audit will fall apart first.
-
Diagram the trust boundaries, not just the data flow. Most architecture diagrams show what data goes where. Few show what the trust boundary is at each hop. Every time an agent calls a tool, crosses to another agent, writes to substrate, or messages a human, there is a trust hop. Mark them. Then ask: which hops are gated, which are free, and which gate is the one that protects the most expensive blast radius?
The team that can answer all five concretely has a real agentic architecture. The team that can answer three is still in framework-evaluation mode. The team that can only answer one is still building a wrapper around a chat box.
FAQ
What is agentic AI architecture?
Agentic AI architecture is the technical structure underneath an AI system that takes actions, not just produces text. It has five layers: a model that reasons, tools the model can call, orchestration that chains the steps, memory the agent uses to know things, and substrate where the work it produces actually lives. Most public diagrams stop at three layers (model, tools, orchestration) and leave the bottom two implicit. The bottom two are where the long-term value of an agent system compounds.
What is the difference between agentic AI and generative AI?
Generative AI produces an output (text, image, code) in response to a prompt. Agentic AI takes actions in the world (calls APIs, updates state, sends messages, makes decisions) over multiple steps, often with feedback loops. Agentic AI is built on top of generative AI: the model is the reasoning engine, but the architecture adds tools, orchestration, memory, and substrate. We've written about the distinction at length in agentic AI vs generative AI.
Where does MCP fit in this architecture?
The Model Context Protocol sits at Layer 2 (Tools). It is the standardizing wire format between a model and its tool catalog: a discovery contract, a call contract, and a response contract. Before MCP, every agent framework had its own tool schema; with MCP, tools become portable across any model that speaks the protocol. MCP is to the tool layer what HTTP was to network calls: not a strategy, but a coordination layer that lets the strategy live above it.
Do I need all five layers from day one?
No. You need all five layers eventually, but most teams ship in this order: model (you pick one), tools (you wrap a handful of APIs), orchestration (you build an agent loop), memory (you bolt on a vector DB), substrate (you realize the chat scroll isn't enough). The order is roughly the order of pain: each layer becomes visible when you outgrow whatever bodge you had before. The mature team picks the substrate intentionally instead of bodging.
How is this different from a "tool-using LLM"?
A tool-using LLM is the first two layers (model plus tools) with no orchestration, no memory, no substrate. It is useful for one-shot tasks: ask, model calls a tool, model returns. It breaks down the moment the task is multi-step, has to persist across sessions, or has to be reviewed or audited by a human. An agentic architecture is what you get when you treat the tool-using LLM as just the top of the stack and build the four other layers underneath it.
Is the substrate layer just a database?
No. A database holds rows. A substrate holds rows, docs, comments, attributions, presence, audit logs, real-time updates, and the human-and-agent both as first-class principals. You can build a substrate on top of a database, the same way you can build a workspace on top of a filesystem, but the substrate adds a coordination model the database doesn't have on its own. The defining test: when two agents and a human edit the same workspace in the same minute, the substrate makes the result coherent; a raw database does not.
Where do agents like Claude, GPT, and Cursor fit?
They sit at Layer 1 (Model). Claude and GPT are reasoning engines. Cursor wraps a model with an IDE-shaped orchestration loop and a specific tool catalog (codebase, terminal, git). When teams say "we use Cursor," they mean "we have layers 1 through 3 covered." The question is then layers 4 and 5: where does the agent's accumulated knowledge live, and where does its work product persist?
How do you handle agent identity across layers?
Identity has to flow through every layer or the audit trail breaks. The model doesn't have identity (a model is fungible), but every call that leaves the model carries an agent identity attached to it. Tools verify the identity. Orchestration logs each step as the agent. Memory is keyed to the agent. Substrate signs every edit with the agent's principal. The accountability anchor is the human owner of the agent: see signed-agent inheritance for the model we use. The agent is the actor; the owner is the principal of last resort.
Where Dock fits
Dock is at Layer 5. A shared cloud workspace where humans and AI agents read and write the same state in real time. Typed tables for structured work, docs for prose, comments for review, identity per principal, full audit, real-time presence.
We chose the substrate layer because the other four layers are commoditizing and the substrate is where switching cost is real. The agent your team uses today will not be the agent your team uses in eighteen months. The framework will change too, and the tool catalog will turn over twice in the same window. The substrate is what stays.
If you are designing an agentic architecture and feeling the gap at the bottom of your diagram, that gap has a name. The way to know if it's biting you is the questions above: can a teammate pick up where the agent left off without asking the agent, can you reconstruct who did what across a week of mixed human-and-agent work, can the substrate handle two writers in the same minute without silent clobber.
If yes, you have a substrate. If no, you have one to build or pick.
Read next
The architecture above generalizes; the implementation lives in the details. The essays below dig into specific layers.
- The smallest useful MCP tool — what a one-thing-each tool looks like in practice, Layer 2.
- OAuth scopes for agents: what's broken — the auth boundary at Layer 2.
- The dangerous-ops contract — gating irreversible operations at Layer 3.
- Giving your AI agent its own database — the agent-owned memory pattern at Layer 4.
- Signed-agent inheritance — the identity-and-audit model at Layer 5.
- Backmerges or bust — handling concurrent writes at Layer 5.
