Enterprise AI Agents Architecture, Protocols & Strategy

This guide is written for the people who actually have to build, buy, or approve these systems: engineering leads, architects, and technical decision-makers who have already sat through the pitch decks and now need to know what's underneath them. We'll cover the real architecture, the two protocol standards reshaping how agents connect to tools and to each other, the failure modes nobody puts in the demo, and a decision framework for build-vs-buy.

Key Takeaways

• Enterprise AI agents differ from RPA and chatbots by combining planning, tool use, and persistent memory in a loop that can adapt mid-task but that flexibility is also where most production failures originate.
• Two open standards now define how agents connect: the Model Context Protocol (MCP), created by Anthropic, standardizes agent-to-tool connections; the Agent2Agent (A2A) protocol, created by Google and now governed by the Linux Foundation, standardizes agent-to-agent communication. They solve different problems and are meant to be used together.
• Multi-agent systems are not automatically better than single-agent systems; they add latency, cost, and coordination failure modes, and should be adopted only when there's a genuine case for domain isolation or parallel execution.
• Production reliability depends on unglamorous engineering: durable checkpointing of execution state, hard recursion limits, schema-validated tool calls, and human-in-the-loop gates on irreversible actions.
• "Agent sprawl" of dozens of overlapping single-purpose bots is a real organizational failure pattern. Walmart's public consolidation of its AI tools into four domain "super agents" is a documented example of a large enterprise correcting it.

What an Enterprise AI Agent Actually Is

An enterprise AI agent is an AI-powered system that can interpret a goal, decide which steps or tools are needed, execute actions, inspect the results, and continue, retry, or escalate based on the state of the task. The defining characteristic is not simply that it uses an LLM, but that the system gives the model some degree of runtime decision-making within explicitly defined tool, data, and policy boundaries.

Not every agent needs persistent memory or multi-agent communication. A production agent can be stateless across sessions, use a deterministic workflow around an LLM, or operate as part of a larger multi-agent system. What matters is the combination of model-driven decisions, access to tools or data, and an execution loop capable of responding to intermediate results.

That last row is the one enterprise team underweight. RPA's biggest weakness is also its biggest strength: it's deterministic, so its failures are predictable and easy to trace. Agents trade that predictability for flexibility, which means the engineering burden shifts from "handle every edge case in the script" to "constrain and validate what the model is allowed to do." Teams that skip that second part are the ones who end up with an agent that occasionally does something confidently wrong.

The Enterprise Agent Stack Five Layers

Rather than treating "agent" as a monolith, it's more useful to think of production systems as five distinct layers, each with its own failure modes and design decisions.

Layer : Orchestration and Reasoning Core:

The LLM itself, typically operating in a ReAct-style loop (Reason → Act → Observe → Repeat) or a more structured state machine. This layer decides what to do next based on the current state and available tools.

Layer 2: Memory and Context Engine:

Split into short-term working memory (the current task's context window conversation history, intermediate results) and long-term memory (persisted across sessions, usually a vector database or graph store holding prior decisions, user preferences, and organizational knowledge). It's worth being precise here: retrieval-augmented generation and agent memory are often conflated, but they solve different problems. RAG retrieves relevant documents or facts to ground a response. Long-term agent memory retains what the agent itself has previously decided and learned closer to episodic memory than document search. An agent can have excellent RAG and still have no memory of what it did yesterday if that second layer isn't built.

Layer 3: Integration and Protocol Layer:

How the agent actually reaches CRMs, ERPs, databases, and internal APIs. This is where MCP lives (detailed below).

Layer 4: Inter-Agent Collaboration Layer:

Relevant only in multi-agent architectures is how independent agents, potentially built on different frameworks, discover each other and delegate tasks. This is where A2A lives.

Layer 5: Governance, Observability and Guardrails:

Access control, audit logging, rate limiting, and human-approval checkpoints. This layer is what separates a working prototype from something a compliance team will sign off on, and it's the layer most often bolted on late instead of designed in from the start.

A simplified data flow through these layers, for a request like "process this customer's return":

User/System Trigger

│

▼

Orchestration Core (interprets goal, plans steps)

│

▼

Memory Layer (pulls relevant history/preferences)

│

▼

MCP Client ──► MCP Gateway ──► MCP Server(s) ──► CRM / ERP / Order DB

│

▼

Validation (schema check, policy check on the result)

│

▼

Human-in-the-Loop Gate (only if action exceeds risk threshold)

│

▼

Action Executed + Checkpoint Saved

│

▼

Response / Next Step

MCP and A2A Two Protocols Reshaping Agent Integration

Model Context Protocol (MCP):

It was released by Anthropic in November 2024 as an open standard for connecting AI applications with external tools, data sources, and systems. Its client-server model allows an AI application to interact with MCP servers that expose capabilities such as databases, files, business applications, and APIs.

MCP is no longer solely an Anthropic-controlled project. In December 2025, Anthropic donated MCP to the Agentic AI Foundation (AAIF), a directed fund under the Linux Foundation, alongside other foundational agentic-AI projects.

The practical value of MCP is interoperability: instead of designing a completely different tool integration for every agent framework, teams can expose capabilities through a common protocol and reuse compatible integrations across supported clients and applications.

Agent2Agent (A2A):

It was introduced by Google in April 2025 to provide a common communication and task-exchange model for independent AI agents. Google transferred the project to the Linux Foundation in June 2025, and the protocol subsequently developed under open governance.

A2A reached its stable 1.0 specification in 2026. The current specification defines mechanisms for agents to discover capabilities, negotiate supported interaction modes, manage tasks, and exchange information without requiring one agent to expose its internal memory, tools, or reasoning process.

The Linux Foundation reported more than 150 organizations supporting A2A at the protocol's one-year milestone in April 2026, including major technology companies such as AWS, Google, IBM, Microsoft, Salesforce, SAP, and ServiceNow.

The architectural distinction remains straightforward: MCP connects an agent to tools and data; A2A connects an agent to another independent agent. They can therefore appear in the same enterprise architecture rather than being competing alternatives.

A useful mental shortcut: MCP is vertical (agent down to its tools); A2A is horizontal (agent across to other agents). A single-agent system deployed inside one team usually only needs MCP. A multi-agent system spanning departments or one that needs to interact with a different company's agent, like a procurement agent negotiating with a supplier's agent without either side exposing internal logic needs A2A on top of MCP.

Single-Agent vs. Multi-Agent

Multi-agent architecture should be treated as an architectural choice, not as the default definition of an advanced agent system.

Adding another agent introduces another coordination boundary. That can increase latency, token consumption, state-management complexity, observability requirements, and the number of places where an incorrect result can propagate. The benefit has to justify that additional complexity.

A multi-agent design becomes easier to justify when agents have genuinely different responsibilities, permissions, data boundaries, ownership, or execution requirements. Parallel tasks can also justify the pattern when independent operations can run concurrently. If one agent can safely perform the workflow with a well-defined toolset and policy layer, introducing additional agents may add complexity without adding enough capability to compensate for it.

Walmart and the Agent-Sprawl Question

Walmart provides a useful public example of an enterprise moving from many individual AI tools toward a more unified agent architecture. In July 2025, Walmart CTO Suresh Kumar described a company-wide framework built around four “super agents” for customers, associates, partners, and developers. The stated motivation included the observation that multiple useful agents could quickly become overwhelming and confusing, leading Walmart to pursue a unified entry-point model.

This does not prove that every enterprise should consolidate agents into a small number of super agents. It does show an important architectural issue: agent proliferation creates an interface and governance problem even when each individual agent is useful. Teams evaluating multi-agent architectures should therefore define ownership, discovery, permissions, and lifecycle management before allowing separate teams to create overlapping agents.

Why Agents Fail in Production

Demo environments hide the failure modes that show up at scale. Four are worth planning for explicitly.

The infinite-loop trap:

In a ReAct-style loop, nothing inherently stops an agent from reasoning, acting, observing, and reasoning again indefinitely if it never reaches a satisfying state, especially when a tool call returns an ambiguous or empty result. Mitigation: hard recursion/step limits and token-budget circuit breakers, enforced outside the model, not requested of it.

Token cost compounding:

Every additional reasoning step in an agentic loop is a full LLM call. A task that a human would describe as "one request" can involve ten or more model calls once you count planning, tool selection, and result evaluation at each step meaning the actual cost can be several times what a naive single-call estimate would suggest. Mitigation: route the expensive planning step to a strong model, but execute narrow, well-defined sub-tasks with a smaller or cheaper model where possible, and set per-task budget ceilings.

Tool hallucination:

Agents will occasionally call a tool with parameters that don't match its actual schema, or invoke a tool that doesn't exist. This is far more dangerous than a hallucinated sentence, because it can trigger a real write action. Mitigation: strict schema validation (JSON Schema or equivalent) on every tool call before execution, not after.

Multi-agent cascade errors:

In a multi-agent chain, one agent's slightly-wrong output becomes the next agent's input, and the error compounds rather than getting caught. Mitigation: contract-based structured outputs between agents (not free-text handoffs) and a verification step at each handoff, not just at the end.

Stateless execution and lost work:

Agentic workflows that don't persist their execution state have to restart from scratch after any interruption, a network timeout, an API rate limit, a crashed process. In frameworks like LangGraph and similar orchestration tools, the standard fix is durable checkpointing: writing execution state to a database (commonly Postgres or Redis) after each step, so a failed run can resume from its last completed step instead of re-running and re-paying for everything before it. If your evaluation of an orchestration framework doesn't include how (or whether) it checkpoints state, that's a gap worth closing before committing to it.

Evaluation failure and quality drift:

An agent can complete a workflow successfully from a system perspective while still producing the wrong business outcome. For example, a tool call may succeed, the workflow may reach its final state, and the response may look valid while the agent selected the wrong record, missed an important condition, or produced an incomplete result. Mitigation: evaluate agents against representative production tasks, not just whether the workflow technically completes. Track task success rate, tool-selection accuracy, human correction rate, policy violations, escalation rate, and regression performance as the system and underlying models change.

Treat Agents as a New Identity Class, Not a Feature

The security conversation around agents tends to stay abstract ("use encryption, use RBAC"). The more useful framing is Zero-Trust Agent Architecture (ZTAA): treat every agent as its own identity with its own least-privilege permissions, never as an extension of a shared service account.

A few specifics that matter more in practice than the generic checklist:

• MCP Gateway as a control plane: Rather than letting an agent's MCP client connect directly to internal MCP servers, enterprise deployments route that traffic through a gateway that handles OAuth/identity-provider integration, rate limiting, and logging so security policy is enforced centrally rather than reimplemented inside every agent.
• Per-user OAuth pass-through, not shared credentials: An agent acting on behalf of a specific employee should inherit that employee's actual permissions via delegated OAuth, not a broad service account that can do more than the requesting user could do themselves. This alone prevents a large share of "the agent did something a person wasn't authorized to do" incidents.
• Indirect prompt injection: The more dangerous version of prompt injection in enterprise settings isn't a user typing something malicious into a chat box, it's an instruction hidden inside a document, email, or web page that the agent retrieves and treats as trustworthy context. Any agent that reads external or user-supplied content as part of its workflow needs that content treated as untrusted data, not instructions, by design.
• Deterministic human-in-the-loop gates: Don't leave "should this need approval?" to the model's judgment. Hard-code approval requirements for specific action classes financial transfers above a set threshold, irreversible deletions, anything touching regulated data so the gate exists regardless of how the agent reasons about the request.

Agent identity and authorization boundaries:

Authentication and authorization should be treated as separate controls in an enterprise agent architecture. Authentication establishes who the user or agent is; authorization determines which resources and actions that identity is permitted to access. The agent should therefore have an explicit identity and a defined permission boundary rather than inheriting unrestricted access from a shared service account. Where an agent acts on behalf of a user, delegated permissions should be enforced at the tool or API layer so that the agent cannot perform an action that the requesting user would not be authorized to perform.

These permissions should also be observable. Every sensitive tool call should record the requesting identity, agent identity, requested action, authorization decision, execution result, and relevant timestamp. This creates an audit trail that can be reviewed when an agent performs an unexpected action and makes it easier to investigate security incidents without relying on the model's own explanation of what happened.

Where Enterprise AI Agents Are Actually Delivering Value

Rather than a generic list of every department that "can use AI," the workflows below share a common trait: they're high-volume, have clear success criteria, and involve enough system integration that a rules-based script would be brittle.

• Finance and accounting: Invoice extraction and matching against POs, anomaly flagging in expense data, and generating draft compliance reports for human review all cases where the agent's job is to reduce the volume of routine work reaching a human, not to make the final call.
• Customer support:. End-to-end resolution that spans identity verification, CRM history, order status, and logistics is genuinely different from a chatbot that can only answer FAQ-style questions, because it's writing to systems of record, not just reading from a knowledge base.
• Software engineering: Automated first-pass PR review, test case suggestions, and documentation drafting used as a force multiplier ahead of human review, not a replacement for it.
• HR: Resume screening and onboarding-task orchestration, with the caveat that any step influencing a hiring decision needs explicit human sign-off and bias auditing, not just "the model said no."

Build vs. Buy A Practical Decision Framework

A reasonable rule of thumb: buy for anything that isn't your competitive differentiator, build for anything that is. Most Enterprise land on a hybrid a purchased platform (Microsoft Copilot Studio, Salesforce Agentforce, AWS Bedrock AgentCore, or a custom stack built on LangGraph/AutoGen-style orchestration) for the common 80%, with custom agents built in-house for the workflows that are actually specific to how the business operates.

Before evaluating vendors or committing engineering time, confirm the basics are in place: centralized, reasonably clean data; APIs that don't require screen-scraping; a defined owner for governance and security review; and executive sponsorship for the workflow you're targeting. Skipping this step is the most common reason agent pilots stall, not model quality.

Measuring Whether It's Actually Working

Cost savings alone is an incomplete ROI story. Track task completion rate, resolution time, error/escalation rate, and Automation rate (percentage of instances completed without human intervention) alongside cost per transaction. The standard formula still applies

ROI (%) = ((Annual Benefit − Total Cost) ÷ Total Cost) × 100

but treat it as one input, not the whole scorecard. An agent with a strong ROI number and a rising escalation rate is quietly shifting cost onto your humans-in-the-loop, and that's worth catching before it shows up as a support-quality complaint rather than a spreadsheet line.