Building an AI Control Plane for Agents on Kubernetes: What I Built, What Broke, and What It Changed

Author: Animesh Kumar Sinha/ Technical AI Architect / https://www.linkedin.com/in/animesh-kumar-sinha-56792119/

Note: This article is a volunteer contribution, and we’re grateful to Animesh for sharing his time and expertise with our readers.

Introduction:

Animesh Kumar Sinha is a solution architect with about 20 years in enterprise software, and his recent work focuses on agentic AI, LangGraph, RAG systems, and Databricks. His team ran three agentic systems on one Kubernetes cluster. Each worked well on its own, but nobody could easily tell which prompt made a decision, which agent spent the tokens, or how to stop one agent without hurting the others.

To fix this, he built an AI control plane: one place to register, version, govern, observe, and stop every agent. In this article, he covers the full design, from the agent registry, gateways, and policy as code to human approval, tracing and audit, along with what broke along the way and what changed once it was live. Read his other article on MCP at:
https://journals-times.com/2026/10/26/mcp-in-depth-turning-a-multi-agent-domain-rag-system-into-mcp-servers-on-kubernetes/

The day I realised we had agents but no control

A year ago I would have told you our agents were in good shape. We had three agentic systems running on the same Kubernetes cluster, and each one worked.

The first was an agentic decision system: a set of LangGraph agents that pull market data, run impact analysis, apply business rules, and propose changes to business values that directly affect revenue. The second was a domain RAG system: a planner, writer, and reviewer that searches many enterprise sources, a vector store, a semantic layer, and a lineage graph to answer questions about the domain. The third was an agentic pipeline builder that turns a plain-English requirement into a validated PySpark pipeline on Databricks.

Each system had been built well. Each had its own logs, its own way of calling the LLM, its own retry logic, and its own idea of what “production ready” meant. That was the problem.

When someone from the business asked me a simple question, I often could not answer it quickly:

• Which version of the decision agent produced this recommendation, and which prompt was it using?
• How much did the domain search spend on tokens last week, and which agent spent most of it?
• Can the pipeline builder’s agent touch production tables, or only development ones?
• If a decision agent starts behaving strangely at 2 a.m., how do we stop it without taking the other two systems down?
• Can we prove to an auditor what an agent did on a given day, and why?

The answers existed, but they were spread across three codebases, three log formats, and a few people’s heads. Every new agent made it worse. We were adding agents faster than we were adding ways to govern them.

So I built an AI control plane: one place to register, version, configure, govern, observe, and stop every agent we run, no matter which system it belongs to. This article is the full story: what a control plane is, how I designed each part, the code behind it, what went wrong, and what it changed.

It is long. Each section stands on its own, so jump to the part you need.

What an AI Control Plane Actually Is

The term comes from networking and Kubernetes, and the idea carries over cleanly.

In Kubernetes, the data plane is where the work happens: the nodes and pods running your containers. The control plane is the API server, the scheduler, and the controllers. It decides what should run, where, with what configuration, and it keeps reconciling the cluster towards that desired state. Your pods never call the control plane to serve a request. The control plane manages them from the side.

An AI control plane applies the same split to agents:

• The data plane is the agents themselves, the LLM calls they make, and the tools they use.
• The control plane decides which agents exist, which version of each is live, what each one is allowed to do, how much it may spend, and what evidence it leaves behind.

The simplest test I use: an observability tool watches agents and tells you what went wrong afterwards. A control plane governs them. It can block a tool call, require human approval, switch a model, cap a budget, or turn an agent off before damage reaches a user.

I was not inventing a new category. Every major AI platform now ships its own version of this idea, which told me I was solving a real problem:

• Microsoft Foundry Control Plane describes itself as a unified management interface for “visibility, governance, and control for AI agents, models, and tools.” It brings together a fleet inventory of agents, continuous evaluation, compliance policies tied to Azure Policy, Defender and Purview, and cost and token tracking.
• Amazon Bedrock AgentCore splits the problem into services: Runtime, Gateway, Identity, Memory, Observability, Evaluations, and Policy. Its Policy service intercepts every tool call at the Gateway and checks it against rules written in Cedar, AWS’s open-source policy language. Observability is OpenTelemetry-compatible.
• Google’s Gemini Enterprise Agent Platform, announced in April 2026, includes an Agent Registry to catalog agents and an Agent Gateway for identity and access control, alongside audit logs and governance policies.
• Databricks AI Gateway governs model and MCP services as Unity Catalog objects, with rate limits, budgets, guardrail policies, usage tracking and request logging to Delta tables.

Read those four descriptions side by side, and the same building blocks appear every time: a registry, a gateway that every call passes through, identity, policy, evaluation and observability. That became my checklist.

Why not just buy one of them? Because my agents run on our own Kubernetes cluster, span Azure and Databricks, and use LangGraph rather than any one vendor’s agent runtime. I needed something that fit the platform we already had. I borrowed the ideas and built a thin control plane from open-source parts and services we already ran.

The data plane I had to govern

Before designing anything, I wrote down exactly what the control plane would manage. Three systems, all on one Kubernetes cluster, all built on LangGraph, all calling LLMs and tools.

Agentic decision system. Agents ingest market data, run impact analysis, apply business rules and safety limits, and propose changes to business values. This is the only system whose output directly changes something customers and the business pay for. A mistake here is not a bad answer on a screen; it is money.

Domain RAG and search. A planner breaks a question down and searches many sources in parallel: a vector store with semantic search, BM25 keyword search, a semantic layer, a lineage graph, and the usual enterprise tools. A writer drafts an answer and a reviewer checks every claim. It reads data but never changes it. It also tracks evaluation metrics, so the control plane had to fit around metrics that already existed.

Pipeline builder. An enterprise pipeline builder. A requirement goes in; an agent generates PySpark and SQL, validates it against the codebase, retries on failure, and deploys through Databricks Asset Bundles. It writes code that will later touch production data.

The most useful decision I made early was to give each system a risk tier. The tier decides how strict every other part of the control plane is for that system.

Tiering stopped a lot of arguments. Without it, every rule felt either too strict for the search system or too loose for the decision agents. With it, one control plane could serve all three without treating them the same.

AI Control Plane: Design Principles And The Architecture

Before writing a line of code, I settled on five principles. Every later decision came back to one of them.

1. Keep the control plane off the hot path. If the control plane is down, agents must keep running on their last known configuration and policy. A control plane outage must never become an outage of every agent.
2. Enforce at the gateways. Agents can be wrong, buggy, or manipulated by a prompt injection. A gateway sees every call regardless. So rules are enforced where traffic passes, not inside each agent’s code.
3. Git is the source of truth. Every agent definition, prompt version, and policy lives in Git. If it is not in Git, it does not exist in production.
4. Same building blocks, different strictness. One registry, one set of gateways and one policy engine, with strictness set by risk tier.
5. Use what we already run. Kubernetes, Argo CD, Databricks, MLflow, and OpenTelemetry were already in the stack. A control plane that needs a new platform team to operate it will not survive.

With those principles, the architecture fell out naturally.

The control plane configures and watches; only the gateways sit in the request path

The control plane at the top never handles a user request. It holds the desired state, pushes configuration and policy down, and collects traces, costs, and evaluation results back up. The only control-plane pieces in the request path are the two gateways, and they work from a local copy of config and policy, so they keep enforcing even if the control plane itself is briefly unavailable.

Agent registry: every agent becomes a Kubernetes resource

The registry was the first thing I built, and the most important. You cannot govern what you cannot list.

I considered a database with a small UI. I went with a Kubernetes custom resource definition (CRD) instead, for one reason: our agents already run on Kubernetes, and Kubernetes is already very good at storing desired state, versioning it in Git through Argo CD, and running controllers that act on it. An agent became just another resource, like a Deployment.

Every agent has one manifest. Here is a trimmed version of one of the decision agents:

apiVersion: aicp.internal/v1
kind: Agent
metadata:
  name: value-optimizer
  namespace: decisions
  labels:
    system: agentic-decisions
    riskTier: "1"
spec:
  owner: platform-architecture
  version: 3.4.0
  image: registry.internal/agents/value-optimizer:3.4.0
  model:
    primary: claude-sonnet
    fallback: claude-haiku
  prompt:
    registry: mlflow
    name: value-optimizer-system
    version: 12
  tools:                       # the only MCP tools this agent may call
    - decisions.get_market_data
    - decisions.get_impact_model
    - decisions.propose_change
  dataScopes: [decisions.curated.read]
  budget:
    tokensPerDay: 4000000
    maxTokensPerRun: 60000
  approvals:
    humanApprovalAboveChangePct: 5
  evals:
    suite: decisions-golden-v7
    minPassRate: 0.95
  killSwitch: fals

A few fields do most of the work:

• tools is an allow-list. The MCP gateway reads it and rejects any tool call not on the list. A prompt injection that convinces the agent to call something else gets a polite denial at the gateway.
• prompt points at a specific version in the MLflow prompt registry, which we already used on Databricks. The prompt is versioned just like the code.
• budget gives the LLM gateway the limits to enforce per agent.
• evals names the golden-set suite the agent must pass before it can be promoted.
• riskTier tells every other component how strict to be.

A small controller watches these resources and keeps the gateways and the cockpit in sync. When someone adds an agent, it appears in the cockpit with its owner and tier the moment the manifest is merged. When someone deletes the manifest, the gateways stop accepting calls from it.

The registry also solved a problem I had not planned for: ownership. Every agent has a named owning team. When an alert fires, there is no meeting to work out whose agent it is.

Versioning and promotion: CI/CD for agents

With ordinary software, a new version is a new build. With an agent, a new version can be a new prompt, a new model, a new tool, or a new threshold, and any one of them can change behaviour as much as a code change. Early on, prompts were edited in place and went live immediately. When an answer changed, nobody could say why.

So I treated every one of those things as part of the agent’s version. A change to any of them is a pull request against the agent’s manifest, and it goes through the same path to production.

The steps:

1. Pull request. The manifest change and the prompt version it points to are reviewed like code. For tier 1 agents, the review needs someone from the business side as well as engineering.
2. Golden-set evaluation. Argo CD deploys the new version to staging, and a job runs its eval suite. Each suite is a set of real questions or scenarios with known good outcomes: for the decision agents, historical cases with the correct recommendation; for the domain RAG, questions with expected sources; for the pipeline builder, requirements with pipelines that must validate. Results are logged to MLflow, so every version has a score attached.
3. The gate. If the pass rate is below the minPassRate in the manifest, the release stops and goes back to the owner. No exceptions for tier 1.
4. Canary. Argo Rollouts sends a small share of real traffic to the new version. The same online scorers that run in production compare it with the live version.
5. Full rollout or automatic rollback. If online scores hold, traffic shifts fully. If they drop, the rollout aborts and traffic returns to the previous version without anyone being woken up.

The biggest change was cultural rather than technical. “Can I just tweak the prompt?” now has a clear answer: yes, through a pull request, and the eval suite will tell you whether your tweak helped. Rolling back is no longer an emergency; it is choosing the previous version in Git.

Runtime configuration and the kill switch

Some changes cannot wait for a release. A model provider has an outage. An agent starts looping and burning tokens. A tool’s backend is returning bad data. In those moments, “open a pull request and wait for the eval suite” is the wrong answer.

So the control plane separates two kinds of change:

• Versioned changes (prompt, tools, thresholds, eval suite) go through promotion, as described above.
• Operational switches take effect within seconds, without a deployment, and every use is logged.

The operational switches are deliberately few:

The gateways read these values from a small configuration service and cache them locally. If the configuration service is unreachable, the gateways keep the last known values. That follows the first principle: a control plane failure must not stop agents that were working.

The kill switch deserves special mention. Before the control plane, stopping one misbehaving agent meant scaling its Deployment to zero, which often took other agents in the same service down with it, and needed someone with cluster access. Now it is a single switch that the gateways enforce: the agent’s next LLM call and next tool call are refused with a clear reason, and its in-flight runs end cleanly. For the decision agents, we use read-only mode far more often than the full kill switch. It keeps analysis running while making sure nothing is submitted until a person has looked.

The two gateways: where the rules are enforced

Everything an agent does that matters goes through one of two doors: it calls a model, or it calls a tool. So those are the two places I put enforcement. Agents are not allowed to call a model provider or a backend directly; network policies in the cluster block it.

The LLM gateway

Every model call from every agent goes through one internal endpoint. It does six jobs:

• Routing and fallback. The agent asks for a model by role, not by provider. The gateway picks the model from the registry, and switches to the fallback if the primary fails or a runtime switch says so.
• Budgets. Tokens are counted per call, per run, and per day against the agent’s manifest. A runaway loop hits its per-run limit and stops, instead of running all night.
• Caching. Repeated embeddings and identical requests are served from cache. In the domain RAG system, this alone cut embedding calls per query by 5x, because the planner kept running the same searches.
• PII redaction. Inputs are scanned before they leave the cluster, and sensitive values are masked.
• Guardrails. Jailbreak and prompt-injection checks on inputs, and format and content checks on outputs. More on these below.
• Telemetry. Every call emits a span with the agent, model, tokens, latency, and cost.

The MCP gateway

I wrote separately about putting all our enterprise sources behind MCP servers. In the control plane, the MCP gateway became the main enforcement point, because tool calls are where agents actually act on the world.

Each tool call goes through five checks, in order:

1. Identity. The call carries a token for the user the agent is working for. The gateway validates it and exchanges it on behalf of that user for a token scoped to the target server, so data access always runs with the user’s own permissions.
2. Registry. Is this tool on this agent’s allow-list? If not, deny.
3. Switches. Is the agent killed, in read-only mode, or over budget?
4. Policy. The policy engine looks at the actual arguments, for example the size of a proposed change, and returns allow, deny, or hold for human approval.
5. Trace. Whatever the outcome, the gateway records a span and an audit entry.

The important property is that none of this depends on the agent behaving well. An agent that has been confused by a malicious document can ask for anything it likes. It still only gets what its manifest and the policy allow.

Policy as code, and a human in the loop for high-risk decisions

At first, rules lived as if statements inside each agent. The decision agents checked their own limits; the pipeline builder checked its own environments. That meant every rule existed in three slightly different versions, and changing one meant three releases.

I moved them into Open Policy Agent (OPA), a general-purpose policy engine that is common in Kubernetes shops. Policies are written in a language called Rego, stored in Git, tested like code, and shipped to the gateways as bundles. The gateways ask OPA for a decision on every sensitive call, and OPA answers locally in a fraction of a millisecond, so it adds no meaningful latency.

Here is a simplified version of the rule for proposed changes:

package aicp.tools.decisions
import rego.v1
default decision := {"result": "deny", "reason": "no matching rule"}
# Small changes above the safety floor go straight through.
decision := {"result": "allow"} if {
    input.tool == "decisions.propose_change"
    abs(input.args.change_pct) <= data.agents[input.agent].approvals.humanApprovalAboveChangePct
    input.args.new_value >= data.limits.safety_floor
}
# Larger changes are held for a person to approve.
decision := {"result": "hold", "reason": "change above approval threshold"} if {
    input.tool == "decisions.propose_change"
    abs(input.args.change_pct) > data.agents[input.agent].approvals.humanApprovalAboveChangePct
    input.args.new_value >= data.limits.safety_floor
}
# Nothing may go below the safety floor, approved or not.
decision := {"result": "deny", "reason": "below safety floor"} if {
    input.tool == "decisions.propose_change"
    input.args.new_value < data.limits.safety_floor
}

The threshold comes from the agent’s manifest, so tightening it for one agent is a one-line change in the registry. The safety floor comes from shared data that the business owns.

What “hold” means in practice

A hold has to land somewhere a person will see it. Our decision agents are LangGraph graphs, and LangGraph has a built-in way to pause: a node calls interrupt(), the graph saves its state to a checkpoint, and it waits until it is resumed with Command(resume=...) on the same thread.

from langgraph.types import interrupt, Command
def submit_change(state: DecisionState):
    decision = mcp.call("decisions.propose_change", state["proposal"])
    if decision.result == "hold":
        answer = interrupt({
            "type": "change_approval",
            "item": state["proposal"]["item_id"],
            "change_pct": state["proposal"]["change_pct"],
            "evidence": state["evidence"],      # why the agent proposed it
        })
        if not answer["approved"]:
            return {"status": "rejected", "note": answer.get("note")}
        decision = mcp.call("decisions.propose_change",
                            {**state["proposal"], "approval_id": answer["approval_id"]})
    return {"status": decision.result}

The interrupt payload appears in an approval queue in the cockpit, with the agent’s evidence attached. The reviewer approves or rejects with one click, the graph resumes, and the approval ID travels with the second call so the policy can verify that a real approval exists. One caveat from the LangGraph docs is worth repeating: when a graph resumes, the node runs again from the top, so anything before the interrupt() must be safe to repeat.

Guardrails and responsible AI

Policy decides what an agent is allowed to do. Guardrails check what goes into and comes out of the model. I keep them in the LLM gateway so every system gets them without writing any code.

On the way in:

• PII redaction. Customer names, emails, phone numbers, and account identifiers are detected and masked before a prompt leaves the cluster. The mapping stays inside the gateway, so the response can be unmasked for the user if they are allowed to see it.
• Prompt-injection and jailbreak checks. This matters most for the domain RAG system, because it reads Confluence pages, Jira tickets, and documents that anyone can edit. Retrieved content is treated as data, not instructions: it is wrapped and labelled as untrusted, and a classifier flags text that looks like it is trying to give the model orders.
• Content safety. A standard safety filter on user input, mostly as a backstop.

On the way out:

• Schema validation. Agents that return structured output, like a change proposal or a pipeline spec, must match a JSON schema. Anything that does not parse is rejected and retried, not passed downstream.
• Grounding checks. For the domain RAG system, every claim must cite a source that was actually retrieved in that run. The reviewer agent does this inside the system; the gateway enforces a simpler version: no citations, no answer.
• Domain rules. For the pipeline builder, generated code is checked for things it must never contain, such as writes to production schemas from a development job or hard-coded credentials.

I learned to be careful with guardrails in two ways. First, every check adds latency, and some add cost, so they are set per tier: the domain RAG system gets the full input checks because it reads untrusted content, while internal calls from the decision agents skip some of them. Second, a guardrail that blocks silently is worse than none, because people stop trusting the system. Every block returns a clear reason, and every block is logged so we can review false positives every week.

This is also where model risk management comes in. Each agent’s manifest records which model it uses and which eval suite approved it. When a model provider ships a new version, we know exactly which agents are affected, and the promotion flow decides whether they move to it.

Observability: one trace from question to answer

Before the control plane, each system logged in its own way. Tracing one recommendation meant reading logs from three services and matching timestamps by hand.

Now every request gets one trace ID at the entry point, and it travels everywhere: through each agent, the LLM gateway, the MCP gateway, and the MCP servers. I standardised on OpenTelemetry, using its semantic conventions for generative AI. These define common span names and attributes, so every tool that understands them can read our traces:

• invoke_agent {agent name} spans for each agent step
• chat {model} spans for each model call, with gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
• execute_tool {tool name} spans for each tool call, with gen_ai.tool.name

On top of the standard attributes, we add our own: the agent’s version from the registry, its risk tier, the prompt version, the policy decision for each tool call, and the cost of each model call.

One honest note: the OpenTelemetry GenAI conventions are still marked as in development, and attribute names have changed between releases. I put a thin wrapper around them in a shared library, so when names change we update one place, not every agent.

Traces go to the collector in the cluster, then to our tracing backend. A copy of the model-call spans also lands in Delta tables on Databricks, because that is where the cost and evaluation analysis happens.

The most useful view this created is cost per agent per day. We had always known roughly what we spent on LLMs. We had never known which agent spent it, or on what. The first week the dashboard was live, the answer surprised everyone: the biggest spender was not the busiest agent but the one with the longest retrieved context. That finding led straight back to the deduplication work in the RAG system.

Evaluation and drift: knowing when quality changes

Observability tells you what happened. Evaluation tells you whether it was any good. The control plane runs evaluation at three points.

1. Offline, before release: golden sets. Each agent has a suite of cases with known good outcomes, kept in Git next to the manifest and run by the promotion flow. I treat these suites as living things. Every production incident and every case a reviewer marks as wrong becomes a new golden case, so the same mistake cannot ship twice.

2. Online, in production: sampled scoring. A share of live traffic is scored continuously. Each system has metrics that fit what it does:

Some of these are deterministic checks, which are cheap and reliable. Others use an LLM as a judge, which is flexible but costs money and can itself drift. I learned to prefer deterministic checks wherever one exists, and to keep LLM judges for questions only a model can answer, like whether an answer is supported by its sources. MLflow’s GenAI evaluation tooling handles both kinds, and since we already used MLflow on Databricks, scores for offline and online runs end up in the same place.

3. Over time: drift detection. Agents drift even when nobody changes them. Data changes, user questions change, and model providers update models behind the same name. The control plane compares each week’s online scores and input patterns against a baseline, and alerts the owning team when a metric moves beyond its band. Drift alerts are what most often trigger a new golden-set case and a new release.

A/B harnesses tie it together. When we want to compare two prompts or two models, the canary mechanism from the promotion flow splits traffic, and the online scorers compare the two versions on the same live questions. That replaced a lot of opinion with evidence.

Audit and Governance: Evidence, Not Dashboards

A dashboard shows what is happening now. An auditor asks what happened eight months ago, and why. Those are different problems.

For tier 1 agents in particular, the control plane writes an audit record for every decision that matters: every change proposed, every policy decision, every hold, approval or rejection, and every change to an agent’s version or configuration. Each record carries the trace ID, the agent version, the prompt version, the policy version that made the decision, and the identity of the user or reviewer involved. Records go to an append-only Delta table with Unity Catalog permissions, so the people who run the agents cannot quietly edit the history.

The practical test I use is this: pick any change an agent proposed months ago and answer, in a few minutes, which agent proposed it, which version, using which prompt and data, which policy allowed it, and who approved it. Before the control plane, that took days and some guesswork. Now it is a query.

I also mapped the control plane to the frameworks our risk and compliance colleagues already use, rather than inventing a new vocabulary:

• The NIST AI Risk Management Framework organises AI risk work into four functions: Govern, Map, Measure and Manage. The registry and ownership model support Govern, risk tiers support Map, evaluation and drift support Measure, and policies, switches and rollback support Manage.
• The EU AI Act expects documentation, transparency and human oversight for higher-risk AI systems. Even where it does not apply directly, human approval for large changes and a complete decision trail are the kind of controls it describes.

The registry made one more governance task easy: an up-to-date inventory. Every agent, its owner, its tier, its model, and the data it can touch, generated from the manifests. That list used to be a spreadsheet that was out of date the day it was written.

The Cockpit: One Screen For Every Agent

All of this would be much less useful if it lived in five different tools. The cockpit is the single place people go. It is deliberately simple: a set of dashboards over the trace store and the Delta tables, plus two small pages for approvals and switches.

It answers different questions for different people:

• Fleet view, for architects and leads. Every registered agent with its system, owner, tier, live version, health, today’s spend against budget, and current eval scores. Red means something needs a human.
• Agent view, for owning teams. Version history with the eval score of each release, online metrics over time, drift alerts, top errors and the most expensive runs. One click opens a full trace.
• Trace view, for debugging. The whole tree for one request: which agents ran, each model call with tokens and cost, each tool call with its policy decision, and where time was spent.
• Approval queue, for business reviewers. Held changes with the agent’s evidence and the policy reason. Approve or reject, with a note.
• Switchboard, for on-call. The operational switches from earlier, each with who changed it, when, and why.
• Audit view, for risk and compliance. Search the decision log by item, date, agent or reviewer, and export the evidence.

The approval queue was the page that won the business over. Before, a large change proposed by an agent either went through unchecked or was caught by someone reading reports afterwards. Now the right person sees it before it happens, with the reasoning in front of them.

What Went Wrong Along The Way

Not everything worked the first time. These are the mistakes that cost me the most, so you can skip them.

I started with the dashboard. My first instinct was to build the cockpit, because it is the visible part. But a dashboard over inconsistent data is just a nicer way to be confused. The registry and a shared tracing library should come first; the cockpit is easy once the data is clean.

I put too much in the request path. An early version had the gateways call the control plane for every decision. When the configuration service restarted during an upgrade, every agent paused with it. That is where principle one came from: gateways now work from a local copy and refresh in the background.

Too many policies, too early. Once policy as code worked, everyone wanted a rule for everything, and the false positives started. A guardrail that blocks good requests teaches people to look for ways around it. We now add a policy only when it maps to a real risk or a real incident, run new policies in log-only mode for a while, and promote them to enforce when the logs show they behave.

Evaluation got expensive. Scoring every request with an LLM judge cost more than some of the agents being judged. We moved to sampling, used deterministic checks wherever possible, and kept full scoring only for tier 1.

Traces got huge. A single multi-agent request can produce hundreds of spans, each with prompts and responses attached. We now keep full payloads only for a short period and for sampled or failed requests, and keep the lightweight span data longer.

Teams tried to bypass the gateways. When a deadline looms, calling the model provider directly is tempting. Network policies now block direct egress from agent namespaces, and the shared client library makes using the gateway the easiest path, not just the required one.

The registry drifted from reality. For a while, some agents ran in the cluster that had no manifest, or a manifest that did not match what was deployed. The controller now reconciles both ways: an agent without a valid manifest cannot get a token from the gateways, so it simply does not work.

How it helps today, build vs buy, and how to roll it out

What changed

Go back to the questions from the start of this article. Each one now has a fast answer:

The less visible change matters just as much. Adding a new agent used to mean rebuilding auth, retries, logging and limits. Now a team writes the agent, writes a manifest and an eval suite, and gets identity, policy, budgets, tracing and the cockpit for free. Governance stopped being something that slowed teams down and became the fastest way to production.

Build Or Buy?

I would not tell everyone to build. If your agents run mostly on one cloud’s agent platform, that platform’s own control plane (Microsoft Foundry Control Plane, Amazon Bedrock AgentCore, Google’s Gemini Enterprise Agent Platform or Databricks AI Gateway) will get you most of the way faster than anything you build. Commercial AI observability and guardrail products are also strong for the evaluation and monitoring parts.

Building a thin layer made sense for us because our agents span Kubernetes, Azure, and Databricks, use LangGraph rather than one vendor’s runtime, and needed rules tied to our own business data, like safety floors. Most of what we “built” is configuration and glue around open-source parts we already ran: Kubernetes CRDs, Argo CD and Argo Rollouts, OPA, OpenTelemetry and MLflow. The custom code is mainly the two gateways, the controller, and the cockpit pages.

A reasonable middle path is to build the parts that encode your business (registry, policies, risk tiers) and buy or adopt the parts that are generic (tracing backend, guardrail models, evaluation tooling).

A Phased Rollout

If I did it again, I would roll it out in this order:

1. Phase 1: see everything. Agent registry, the shared tracing library, and the LLM gateway with cost tracking. Low risk, immediate value, and every later phase depends on it.
2. Phase 2: control the risky parts. The MCP gateway will allow lists and on-behalf-of identity, policies in log-only mode, then enforce mode for tier 1, and eval-gated promotion.
3. Phase 3: close the loop. The approval queue, operational switches, online scoring, drift alerts, and automatic rollback.

Each phase is useful on its own, so the work pays for itself before it is finished.

Closing Thoughts

The hardest part of building agents is no longer getting one to work. It is running many of them safely, cheaply and accountably, when each one can call tools, spend money, and make decisions on its own.

The answer I arrived at is not new. It is the same idea that made Kubernetes manageable: separate the work from the management of the work. Let the agents do the work in the data plane. Put registration, versioning, configuration, policy, evaluation, and evidence in a control plane beside them. Enforce the rules at the gateways that every call must pass through, and never let the control plane itself become a single point of failure.

If you are starting out, start small: a registry and one trace ID per request will teach you more about your agents in a week than months of reading logs.

In a future article, I will go deeper into the evaluation side and how we build and maintain golden sets for agents whose correct answer changes over time. If you are building something similar, I would like to hear how you handled policy and human approval. Leave a comment.

Sources

• Microsoft Foundry Control Plane overview
• Amazon Bedrock AgentCore FAQs
• Databricks AI Gateway documentation
• Google launches Gemini Enterprise Agent Platform for governance
• OpenTelemetry GenAI semantic conventions guide
• LangGraph interrupts documentation
• NIST’s AI Risk Management Framework explained (Cloud Security Alliance)