MCP in Depth: Turning a Multi-Agent Domain RAG System into MCP Servers on Kubernetes

Written By Animesh Kumar Sinha/ Technical Architect

How I put every enterprise source behind MCP, ran the servers on Kubernetes, and kept user identity intact end to end

The problem: one question, many places to look

Here is the kind of question that lands on an architect’s desk in a large enterprise data domain almost every week:

“Why did this metric change last Tuesday, which job changed it, and what else does that job touch?”

To answer it properly, somebody has to open Confluence for the business rule, Jira for the change ticket, GitHub for the job code, Unity Catalog for the table and its owner, the data lake for the actual rows, and the lineage graph to see what sits upstream and downstream. If the logic is old enough, there is also an Excel workbook with a VBA macro that nobody wants to admit is still part of the flow.

The knowledge exists. It is just scattered across multiple sources, and only a few people know which one to open first.

So I built a multi-agent domain RAG system. A planner agent breaks the question down and searches multiple enterprise sources in parallel. Then a writer and a reviewer turn the evidence into one grounded answer, with links. The stack is Python, LangGraph, ArcadeDB, Flask, and Databricks.

The retrieval turned out to be the easy part. The hard part was the plumbing. Every agent needed its own connector to every source, with its own authentication, its own retries, and its own way of describing what it can do. With N agents and M sources, you end up maintaining N × M integrations. That is when I moved every source behind MCP and deployed the servers on Kubernetes.

This article walks through MCP end to end, using that system as the running example: what the protocol is, its primitives, its transports, running servers on Kubernetes, putting a gateway in front of them, and securing all of it so a user can never see data through the agent that they couldn’t see directly. Each part has a diagram and real code.

What is MCP (Model Context Protocol)?

MCP, the Model Context Protocol, is an open standard that Anthropic introduced in November 2024 and that now lives under the Linux Foundation’s Agentic AI Foundation. It defines one way for an AI application to discover and call tools and to read data, no matter who built the tool.

In my Java days, JDBC let me write code against one interface and swap Oracle for DB2 without touching the application. MCP does the same job for agents. My planner doesn’t care whether it is talking to Confluence or to a graph database. It speaks MCP, and the server deals with the rest. Also read, “The real challenge isn’t building one brilliant agent. It’s teaching a team of focused agents to think together — without ever speaking to each other directly.” at https://journals-times.com/2026/06/07/orchestrating-intelligence-a-multi-agent-research-system-with-coordinator-pattern-mcp-integration-and-quality-gated-pipelines/

There are three roles, and the naming confuses almost everybody at first:

• Host: the AI application the user works in. In my system, it is the LangGraph agent runtime. For some engineers, it is Claude Desktop or VS Code.
• Client: a connector that lives inside the host. The host creates one client per server, and each client talks to exactly one server.
• Server: a program that exposes the capabilities of one system. I have a domain-search server, a lineage-graph server, a confluence server and so on.

The host owns the conversation and the model. Servers never talk to each other, and they never see the full conversation. They only see the calls made to them.

Every message is JSON-RPC 2.0. Until the 2026-07-28 revision of the spec, every connection started with a handshake: initialize, a reply with the server’s capabilities, then an initialized notification. That is gone now. MCP is stateless: every request stands on its own and carries its protocol version and the client’s capabilities in _meta. If a client wants to know what a server supports before calling it, it asks with server/discover.

Here is a tool call under the current spec:

{"jsonrpc": "2.0", "id": 7, "method": "tools/call",
 "params": {"name": "search_domain",
            "arguments": {"query": "why did this metric change last Tuesday", "top_k": 8},
            "_meta": {"io.modelcontextprotocol/protocolVersion": "2026-07-28",
                      "io.modelcontextprotocol/clientCapabilities": {"elicitation": {}}}}}

Before calling anything, the client asks tools/list to learn what the server offers, and it can cache the answer: list results now carry ttlMs and cacheScope hints that say how long the list stays valid.

This change matters a lot if you run servers at scale. When a connection was a session, a load balancer had to send every request from a client to the same pod. Now any pod can answer any request, which is exactly what Kubernetes wants. If you read an older MCP tutorial that starts with initialize, it was written for the previous revision.

The primitives: what a server offers, and what it can ask for

Servers offer three primitives. On the client side, the one that matters now is elicitation. Sampling and roots still work, but the 2026-07-28 spec deprecated them, so I don’t build on them. For each primitive, the useful question is: who decides when it gets used?

Tools are actions the model can choose to call. They take typed arguments and return results. Most of the value in my system lives here.

Resources are read-only data addressed by a URI. The model doesn’t call them; the host decides what to load into context. I use resources for reference material rather than search results: the approved definition of a glossary term, or the schema of a table.

Prompts are reusable templates the user picks, usually as a slash command. explain_metric_change Packages the questions an analyst always asks into one starting point.

On the client side, elicitation lets a server ask the user a structured question. Under the stateless spec, there is no open channel for the server to send its own request, so the tool returns a input_required result instead, and the client calls the tool again with the answer. The spec calls this the multi-round-trip request pattern. Sampling (borrowing the host’s LLM) and roots (file boundaries) are deprecated, along with protocol-level logging.

Here is a cut-down version of the domain search server, written with v2 of the Python MCP SDK, where FastMCP was renamed MCPServer:

from mcp.server.mcpserver import MCPServer
from mcp.types import ToolAnnotations
from domain.retrieval import HybridRetriever
from domain.glossary import GlossaryStore
mcp = MCPServer("domain-search")
retriever = HybridRetriever()
glossary = GlossaryStore()
@mcp.tool(annotations=ToolAnnotations(read_only_hint=True, open_world_hint=False))
async def search_domain(query: str, top_k: int = 8) -> list[dict]:
    """Search domain knowledge: rules, jobs, tables, tickets, code and docs.
    Use for 'what', 'why' and 'where' questions about how the domain works.
    Do NOT use for live row counts or current values; use run_readonly_sql.
    Every result includes a source URI so the answer can cite it.
    """
    hits = await retriever.search(query, top_k=top_k)
    return [{"text": h.text, "source": h.uri, "score": h.score} for h in hits]
@mcp.resource("glossary://{term}")
def glossary_term(term: str) -> str:
    """The approved business definition of a business term."""
    return glossary.definition(term)
@mcp.prompt()
def explain_metric_change(metric: str, date: str) -> str:
    """Investigate why a metric changed on a given date."""
    return (
        f"Explain why the metric {metric} changed on {date}. "
        "Find the rule, the change ticket, the job that applied it, "
        "and the upstream tables involved. Cite every source."
    )
if __name__ == "__main__":
    mcp.run(transport="streamable-http")

Two lessons from this file.

First, the docstring is the most important line in it. The model reads a tool’s description to decide whether to call it, so I write descriptions for the model, not for developers: what the tool is for, and when not to use it. readOnlyHint tells the client the tool changes nothing, and because the function returns typed data, the SDK sends structured content next to the text.

Second, I expose one coarse search_domain tool, not separate tools for semantic search, BM25, and graph expansion. The agent should decide what to look for, not how retrieval works inside. Exposing those internals only gives the model more ways to get it wrong.

The system: plan, fan out, write, review

The agent side is a LangGraph state graph with three agents and one parallel step.

1. The planner rewrites the question, splits it into sub-questions, and picks which sources each one needs.
2. The fan-out calls one MCP tool per source, all in parallel: the domain index, GitHub, SQL schemas, Confluence, Jira, Unity Catalog, the data lake, and the lineage graph.
3. The writer drafts an answer from the evidence, with a citation on every claim.
4. The reviewer checks each claim against the evidence it cites. If a claim isn’t supported, it sends the question back to the planner with a note on what is missing.

In LangGraph, the fan-out is a few lines with the Send API. Each Send becomes its own branch, and LangGraph waits for all of them before the writer runs:

import operator
from typing import Annotated, TypedDict
from langgraph.types import Send
class SearchState(TypedDict):
    question: str
    plan: list[PlanStep]
    # Every parallel branch appends here; the reducer merges instead of overwriting.
    evidence: Annotated[list[dict], operator.add]
def fan_out(state: SearchState) -> list[Send]:
    return [
        Send("call_source", {"source": step.source, "query": step.query})
        for step in state["plan"]
    ]
graph.add_conditional_edges("planner", fan_out, ["call_source"])
graph.add_edge("call_source", "writer")

The call_source node is where MCP comes in. It knows nothing about Confluence or ArcadeDB. It looks up which MCP server owns the source, calls the tool, and returns its results under evidence. Because evidence has the operator.add reducer, the results from every branch are merged instead of overwriting each other. That bug is easy to miss when you test with only one source. Adding a new source means deploying one more server and registering it, with no change to the agent code.

If I had to throw away everything except one piece, I would keep the reviewer. A RAG system without one will happily give a confident answer built on the wrong Confluence page.

Inside the domain search server: hybrid retrieval and GraphRAG

Most of the work behind search_domain solves one problem: in an enterprise domain, the words people use and the words in the documents rarely match.

Query rewriting. An LLM expands acronyms, adds synonyms from the business glossary, and splits compound questions. “Why did the DQ score drop on cust_dim” becomes a question about the data quality score of one specific table, plus a sub-question about the job that changed it.

Semantic search. Chunks are embedded as 3072-dimension vectors. This finds content that means the same thing in different words, such as a Confluence page that says “duplicate records” when the user said “repeated rows”.

BM25 keyword search. Embeddings are poor at exact identifiers. A table name, a job name or a Jira key like ABC-4471 is a string, not a meaning. BM25 finds those exact matches, so I run it next to semantic search, not instead of it.

Fusion. The two ranked lists are merged with reciprocal rank fusion (RRF). A document scores well if it ranks high on either list, and best if it ranks high on both:

score(d) = Σ  1 / (k + rank_i(d))     for i in {semantic, bm25}

Here k is a smoothing constant; 60 is the usual starting point. I like RRF because it works on ranks, not raw scores, so I never have to calibrate cosine similarity against BM25 scores.

GraphRAG expansion. This is where ArcadeDB earns its place. The top hits become seed nodes in a graph that holds both lineage (which table feeds which) and the glossary (which term defines which column). The server walks one or two hops out from each seed and pulls in connected context that text search alone would never find:

MATCH (t:Table {name: $table})<-[:FEEDS*1..2]-(upstream:Table)
OPTIONAL MATCH (t)-[:HAS_COLUMN]->(c:Column)-[:DEFINED_BY]->(g:GlossaryTerm)
RETURN upstream.name, c.name, g.name, g.definition

A question about a reporting table now comes back with the upstream source table that feeds it and the business definition of the column that changed. Very often, that is the answer.

Reranking. Finally, a reranker scores every remaining chunk against the original question and keeps the best few. Fusion and graph expansion are good for recall; reranking brings the precision back.

None of these steps are visible to the agent. From the planner’s point of view, it called one tool and got back a handful of good chunks, each with its source.

Transports: stdio for your laptop, Streamable HTTP for the cluster

MCP messages need a way to travel, and the spec defines two standard transports. A third, older one still shows up in blog posts, so it is worth knowing why it went away.

With stdio, the host launches the server, and they talk over standard input and output. It is perfect while you build: no ports, no auth, nothing to deploy. Most people meet MCP this way, through a config file in Claude Desktop.

Streamable HTTP is what I run in production. The client POSTs every JSON-RPC message to a single endpoint. For a quick tool call, the server replies with plain JSON. For a long one, it can turn that same response into a Server-Sent Events stream and push progress updates before the final result. For change notifications, the client sends one long-lived subscriptions/listen request instead of the old GET stream.

POST /mcp HTTP/1.1
Host: mcp.internal.example.com
Content-Type: application/json
Accept: application/json, text/event-stream
Authorization: Bearer eyJhbGciOi...
MCP-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: search_domain
{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"search_domain","arguments":{"query":"..."},"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28"}}}

Two headers in that request matter for infrastructure. Mcp-Method and Mcp-Name Repeat the method and the tool name outside the body, so a gateway or load balancer can route the call without parsing JSON. And there is no Mcp-Session-Id anymore, because the 2026-07-28 spec removed sessions from the protocol.

If a tool needs to remember something between calls, it returns an explicit handle and accepts that handle as an argument on the next call. Any pod can answer any request, and that single change made the Kubernetes part easy.

Running MCP servers on Kubernetes

Each MCP server is its own Deployment. I split servers by source, not by team: domain-search, lineage-graph, atlassian for Confluence and Jira, and databricks for Unity Catalog and read-only SQL. A slow Confluence API should never take graph search down with it.

A few choices did most of the work:

• Stateless by design. With no protocol sessions, pods sit behind a plain Service with no sticky sessions or session affinity. Even elicitation works across pods: the tool returnsinput_required, the client calls again with the answer, and whichever pod receives the retry can finish the job.
• Probes and autoscaling. The readiness probe checks the backend connection, not just the process. Each server scales on its own load, so a burst of graph queries never scales up the Jira server.
• NetworkPolicy. Only the gateway can reach the MCP pods. Nothing else in the cluster can call a tool directly.
• One Helm chart. Every server uses the same chart. A new server is just a values file: the image, the secrets it needs and the team that owns it.

# values-lineage-graph.yaml
name: lineage-graph
image: registry.internal/mcp/lineage-graph:1.6.2
transport: streamable-http
port: 8080
path: /mcp
replicas: {min: 2, max: 6}
readiness: {path: /healthz, checkBackend: arcadedb}
secrets:
  - arcadedb-readonly
networkPolicy:
  allowFrom: [mcp-gateway]
owner: data-architecture

The /healthz route is a small HTTP handler mounted next to the MCP endpoint. It runs a cheap query against the backend and returns 503 if that fails, so Kubernetes stops sending traffic to a pod that can’t actually answer.

The MCP gateway: one door for every server

With a handful of servers, agents can keep a list of URLs. As the number grows, that list becomes the problem: every agent hard-codes endpoints, every server repeats the same auth code, and nobody can see all the traffic in one place. So I put a gateway in front.

The gateway does five jobs:

1. Registry. Every server is registered with its name, version, owner, endpoint, and the tenants allowed to use it. The registry also tracks each server’s health and its current tool list.
2. Discovery. A client connects to the gateway as if it were one big MCP server. When it calls tools/list, the gateway merges the tool lists of every server this user’s tenant may use, and prefixes each name: lineage.query_lineage, atlassian.search_confluence.
3. Routing. Ontools/call, the prefixed tool name arrives in the Mcp-Name header, so the gateway knows which server owns the tool without parsing the body. It strips the prefix and forwards the call.
4. Transport bridging. Some useful servers only speak stdio. The gateway runs them as child processes and exposes them over Streamable HTTP, so clients only ever see one transport.
5. Multi-tenancy and audit. A tenant is a business domain. Users in one domain see that domain’s tools; another tenant sees a different set. Every call is logged with the user, tenant, tool, latency, and a trace ID.

Filtering tools per tenant has a benefit people miss: it keeps the tool list short. Every tool description is sent to the model and takes up context. Fifty descriptions for tools the user can’t even use cost tokens on every turn and make the model worse at picking the right one.

Stateless servers can’t push messages between calls, so the gateway doesn’t wait to be told about new tools. When a server is deployed, its registry entry is updated, and the gateway refreshes its merged tool list; for everything else it follows the ttlMs cache hint on tools/list. Clients that want to hear about changes keep one subscriptions/listen request open to the gateway, so a team can ship a new tool with no redeploy on the agent side.

Security: OAuth 2.1 and the user’s identity, end-to-end

One rule shaped every security decision: a user must never see data through the agent that they couldn’t see directly. If someone has no access to a restricted table in Unity Catalog, asking the agent nicely should not change that.

MCP’s authorization spec, built on OAuth 2.1, gives you most of what you need, as long as you read it carefully.

The gateway is the OAuth resource server. Clients only ever talk to the gateway, so that is where authorization starts. When a client calls without a token, the gateway answers 401 with a WWW-Authenticate header pointing to its protected resource metadata (RFC 9728). The client reads that metadata, finds the authorization server (Entra ID in my case, Ping in some setups), and runs the authorization code flow with PKCE. The 2026-07-28 revision also prefers Client ID Metadata Documents over dynamic client registration and requires clients to validate the iss parameter, which closes off mix-up attacks.

Every token has one audience. The client asks for a token with a resource parameter that names the gateway (RFC 8707), so the token is valid for the gateway and nothing else. The gateway validates it, then exchanges it on behalf of the user for a second token whose audience is the one MCP server it routes to. Each server still rejects any token that wasn’t issued for it, and no token is ever valid in two places.

No token passthrough. This one is easy to get wrong. The spec forbids an MCP server from forwarding the token it received to downstream APIs. So the server does a second on-behalf-of exchange: it trades its token for a new Databricks token issued for the same user. Databricks then applies Unity Catalog permissions as that user, row filters and column masks included.

The highlighted steps are the important part. Every hop swaps the token for a new one meant only for the next hop; nothing is ever forwarded as it is. In code, the exchange the MCP server makes before it touches Databricks is one call:

import msal
cca = msal.ConfidentialClientApplication(
    client_id=settings.client_id,
    client_credential=settings.client_secret,
    authority=f"https://login.microsoftonline.com/{settings.tenant_id}",
)
AZURE_DATABRICKS = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default"
def databricks_token_for(user_token: str) -> str:
    result = cca.acquire_token_on_behalf_of(
        user_assertion=user_token,
        scopes=[AZURE_DATABRICKS],
    )
    if "access_token" not in result:
        raise PermissionError(result.get("error_description", "OBO exchange failed"))
    return result["access_token"]

Two more layers sit around this. Inside the cluster, the gateway and the servers talk over mTLS, so a stolen token alone can’t reach a pod. And every data tool is read-only by design: run_readonly_sql runs on a read-only SQL warehouse, accepts only SELECT statements and caps the number of rows it returns.

Making it cheap and visible: hashing, caching and tracing

Three problems showed up as soon as real content went in.

Duplicates everywhere. The same Confluence page gets copied into three spaces. The same SQL sits in five repos. An Excel export of a table gets attached to a Jira ticket. Duplicates bloat the index and, worse, fill the writer’s context with the same fact five times.

The fix is a content hash on every chunk. Each chunk is normalised (whitespace, case and boilerplate headers removed), hashed with SHA-256 and checked against the index before anything else happens. A known hash is linked to the existing chunk instead of being stored again, and a near-duplicate check catches chunks that differ only by small edits. That cut the index size by 28%. At query time, the same hashes remove repeated chunks before they reach the writer.

Paying for the same embedding twice. Embedding calls are a high variable cost per query. I put a cache in front of the embedding model, keyed by a hash of the normalised text plus the model name. Before the cache, the system ran the same search several times while answering one question, and every repeat paid for its embeddings again. With the cache in place, embedding calls per query dropped 5x.

import hashlib
def chunk_key(text: str, model: str) -> str:
    normalised = " ".join(text.lower().split())
    return hashlib.sha256(f"{model}:{normalised}".encode()).hexdigest()
async def embed(text: str, model: str = EMBED_MODEL) -> list[float]:
    key = chunk_key(text, model)
    if (cached := await cache.get(key)) is not None:
        return cached
    vector = await embedding_client.embed(text, model=model)
    await cache.set(key, vector)
    return vector

Logic hidden in Excel. A surprising amount of business logic lives in workbooks and their VBA macros. I extract the macros with olevba from the oletools package and the formulas and named ranges with openpyxl, then index each VBA procedure as its own chunk. Now a question like “where is this adjustment factor calculated?” can return a macro as easily as a Python job.

Seeing what actually happened. Every question gets one trace ID at the planner. It travels through each MCP call, the gateway log, and the server spans, so I can open a single trace and see which sources were called, what each one returned, how long it took, how many tokens it used, and what the reviewer decided. Without tracing, tuning a multi-agent system is mostly guessing.

What went wrong first

None of my early problems were in the MCP layer. The protocol, the transport, and the deployment were the easy part. Everything that hurt was in the RAG layer underneath.

I treated ingestion as a one-time job, not a data quality problem. I pointed the ingesters at every source and indexed whatever came back. The index filled up with copies: the same page in several spaces, the same query in several repos, table exports attached to tickets. Results looked rich but said the same thing over and over. Keeping the data clean turned out to matter more than any retrieval trick, and that is where the content hashing came from.

There was no cache. Every embedding and every search was paid for fresh, every time. I left caching out at the start because it felt like an optimisation for later. That was a mistake I paid for in tokens.

Too many tokens to find one fact. A simple question could pull long chunks from several sources, duplicates included, and the model had to read all of it to find a single piece of information. Most of those tokens went on reading things it didn’t need.

The same search, many times over. Without a cache, the system was forced to run the same search again and again while answering one question, and each repeat cost embeddings and context all over again. Now a repeated search is answered from the cache instead of being run again.

The lesson I took away: clean the data and stop repeating work before you tune anything clever. MCP made the system easy to connect. It did nothing to make the data cleaner.

A playbook for building your own MCP servers

If you are about to build MCP servers for your own domain, this is the checklist I would hand you.

1. One server per source or bounded context. Not one giant server, and not one per team.
2. Write tool descriptions for the model. Say what the tool is for and when not to use it. Then test the description by asking the model which tool it would pick for a few real questions.
3. Prefer coarse tools. Hide retrieval internals behind one search tool.
4. Read-only by default. Mark tools with readOnlyHint, and make any write tool a separate, deliberate decision with its own review.
5. Return a source with every result. If a result has no URI, the writer can’t cite it, and the reviewer can’t check it.
6. Use stdio to build and Streamable HTTP to ship. Target the 2026-07-28 revision, and keep state in explicit handles, not in the connection.
7. Never pass tokens through. Use audience-bound tokens and an on-behalf-of exchange for every downstream system.
8. Put the gateway in early. Namespacing, tenant filtering and audit are painful to bolt on later.
9. Trace every call with one ID, from the planner through the gateway to the backend.
10. Test with the MCP Inspector first. npx @modelcontextprotocol/inspector lets you call every tool by hand before an agent ever sees it.
11. Template the deployment. One Helm chart, one values file per server.

Closing thoughts

MCP didn’t make my retrieval better. Hybrid search, graph expansion, and a strict reviewer did that. What MCP changed is everything around the retrieval: sources became servers with a clear contract, the agents stopped caring where data lives, and security moved to one place where I can actually reason about it.

If you take one idea from this article, take this one: design MCP servers as products, not plumbing. Give them owners, versions, good descriptions and real security, and the agents on top become surprisingly simple.

In a future part of this series, I’ll go deeper into the reviewer agent and how it decides whether an answer is grounded. If you are building something similar, I’d like to hear how you handled the gateway and the identity pieces. Leave a comment. Also read https://stackoverflow.blog/2026/05/08/no-dumb-questions-mcp/