The Problem MCP Solves
Before the Model Context Protocol, connecting AI assistants to data infrastructure required solving the N×M problem: every AI application needed custom integrations to every data source. Each connector required custom authentication, unique error handling, and ongoing maintenance.
MCP replaces fragmented custom integrations with one protocol that works across AI applications — an analogy sometimes used is “USB-C for AI.”
MCP is an open standard developed by Anthropic and now maintained by the Agentic AI Foundation under the Linux Foundation. As of early 2026: 97+ million SDK downloads, 10,000+ production servers, and adoption by OpenAI, Google DeepMind, Microsoft, and enterprise technology leaders.
The Three-Layer Architecture
MCP defines three participants in any conversation between an AI and your data systems.
The Host is the AI application you interact with directly: Claude Desktop, VS Code with Copilot, Cursor, or a custom agent. The host is the security boundary, the orchestrator that knows who you are and what you’re authorized to access.
The Client is a bridge. Each MCP client maintains exactly one connection to an MCP server. If your host needs to query both a Postgres database and your filesystem, the host spawns two clients, each managing its own connection.
The Server is where your data lives—or more precisely, where you expose your data through MCP. An MCP server is a program that implements the protocol and exposes capabilities: database queries, metadata browsing, pipeline monitoring, data quality checks. Building a server means writing business logic in your preferred language (Python, TypeScript, Go, Rust, etc.) and decorating functions to mark them as tools, resources, or prompts.
Host (Claude, Cursor, VS Code) ├── MCP Client → MCP Server (Postgres) ├── MCP Client → MCP Server (Filesystem) └── MCP Client → MCP Server (Your Data Warehouse)Communication: JSON-RPC and Message Format
All MCP communication happens over JSON-RPC 2.0, a lightweight, stateless protocol that works everywhere. When a client connects to a server, they begin with an initialization handshake—the client announces its capabilities, the server responds with what it can do. From there, the client can discover available tools, invoke them, and stream responses back.
For data engineers, understanding the message format helps with debugging. Every MCP message has a protocol version, an ID for request-response matching, a method name, and parameters. When you call a tool, the client sends a tools/call message with the tool name and arguments. The server responds with results or an error. It’s straightforward and easy to inspect.
The real power is in how MCP abstracts away the complexity. Instead of building REST clients, parsing OpenAPI specs, and handling authentication per-API, you define Python functions and decorators, and the SDK handles serialization, lifecycle management, and protocol negotiation for you.
Transport: How Bits Move
MCP supports two transport mechanisms, and your choice depends on deployment topology.
stdio transport is for local servers—the host spawns the server as a subprocess and communicates via standard input/output streams. No network configuration, no ports to open, no TLS to manage. Your server’s credentials stay in environment variables on that machine, isolated from the network. This is ideal for Claude Desktop and development work.
Streamable HTTP transport is for remote servers—servers running in cloud environments, shared across teams, or behind authentication. It uses standard HTTP POST for client-to-server messages and optional Server-Sent Events (SSE) for streaming responses. HTTP supports OAuth 2.1 authentication with PKCE (Proof Key for Code Exchange), which prevents authorization code interception. Tokens are scoped to specific MCP servers via RFC 8707 Resource Indicators, so a compromised server can’t use tokens meant for another.
Most teams start with stdio during development, then graduate to HTTP when they want centralized deployment or cross-team access. The same Python or TypeScript code works with both transports; you just change how the server starts.
The Three Server Primitives
Everything an MCP server exposes falls into three categories. These primitives define what capabilities the server makes available.
Tools are executable functions that the AI model can invoke. Think database queries, pipeline triggers, data validation checks. Tools can have side effects—they might modify data or trigger jobs—so they require explicit invocation. The AI decides when and how to call them based on the user’s request. Tools are the most commonly used primitive because they let the AI do work.
Resources are read-only data sources. They’re like GET endpoints—the server provides information, and the AI can fetch them for context without modifying anything. A resource might expose your warehouse’s table schemas, your dbt project’s documentation, or metadata about your pipelines. Resources give the AI the information it needs to reason about what to do next.
Prompts are reusable templates that guide AI interactions. Rather than users writing the same request every time (“Analyze this table for quality issues”), you define a “Data Quality Analysis” prompt template. The AI invokes it, the server returns a structured request prompt, and the user gets consistency. Prompts are user-initiated—the user chooses them from a menu.
For data engineering specifically: tools make the AI productive (it can query tables, run checks, monitor pipelines), resources make it informed (it understands your schema, lineage, and current state), and prompts make it useful (teams get repeatable, standardized workflows).
Security and Trust Boundaries
Security in MCP follows a critical principle: the host application acts as a security intermediary between the LLM and your backend systems. The AI model never sees credentials. The host doesn’t expose every server to every request—it controls what the user is allowed to access, and which servers get invoked.
Each MCP server runs in its own process with its own credentials, so compromising one server doesn’t compromise others. This isolation is enforced at the OS level for stdio transport and by separate authentication for HTTP transport.
For sensitive operations, explicit user consent is required. When the AI wants to invoke a tool with side effects—triggering a pipeline, modifying data—the host presents it to the user for approval before execution. The user sees exactly what’s about to happen and can reject it.
A critical security pattern called “Roots” lets servers specify filesystem boundaries. A server might be granted access to /data/warehouse/ but explicitly denied /etc/passwd. This prevents servers from escaping their intended scope.
For authentication on remote servers, MCP mandates OAuth 2.1 with PKCE. Dynamic Client Registration (RFC 7591) lets servers acquire credentials at runtime without storing secrets in configuration files. Token validation happens on both sides—the host ensures tokens came from the right issuer, the server validates tokens are scoped for its use, and neither passes tokens upstream to backend services (preventing “confused deputy” attacks where a malicious request tricks a server into using its authority on behalf of an attacker).
These layers of isolation mean you can build MCP servers that access production databases without the AI seeing credentials, without other servers accessing those credentials, and with full auditability of who authorized what access when.
Why MCP, Not REST APIs?
The obvious question: we already have REST APIs and OpenAPI specifications. Why does MCP exist?
The answer comes down to the intended consumer. Traditional REST APIs are designed for human developers who read documentation, understand schemas, and write specific integration code. The API defines a contract, the developer understands that contract, and builds accordingly. Request-response, stateless, client-initiated only.
MCP is designed for AI models that discover capabilities at runtime, need rich context about what tools can do, and benefit from bidirectional communication. An AI assistant doesn’t read your API documentation. It queries available tools, reads their descriptions and schemas, and decides how to use them based on the conversation. MCP supports server-initiated communication through sampling (letting your server request that the LLM process data) and elicitation (letting your server ask the user for more information). These bidirectional capabilities enable workflows that are awkward or impossible with REST.
MCP also handles multiple content types naturally—text, images, resources, nested structures—whereas REST APIs typically return fixed JSON schemas. For exploratory data analysis where the AI needs to understand complex structures and ask follow-up questions, MCP’s flexibility matters.
That said, MCP isn’t always the right tool. For some data warehouse access patterns, native CLI commands might be more token-efficient than MCP. For high-throughput machine-to-machine integrations, REST is often simpler. MCP shines when you’re connecting human-facing AI assistants to your internal systems, letting those assistants explore, question, and guide action.
The Protocol in Production
MCP scales without rewriting. The same Python FastMCP server that runs locally in stdio mode can be deployed to a cloud environment with HTTP transport and OAuth authentication. A team can start with Claude Desktop, then add Cursor, then integrate with a custom agent, without changing server code or rebuilding connectors.
Over 5,800 community MCP servers exist as of early 2026. The MCP ecosystem includes official servers for major databases (Snowflake, BigQuery via Google’s GenAI Toolbox, ClickHouse), data tools (dbt, Databricks), and platforms (GitHub, Slack, Jira). Community servers cover additional platforms, and a basic custom server is approximately 50 lines of Python.
For data engineers, the architecture enables giving AI assistants real-time access to data infrastructure — schema, lineage, quality metrics, pipeline status — without exposing credentials, without duplicating authentication logic, and without building separate integrations per AI tool.