The Model Context Protocol (MCP) integration in skill-up allows agents to interact with external tools and data sources during evaluation. The system supports both real MCP servers (connected via stdio or HTTP) and mocked servers for deterministic testing.
MCP servers are configured within the mcp block of an eval.yaml file or overridden in a case.yaml. Each server must specify a mode (real or mocked) and can either define its configuration inline or reference an external file via config_ref. internal/mcp/provisioner.go48-55
| Field | Type | Description |
|---|---|---|
name | string | Unique identifier for the MCP server. |
mode | string | Either real or mocked. internal/mcp/provisioner.go103-112 |
config_ref | string | Path to a YAML file containing server details. internal/mcp/provisioner.go115-127 |
transport | string | stdio or http. internal/mcp/provisioner.go21-22 |
command | string | Executable to run (for stdio). internal/mcp/provisioner.go149-152 |
args | []string | Arguments for the command. internal/mcp/provisioner.go153-157 |
endpoint | string | URL for the server (for http). internal/mcp/provisioner.go134-137 |
Sources: internal/mcp/provisioner.go18-23 internal/mcp/provisioner.go102-113 internal/mcp/provisioner.go210-219 internal/config/mcp_merge.go9-12
The Provisioner struct in internal/mcp/provisioner.go is responsible for resolving environment variables, loading external configurations, and preparing a runtime.MCPServerConfig that the engine can use to initialize the agent. internal/mcp/provisioner.go33-36
The Provisioner resolves variables in the format ${VAR_NAME} within endpoints and headers. It requires all authentication to be available non-interactively from the process environment. internal/mcp/provisioner.go30-32
required_env in their config_ref file. If these are missing from the host, provisioning fails. internal/mcp/provisioner.go165-168resolveReferencedEnv function extracts variable names using envRefPattern and fetches values via a LookupEnv function. internal/mcp/provisioner.go27 internal/mcp/provisioner.go141-146HeaderEnv to track which environment variables must be propagated to the agent. internal/mcp/provisioner.go169-172Cases can override eval-level MCP configurations using MergeCaseMCP. This allows different cases in the same evaluation suite to receive different mock responses (e.g., testing "project open" vs "project closed" states) while keeping the same server and tool names. proposals/0003-per-case-mocked-mcp-responses.md42-46 internal/config/mcp_merge.go29
mode: mocked. internal/config/mcp_merge.go57-59 internal/config/mcp_merge_test.go119-123Sources: internal/mcp/provisioner.go33-58 internal/mcp/provisioner.go129-146 internal/config/mcp_merge.go29-74 internal/config/mcp_merge_test.go8-66 proposals/0003-per-case-mocked-mcp-responses.md76-83
When mode: mocked is selected, skill-up generates a local MCP server using a built-in Node.js script. This allows testing agent tool-calling logic without requiring network access or external binaries. internal/mcp/mock_server.go17-45
Mock servers use a tool_responses map defined in the config_ref file. e2e/testdata/mcp-mocked-marker/evals/fixtures/mcp/marker.yaml1-4
{{ params.name }} syntax. The render function in the mock script handles path lookup and default values. internal/mcp/mock_server.go152-160config_ref in case.yaml, different cases can return different status codes or payloads for the same tool. e2e/testdata/mcp-case-overrides/evals/cases/project-open.yaml10-14 e2e/testdata/mcp-case-overrides/evals/cases/project-closed.yaml10-14A specialized built-in mock named filesystem provides a read-only MCP interface to the local workspace. internal/config/mcp_merge.go11 It implements tools like read_file and includes security checks to prevent path traversal by rejecting symlink escapes. internal/mcp/mock_server.go173-180
The buildMockedServerScript function injects a JSON payload of fixtures into a generic Node.js template (genericMockServerScript). This script implements the MCP JSON-RPC protocol over stdio, including Content-Length framing. internal/mcp/mock_server.go47-59 internal/mcp/mock_server.go120-171
Sources: internal/mcp/mock_server.go17-45 internal/mcp/mock_server.go47-59 internal/mcp/mock_server.go120-171 internal/config/mcp_merge.go57-62 e2e/testdata/mcp-case-overrides/evals/eval.yaml6-10
Once provisioned, the runtime.MCPConfig is passed to the Agent Engine. The engine is responsible for installing and executing these servers within the target environment via InstallMCP. internal/agent/cli.go25-79
Each agent engine implements specific logic to register MCP servers by executing commands in the Runtime. The CLIAgent uses an InstallMCPCmd template to generate these commands. internal/agent/cli.go33-68
InstallMCPCmd template which often resolves to claude mcp add. e2e/mcp_test.go23 internal/agent/cli.go33-36codex mcp bridge) to manage the MCP lifecycle within the sandbox.command as a child process and communicates via stdin/stdout. The mcpProtocolScript provides the framing layer for these interactions. internal/mcp/mock_server.go64-118endpoint and passes headers resolved by the Provisioner. internal/mcp/provisioner.go73-76Sources: internal/agent/cli.go25-79 internal/mcp/provisioner.go73-82 internal/mcp/mock_server.go17-45 e2e/mcp_test.go14-44 internal/agent/cli.go69-72
Refresh this wiki