Agent Engines are the core execution units in skill-up. They bridge the gap between the declarative evaluation configuration and the actual execution of an AI agent within a target Runtime Environments. The framework abstracts agent interactions through the Agent interface, allowing for consistent installation, MCP server management, and session execution across different agent implementations.
The Agent interface internal/agent/agent.go133-148 defines the contract for all supported engines. This abstraction ensures that the evaluation pipeline can interact with any engine—whether built-in or custom—using a unified set of methods:
Install: Prepares the agent in the environment (e.g., via npm or nvm) internal/agent/agent.go137InstallMCP: Provisions Model Context Protocol servers required by the agent internal/agent/agent.go139InstallSkill: Injects the local Skill being evaluated into the agent's search path internal/agent/agent.go141Run: Executes the agent with a provided set of messages and returns a structured SessionResult internal/agent/agent.go143Check: Verifies the availability of the agent binary in the current PATH internal/agent/agent.go145The following diagram illustrates how the Evaluator uses the Agent interface to bridge the Natural Language Space (prompts/messages) to the Code Entity Space (CLI execution and results).
Agent Execution Lifecycle
Sources: internal/agent/agent.go133-148 internal/agent/factory.go39-77 internal/agent/factory.go12-34 internal/evaluator/evaluator.go180-237
skill-up provides built-in support for several industry-standard agent CLIs and a flexible extension point for others. The DetectAgent function internal/agent/factory.go12-34 routes the engine name to the appropriate implementation based on constants defined in the agentkind package internal/agentkind/agentkind.go12-22
| Engine Name | Implementation Class | Key Characteristics |
|---|---|---|
claude_code | ClaudeCodeAgent | Anthropic's Claude Code CLI. Supports advanced token usage tracking. |
codex | CodexAgent | OpenAI-compatible agent. Supports Landlock sandbox bypassing via bypass_sandbox. |
qodercli | QoderCLIAgent | Native support for Qoder CLI with specialized "auto" model tier handling. |
qwen_code | QwenCodeAgent | Support for Qwen-based coding agents internal/agent/factory.go24-25 |
custom | CustomAgent | Local transport engine that communicates via JSON files or stdout. |
claude-code CLI. It handles specific credential mapping for ANTHROPIC_API_KEY and integrates with OpenTelemetry for tracing. For details, see Claude Code Agent..jsonl files and provides an MCP bridge for tool integration. For details, see Codex Agent.qodercli binary, specifically managing the QODER_PERSONAL_ACCESS_TOKEN internal/agent/factory.go67-70 and directory isolation. It also treats the auto model name as a specific tier internal/agent/factory.go45-47 For details, see QoderCLI Agent.The Custom Engine allows users to integrate any local binary as an agent. It is activated when the engine name is not a built-in and a custom configuration is provided internal/agent/factory.go29-31 It supports template variable injection (e.g., ${workspace}, ${input_file}) and can receive inputs/return results via a structured JSON contract.
For details, see Custom Engine.
The internal/agent package provides several shared utilities and base structures used by all engines:
BaseAgent: A common struct internal/agent/agent.go159-162 that holds the Config and version information for specific implementations.Agent Factory: The DetectAgent and DetectAgentWithInitParams functions internal/agent/factory.go12-77 handle the instantiation of the correct agent based on eval.yaml configuration.Kwargs Handling: A standardized way to pass engine-specific options through the Kwargs map internal/agent/kwargs.go12-24 The KwargBypassSandbox (bypass_sandbox) is a project-wide recognized key used to skip process sandboxing internal/agent/kwargs.go16SessionResult internal/agent/agent.go49-63 containing metrics like InputTokens, OutputTokens, Turns, and the FinalMessage.SessionResumer interface internal/agent/agent.go72-74 allows agents to maintain state across multiple turns in a single evaluation case.Agent Data Mapping
Sources: internal/agent/agent.go49-63 internal/agent/factory.go39-58 internal/agent/kwargs.go12-24 internal/agent/agent.go72-74
For details on these shared components, see Agent Shared Infrastructure.
Engines are configured in eval.yaml under the engine block. Key-value pairs in kwargs allow for fine-grained control over agent behavior. Typos in kwargs are flagged as debug logs during agent initialization internal/agent/kwargs.go46-63
Sources: internal/agent/factory.go39-58 internal/agent/kwargs.go12-24 internal/agent/kwargs.go46-63