Core Architecture

Relevant source files

• README.md
• README.zh.md
• docs/guide/getting-started.md
• docs/zh/guide/getting-started.md
• internal/agent/agent.go
• internal/evaluator/evaluator.go
• internal/evaluator/evaluator_test.go
• internal/judge/judge_skill.go
• internal/judge/judge_skill_test.go
• internal/mcp/provisioner_test.go
• internal/report/html.go
• internal/report/junit.go
• internal/report/reporter.go
• internal/report/reporter_test.go
• internal/runner/runner.go

The skill-up architecture is designed as a modular pipeline that transforms high-level evaluation definitions into structured performance reports. It orchestrates the lifecycle of an AI agent's execution within isolated environments, followed by automated grading and artifact collection.

System Overview

The following diagram illustrates the high-level flow from a user's CLI command to the final generated reports, showing how the internal subsystems interact.

High-Level Data Flow

Sources: internal/evaluator/evaluator.go1-5 internal/runner/runner.go1-13 README.md43-54

---

Core Subsystems

CLI Layer

The CLI layer serves as the entry point, handling flag parsing, environment variable resolution, and command routing. The run command is the primary driver, managing the transition from user configuration to the execution phases.

• Key Function: Coordinates the transition from loading eval.yaml to invoking the Runner.
• Responsibility: Merges CLI flags (like --engine, --model, or --api-key) with loaded configuration.
• For details, see CLI Layer.

Configuration and Discovery

skill-up uses a convention-based discovery system. It searches for configuration files and identifies the "Skill Root" by looking for a SKILL.md file to resolve relative paths for fixtures and scripts README.md122-125

• Key Entity: config.EvalConfig represents the root document loaded from eval.yaml internal/runner/runner.go49-50
• Responsibility: Defines the Environment (none, opensandbox, docker), Engine (claude_code, codex, qodercli, custom), and Judge strategies README.md49-51
• Discovery: The Loader resolves the skill name from SKILL.md or directory basenames internal/runner/runner.go103-118
• For details, see Configuration Schema and Loading.

Runner and Evaluator Pipeline

The execution is split into two orchestration levels:

1. Runner: Manages the high-level "Run", handling iteration management (multiple runs via --iteration) and IterationWorkspace setup internal/runner/runner.go1-13 internal/runner/runner.go95-180
2. Evaluator: Manages the lifecycle of a single test case. This includes preparing the Runtime, invoking the Agent, and performing grading via Judge internal/evaluator/evaluator.go1-5 internal/evaluator/evaluator.go179-237

Execution Lifecycle Diagram

This diagram bridges the conceptual "Evaluation Lifecycle" to the specific Go types and methods responsible for each stage.

Sources: internal/evaluator/evaluator.go179-237 internal/runner/runner.go153-172 internal/agent/agent.go133-148 internal/runner/runner.go1-13

• For details, see Runner and Evaluator Pipeline.

Reporting and Artifacts

Every evaluation run generates a structured directory within a workspace. This contains per-iteration folders, agent transcripts, and final reports in multiple formats (JSON, JUnit, HTML).

• Key Entity: evaluator.EvalResult holds the outcome of a single case, including Status (PASS, FAIL, SKIP, ERROR) and Grading details internal/evaluator/evaluator.go67-106
• Workspace Layout: Results are written to <skill-name>-workspace/iteration-N/ internal/runner/runner.go121-133
• Reporters: Specialized types like JUnitReporter and HTMLReporter transform report.Input into final artifacts internal/report/junit.go18-22 internal/report/html.go19-23
• For details, see Reporting and Output Artifacts.

---

Data Flow: Config to Report

The system follows a strict data transformation path:

1. Input: eval.yaml + case.yaml + CLI overrides are processed by the Loader.
2. Loading: config.Loader produces configuration objects and identifies the project root via SKILL.md internal/runner/runner.go103-118
3. Credential Resolution: The Resolver determines the final AgentInitParams (API keys, base URLs) internal/runner/runner.go61-67
4. Execution: evaluator.Evaluator executes the pipeline, producing evaluator.EvalResult which embeds the agent.SessionResult containing tokens, turns, and messages internal/evaluator/evaluator.go67-71
5. Finalization: runner.Runner writes results to result.json and triggers formatted report generation via the report package internal/runner/runner.go191-204

Sources: internal/runner/runner.go1-13 internal/evaluator/evaluator.go67-106 internal/report/reporter.go17-27 README.md43-54