Runner and Evaluator Pipeline

Relevant source files

• e2e/contract_test.go
• internal/agent/agent.go
• internal/evaluator/artifacts_collect_test.go
• internal/evaluator/evaluator.go
• internal/evaluator/evaluator_test.go
• internal/evaluator/fixtures.go
• internal/evaluator/fixtures_test.go
• internal/evaluator/multiturn.go
• internal/evaluator/multiturn_test.go
• internal/judge/judge_skill.go
• internal/judge/judge_skill_test.go
• internal/mcp/provisioner_test.go
• internal/report/html.go
• internal/report/junit.go
• internal/report/reporter.go
• internal/report/reporter_test.go
• internal/runner/runner.go
• internal/runner/workspace_options_test.go

The Runner and Evaluator subsystems form the core execution engine of skill-up. While the Runner handles high-level orchestration (loading configurations, managing multiple iterations, and generating final reports), the Evaluator manages the granular lifecycle of individual test cases, ensuring that environments are prepared, agents are executed, and results are rigorously judged.

1. Runner Orchestration Layer

The Runner struct is the entry point for executing an evaluation suite. It coordinates the transition from static configuration files to a live execution pipeline.

1.1 Key Responsibilities

• Workspace Management: Initializes the output directory structure, creating iteration-N folders and cleaning stale artifacts internal/runner/runner.go71-87
• Iteration Control: Manages multiple runs of the same evaluation suite. It can auto-detect the next available iteration number in the workspace directory if not explicitly provided internal/runner/runner.go127-133
• Concurrency Control: Reads the parallelism setting from eval.yaml (falling back to documented defaults via the Loader) to determine how many cases the Evaluator should process simultaneously internal/runner/runner.go119
• Result Aggregation: Collects results from the Evaluator and invokes the reporting engine to produce result.json, HTML reports, or JUnit XML internal/runner/runner.go175-179 internal/runner/runner.go191-204

1.2 Runner Execution Flow

The following diagram illustrates how the Runner bridges the CLI/Config space into the execution pipeline.

Runner to Evaluator Bridge

Sources: internal/runner/runner.go95-180 internal/runner/runner.go71-87 internal/runner/runner.go119-133

---

2. Evaluator Lifecycle

The Evaluator (specifically defaultEvaluator) implements the logic for executing a set of CaseConfig objects. It manages a worker pool based on the configured concurrency using a semaphore channel internal/evaluator/evaluator.go209-234

2.1 Per-Case Pipeline

Each case follows a strict sequence of stages within the executeCase function internal/evaluator/evaluator.go332-511:

1. Runtime Preparation: A fresh runtime.Runtime is created (e.g., Docker, OpenSandbox, or Host) via the runtime.NewRuntime factory internal/evaluator/evaluator.go343-348
2. Context Upload (Fixtures): The fixtureRegistry coordinates uploading files, initializing Git repositories, and applying diffs to the workspace. This involves components like repoFixtureUploader and gitInitUploader internal/evaluator/fixtures.go23-40 internal/evaluator/fixtures.go114-147
3. Agent Execution: The engine is invoked. The Evaluator handles "with_skill" (skill installed) and "without_skill" (baseline) logic internal/evaluator/evaluator.go420-435 For multi-turn cases, it uses executeMultiTurn to manage session persistence internal/evaluator/multiturn.go72-153
4. Expect Pre-checks: Before formal judging, the Evaluator performs must_contain or must_not_contain checks on the agent's final message via runExpectPreCheck internal/evaluator/evaluator.go472-474
5. Judge Phase: The output is passed to the configured judge (Rule-based, Script, or Agent-LLM) using judge.NewJudge internal/evaluator/evaluator.go477-482
6. Artifact Collection: Files matching collect_artifacts globs are downloaded from the runtime to the local iteration workspace via collectGlobArtifacts internal/evaluator/evaluator.go488-490

2.2 Execution Data Flow

This diagram maps the internal functions and structs involved in the per-case lifecycle.

Evaluator Internal Pipeline

Sources: internal/evaluator/evaluator.go332-511 internal/evaluator/fixtures.go19-40 internal/evaluator/multiturn.go72-153

---

3. Implementation Details

3.1 Multi-Turn Logic

If a case defines input.turns, the Evaluator switches to a stateful execution mode via executeMultiTurn internal/evaluator/multiturn.go72-88

• Variable Capture: Rules in turn.capture use regex to extract values from the assistant's response, which are then injected into subsequent turns via {{variable}} templates internal/evaluator/multiturn.go136-148 internal/evaluator/multiturn.go218-245
• Post-Conditions: Each turn can be validated immediately. If a post_condition fails, the evaluator can skip remaining turns or fail the case early via handlePostCondition internal/evaluator/multiturn.go129-134
• Session Resumption: Agents must implement the SessionResumer interface to support this flow, specifically the RunTurn method which accepts a sessionID internal/evaluator/multiturn.go79-82 internal/agent/agent.go72-74

3.2 Workspace Setup and Fixtures

The Evaluator uses a fixtureRegistry to populate the runtime workspace. Key uploader types include:

• repoFixtureUploader: Copies a directory from the skill's fixtures/ folder to the runtime internal/evaluator/fixtures.go23-40
• gitInitUploader: Runs git init, configures user.name and user.email, and adds remotes internal/evaluator/fixtures.go114-147
• gitCheckoutUploader: Switches to a specific branch, creating it if the repository is fresh. It uses git switch and includes safety checks to prevent masking typos in branch names internal/evaluator/fixtures.go149-195
• contextFilesUploader: Injects ad-hoc files defined directly in case.yaml using a temporary file buffer internal/evaluator/fixtures.go42-76

3.3 Status Tracking

The Evaluator tracks the Status of each case to aggregate final metrics internal/evaluator/evaluator.go67-106:

Status	Meaning
PASS	Judge returned pass and all expect pre-checks passed.
FAIL	Logic check failed (e.g., script exited non-zero or LLM judge failed).
ERROR	System failure (e.g., Docker container failed to start, credential check failed).
SKIP	Case was skipped via CLI filters or internal conditions (e.g. post_condition failure).

Sources: internal/evaluator/evaluator.go67-106 internal/evaluator/multiturn.go25-34 internal/evaluator/fixtures.go19-40

---

4. Key Functions and Classes

Entity	File Path	Role
Runner	internal/runner/runner.go48-58	High-level orchestrator; manages iterations, workspace setup, and result reporting.
Evaluator	internal/evaluator/evaluator.go124-126	Interface for executing all cases through the evaluation pipeline.
defaultEvaluator	internal/evaluator/evaluator.go129-148	Main implementation of the evaluation pipeline; manages concurrency and fixture registry.
EvalResult	internal/evaluator/evaluator.go67-106	Container for agent output, grading results, status, and metadata.
executeCase	internal/evaluator/evaluator.go332-511	The core method defining the per-case lifecycle (Prep -> Exec -> Judge -> Collect).
executeMultiTurn	internal/evaluator/multiturn.go72-153	Orchestrates multi-turn logic, variable capture, and post-condition checks.
TurnResult	internal/evaluator/multiturn.go37-54	Holds the outcome of a single conversation turn for multi-turn evaluations.

Sources: internal/runner/runner.go1-58 internal/evaluator/evaluator.go1-148 internal/evaluator/multiturn.go1-153