This page provides a practical guide to authoring evaluations within the skill-up framework. It covers the structure of the global configuration (eval.yaml), the definition of individual test cases (case.yaml), and the lifecycle of workspace preparation.
Evaluation files are organized within the evals/ directory of a Skill. The eval.yaml file serves as the entrypoint, while individual cases reside in the cases/ subdirectory.
Sources: docs/guide/writing-evals.md7-31 docs/zh/guide/writing-evals.md7-31
The eval.yaml file defines the execution environment, the agent engine to be tested, global case defaults, and reporting preferences. It is unmarshaled into the EvalConfig struct internal/config/schema.go11-21
| Field | Type | Description |
|---|---|---|
schema_version | string | Must be v1alpha1. internal/config/validator.go57-61 |
environment | object | Defines the Environment (none, docker, opensandbox) internal/config/schema.go24-39 |
engine | object | Specifies the EngineConfig (e.g., claude_code, codex, qodercli) internal/config/schema.go96-106 |
cases | object | Lists case files and defines CaseDefaults (timeout, max_turns) internal/config/schema.go117-123 |
benchmark | object | Configures BenchmarkConfig for comparison (with vs. without skill). internal/config/schema.go19 |
mcp | object | Configures MCPConfig for tool integration internal/config/schema.go73-75 |
engine.kwargs is a free-form string map used for agent-specific behavior internal/config/schema.go104
codex to force --dangerously-bypass-approvals-and-sandbox docs/guide/writing-evals.md124DEBUG log docs/guide/writing-evals.md120--engine-kwarg) override eval.yaml docs/guide/writing-evals.md120The collect_artifacts field uses glob patterns to select files from the workspace after an agent run for download to the host.
<output-dir>/<case-id>/<configuration>/outputs/workspace/ maintaining their relative internal structure docs/guide/writing-evals.md136-143.git/ directory is automatically excluded docs/guide/writing-evals.md155doublestar globs internal/config/validator.go98Sources: docs/guide/writing-evals.md60-134 internal/config/schema.go11-180 internal/config/validator.go53-106
Each case.yaml defines a specific scenario and is unmarshaled into a CaseConfig internal/config/schema.go182-195
prompt or a sequence of turns for multi-turn evaluations internal/config/schema.go197-203post_condition gates (e.g., must_contain_any) and capture for regex-based variable extraction between turns CHANGELOG.md62-65 Extraction supports pattern (regex) or jsonpath skills/skill-upper/references/case-yaml.md60Context struct which handles the environment state before the agent starts internal/config/schema.go205-214The Evaluator prepares the Runtime environment using the following steps:
evals/fixtures/repos/ skills/skill-upper/assets/case.yaml.tmpl26git init skills/skill-upper/assets/case.yaml.tmpl28Environment internal/config/schema.go68-70must_contain, exit_code, files_exist) that run before the heavy judge phase skills/skill-upper/assets/case.yaml.tmpl39-46rule_based, agent_judge (LLM-based grading), and script internal/config/validator.go16-21Sources: internal/config/schema.go182-250 internal/config/validator.go108-163 CHANGELOG.md62-73 skills/skill-upper/references/case-yaml.md87-105
This diagram maps the eval.yaml and case.yaml fields to the internal structs defined in the internal/config package.
Sources: internal/config/schema.go11-195 docs/guide/writing-evals.md60-114 internal/config/validator.go45-51
This diagram shows the sequence of operations performed by the Evaluator to prepare the Runtime.
Sources: internal/config/schema.go41-64 docs/guide/writing-evals.md133-156 skills/skill-upper/assets/case.yaml.tmpl25-33 internal/runtime/runtime.go1-10
When engine.name is not a built-in, a custom block must be defined internal/config/schema.go105-106
local (executing a command in the runtime) or http (POSTing to a service) internal/config/validator.go31-34session_result (structured JSON) or text (stdout is the message) docs/zh/guide/writing-evals.md176-178engine.custom.env and are strictly forbidden in command or args fields docs/zh/guide/writing-evals.md186-189${workspace}, ${input_file}, and ${output_file} docs/zh/guide/writing-evals.md183-184Sources: internal/config/schema.go105-106 docs/zh/guide/writing-evals.md141-190
When benchmark.enabled is set to true in eval.yaml internal/config/schema.go19 the system performs two distinct iterations for every case:
The results are aggregated into reports (JSON, HTML, Markdown), allowing for direct comparison of the Skill's impact docs/guide/writing-evals.md107-108 The IterationWorkspace abstraction manages the storage of these artifacts CHANGELOG.md28-29
Sources: docs/guide/writing-evals.md107-108 internal/config/schema.go19 CHANGELOG.md24-30
Refresh this wiki