Configuration Schema and Loading

Relevant source files

• internal/config/README.md
• internal/config/defaults.go
• internal/config/defaults.yaml
• internal/config/loader.go
• internal/config/loader_test.go
• internal/config/mcp_merge.go
• internal/config/mcp_merge_test.go
• internal/config/schema.go
• internal/config/schema_test.go
• internal/config/template_token_test.go
• internal/config/validator.go
• internal/config/validator_test.go

This page provides a deep dive into the configuration system of skill-up. It covers the YAML schemas for evaluations and cases, the logic for discovering skill roots via SKILL.md, and the internal mechanisms for loading, defaulting, and validating configurations within the internal/config package.

Overview

The configuration system is designed around two primary file types:

1. eval.yaml: The entrypoint configuration defining the environment, agent engine, and global defaults internal/config/schema.go10-21
2. case.yaml: Individual test case definitions typically referenced in eval.yaml internal/config/schema.go214-223

The internal/config package handles the lifecycle of these files, from disk discovery to structured Go entities.

Configuration Data Flow

The following diagram illustrates how configuration flows from raw files into the system's core entities.

Diagram: Configuration Entity Mapping

Sources: internal/config/loader.go33-60 internal/config/schema.go10-21 internal/config/validator.go45-50 internal/config/defaults.go19-23

---

Skill Root Discovery

The Loader internal/config/loader.go33-37 identifies the "Skill Root" by searching for a SKILL.md file. This discovery is crucial for resolving relative paths in the configuration and naming the skill in reports.

The Discovery Algorithm

The FindSkillDir function internal/config/loader.go214-229 implements the following logic:

1. Start at the directory containing the eval.yaml.
2. Walk up the directory tree (up to 10 levels, defined by maxSkillDirSearchDepth) internal/config/loader.go17
3. The first directory containing SKILL.md is designated the skillDir.
4. If not found, it falls back to the parent directory of the eval.yaml and logs a warning internal/config/loader.go50-53

Path Normalization

The NewLoader function internal/config/loader.go43-60 normalizes the evalPath using filepath.Abs. This ensures that internal logic for installing skills or mounting workspaces does not break when the CLI is invoked from different working directories internal/config/loader.go40-42

Diagram: Skill Directory Resolution

Sources: internal/config/loader.go43-60 internal/config/loader.go214-229

---

Configuration Workflows

The Loader provides methods to load the full evaluation suite or individual components.

Method	Role
LoadEvalConfig	Reads eval.yaml and applies system defaults internal/config/loader.go67-83
LoadCaseConfig	Reads a specific case file, resolving its path from the skillDir internal/config/loader.go108-133
LoadAll	Orchestrates loading the EvalConfig and all referenced CaseConfig files into an EvalResult internal/config/loader.go151-166

Sources: internal/config/loader.go67-166

---

Configuration Schemas

EvalConfig (eval.yaml)

The EvalConfig struct internal/config/schema.go11-21 is the root document. Key sub-configs include:

Struct	Field	Purpose
Environment	Type	Defines runtime: none, opensandbox, or docker internal/config/schema.go25
EngineConfig	Name	Specifies agent: claude_code, codex, or custom internal/config/schema.go97
CasesConfig	Files	List of relative paths to case YAML files internal/config/schema.go118
JudgeConfig	Type	Global grading strategy: rule_based, script, or agent_judge internal/config/schema.go145
BenchmarkConfig	Enabled	If true, runs cases both with and without the skill internal/config/schema.go197

CaseConfig (case.yaml)

Individual cases can override global defaults. The CaseConfig internal/config/schema.go214-223 includes:

• Input: The prompt or conversation turns internal/config/schema.go231-235 Supports multi-turn conversation with regex-based variable Capture and per-turn PostCondition gates internal/config/schema.go236-241
• Expected: Simple assertions like OutputContains or ExitCode internal/config/schema.go248-251
• Judge: Per-case judge override internal/config/schema.go221
• MCP: Case-level MCP overrides internal/config/schema.go222

MCP Configuration and Merging

MCP servers can be defined at the eval level and overridden or extended at the case level internal/config/mcp_merge.go29-74

• Merge Strategy: Case-level servers with matching names replace eval-level entries internal/config/mcp_merge.go65-67 New names are appended internal/config/mcp_merge.go69-70
• MVP Restrictions: At the case level, every server must use mode: mocked internal/config/mcp_merge.go57-59

Sources: internal/config/schema.go10-251 internal/config/mcp_merge.go1-97 internal/config/validator.go15-28

---

Loading and Defaulting

The loading process involves reading YAML, applying implicit defaults, and resolving IDs.

Default Application

LoadEvalConfig internal/config/loader.go67-83 calls applyDocumentedDefaults internal/config/loader.go91-105 which merges user config with hardcoded system defaults. The package provides a DefaultEvalConfig() factory internal/config/defaults.go21-23 (backed by an embedded defaults.yaml internal/config/defaults.yaml1-27) to ensure consistency.

System Defaults:

• timeout_seconds: 300s internal/config/defaults.yaml20
• max_turns: 10 internal/config/defaults.yaml21
• parallelism: 1 internal/config/defaults.yaml22
• report.formats: [json] internal/config/defaults.yaml25

Case ID Resolution

If a case.yaml does not explicitly define an id, the Loader uses the file's basename (without extension) as the ID via the stripExt helper internal/config/loader.go124-208

Sources: internal/config/loader.go67-208 internal/config/defaults.yaml1-27 internal/config/defaults.go21-23

---

Template Variable Resolution

Configuration files support ${VAR} references, particularly in custom engine definitions.

Token Parsing

The TemplateToken struct internal/config/template_token.go15-26 handles parsing logic for the following forms:

• Plain: ${FOO} internal/config/template_token_test.go19
• Default: ${FOO:-default} internal/config/template_token_test.go20
• Error: ${FOO?must set FOO} internal/config/template_token_test.go22

The ParseTemplateToken function internal/config/template_token.go32-40 implements the splitting logic, prioritizing the :- default operator over the ? error operator internal/config/template_token_test.go15-17

Sources: internal/config/template_token.go1-63 internal/config/template_token_test.go1-63

---

Validation Logic

The Validator struct internal/config/validator.go45-50 performs structural and semantic checks.

Key Validation Rules

1. Schema Version: Must be v1alpha1 internal/config/validator.go57-61
2. Runtime Integrity: If type is docker, it validates specific Docker fields via validateDockerEnvironment internal/config/validator.go73-75
3. Engine Name: Must be a recognized built-in via IsBuiltinEngineName internal/config/validator.go40-82
4. Case Integrity:
   • Either input.prompt or input.turns is required, but not both internal/config/validator.go116-121
   • Each turn in input.turns must have role: user internal/config/validator.go127-129
5. Judge Consistency:
   • script judge requires script_path internal/config/validator.go301-303
   • agent_judge requires model and criteria internal/config/validator.go306-312
6. Artifact Globs: Validates collect_artifacts patterns using doublestar.ValidatePattern internal/config/validator.go328-340
7. MCP Validation: Case-level MCP servers must use mode: mocked and, if not the builtin filesystem server, must provide a config_ref internal/config/validator.go183-188

Custom Engine Validation

Validation for engine.custom is intentionally deferred until CLI overrides (like --engine) settle the final engine name internal/config/validator.go84-85 This ensures that the validator does not reject configurations that might be superseded by runtime flags.

Sources: internal/config/validator.go1-340 internal/config/schema.go10-251 internal/config/mcp_merge.go1-97