skill-up is a declarative CLI evaluation framework designed specifically for Agent Skill developers. It provides a standardized workflow to verify that AI agents (such as Claude Code, Codex, or Qwen Code) correctly utilize custom skills within isolated, reproducible environments.
The framework automates the process of workspace preparation, skill installation, agent execution, and automated grading, replacing ad-hoc testing scripts with a structured pipeline that generates detailed reports in multiple formats (JSON, HTML, JUnit, and Markdown).
The system is built around five core abstractions that bridge the gap between human-readable configuration and executable code:
| Concept | Description | Code Entity |
|---|---|---|
| Skill | The tool or capability being evaluated, defined by a SKILL.md file. | SkillRef internal/config/schema.go87-92 |
| Engine | The AI Agent client (e.g., claude_code, qwen_code) that executes the task. | EngineConfig internal/config/schema.go95-105 |
| Runtime | The isolation layer where the evaluation runs (Docker, OpenSandbox, or Host). | runtime.Runtime interface internal/runtime/runtime.go42-59 |
| Case | A specific test scenario containing a prompt and expected outcomes. | CaseConfig internal/config/schema.go173-205 |
| Judge | The logic used to grade the agent's performance (Rule-based, Script, or LLM). | JudgeConfig internal/config/schema.go215-220 |
Sources: README.md43-57 internal/config/schema.go9-20 internal/config/schema.go173-205 internal/config/schema.go215-220
The following diagram illustrates how the major subsystems interact to transform a declarative eval.yaml into a final evaluation report.
Sources: internal/config/schema.go10-20 internal/runtime/runtime.go42-59 internal/evaluator/artifacts_collect.go17-26 internal/evaluator/artifacts_collect.go62-66
Skill-up maps high-level user intentions defined in YAML to specific Go structures and interfaces.
Sources: internal/config/schema.go10-20 internal/config/schema.go41-64 internal/runtime/runtime.go42-59 internal/evaluator/artifacts_collect.go116-120
eval.yaml. Runtimes like docker or opensandbox provide isolation, while none runs on the host internal/config/schema.go23-38claude_code, codex, qodercli, and the qwen_code engine (aliased as qwen) CHANGELOG.md112-121input.turns) with per-turn post_condition gates and regex-based variable capture CHANGELOG.md61-68CustomEngine using local or http transport CHANGELOG.md121-135 The http transport supports multipart file uploads and SessionInput JSON injection docs/guide/writing-evals.md161-175rule_based: Matches patterns like must_contain, tool_called, or exit_code internal/config/schema.go222-230script: Executes a script where exit code 0 indicates success README.md51agent_judge: Uses an LLM with configurable context profiles (minimal, standard) to evaluate complex criteria CHANGELOG.md39-44doublestar glob patterns after a run, preserving relative paths even if the agent fails or times out docs/guide/writing-evals.md134-144skill-upper meta-skill allows developers to use an AI agent to scaffold, validate, and interpret evaluation suites README.md65-70real and mocked modes, with per-case fixture overrides CHANGELOG.md10-19judge.skills in agent_judge, allowing domain-specific rubrics to be installed only for the judge agent CHANGELOG.md23-27To dive deeper into specific areas of the framework, refer to the following pages:
install.sh or building from source, verifying the install, and running a first evaluation with skill-upper assistance. docs/guide/getting-started.md1-29.skill-up.yaml schema, environment variable overrides, and engine-specific kwargs. docs/guide/writing-evals.md118-128scripts/windows/. README.md144-147Sources: CHANGELOG.md8-140 README.md43-72 docs/guide/writing-evals.md1-160 internal/config/schema.go10-105
Refresh this wiki
This wiki was recently refreshed. Please wait 1 day to refresh again.