skill-up is a specialized evaluation framework designed for Agent Skill developers. It provides a declarative way to verify Skill behavior across multiple Agent Engines (Claude Code, Codex, Qoder CLI, and Qwen Code) in both local development and CI environments README.md43-45
Users can install skill-up via a shell script, Homebrew (if configured), or build from source using the Go toolchain.
The install.sh script automates OS and architecture detection to fetch the correct binary from GitHub Releases.
The script performs the following logic:
uname -s and uname -m to determine the platform install.sh16-37SKILL_UP_VERSION is not set install.sh39-42 install.sh68-74checksums.txt file, then verifies integrity using sha256sum or shasum install.sh44-59 install.sh97-99$HOME/.local/bin (default) or a custom INSTALL_DIR install.sh7 install.sh107-109For development or custom environments, skill-up can be built using Go 1.25 or later README.md136-137
The entry point for the application is located at cmd/skill-up/main.go AGENTS.md17
Run the following command to ensure the binary is in your PATH and functioning:
Sources: docs/guide/getting-started.md88-91 install.sh1-128 AGENTS.md44-55
skill-up relies on a specific directory structure within your Skill project to discover and execute evaluations.
| Component | File/Directory | Description |
|---|---|---|
| Skill Root | SKILL.md | The marker file that defines the Skill. skill-up uses this to locate the Skill for automatic installation docs/guide/getting-started.md107 docs/guide/getting-started.md140 |
| Eval Config | evals/eval.yaml | The main entry point. Defines the environment, engine, and references cases docs/guide/getting-started.md124-138 |
| Case Files | evals/cases/*.yaml | Individual test cases defining the input.prompt and expect assertions. Filename determines Case ID docs/guide/getting-started.md148-159 |
| Fixtures | evals/fixtures/ | Optional directory for repository templates, patch files, or grading scripts docs/guide/getting-started.md113-116 |
The following diagram illustrates how the skill-up CLI maps filesystem entities to internal execution logic.
Natural Language to Code Entity Mapping: Configuration
Sources: docs/guide/getting-started.md103-116 docs/guide/getting-started.md124-138 README.md45 CONTRIBUTING.md49-55
The most efficient way to get started is using the skill-upper meta-skill, which assists in scaffolding and running evaluations through natural language README.md65-69
skill-upper is an Agent Skill that runs inside your AI agent (like Claude Code or Codex). It checks for the skill-up CLI and guides installation if missing README.md83-85
Open your Skill project and ask the AI agent to create an evaluation.
"Use skill-upper to add evals for this Skill. Add this evaluation case: Input: write a hello world program. Evaluation: check that the output contains hello and world." README.md97-104
skill-upper will:
SKILL.md README.md89-95evals/eval.yaml and evals/cases/*.yaml files README.md108-116skill-up validate command README.md190-196Run the evaluation via the CLI or by asking the agent:
Evaluation Lifecycle Diagram
Sources: docs/guide/getting-started.md180-192 README.md117-121 README.md52-54 CONTRIBUTING.md104-121
After a run, skill-up creates a workspace directory (e.g., my-skill-workspace/) containing:
iteration-1/: The results for the first run README.md204result.json: Detailed execution trace and judge results README.md52benchmark.md: A visual summary of the evaluation README.md52report.html: An interactive HTML report README.md52Sources:
Refresh this wiki
This wiki was recently refreshed. Please wait 1 day to refresh again.