skill-upper is a specialized Agent Skill bundled within the skill-up repository at skills/skill-upper/. It serves as a "meta-skill" designed to assist developers in the end-to-end lifecycle of creating, running, and interpreting evaluations for other Agent Skills. By leveraging skill-upper, developers can use natural language to scaffold complex YAML configurations, select appropriate judging strategies, and analyze evaluation reports without manually memorizing CLI flags or schema details skills/skill-upper/SKILL.md2-4
The primary objective of skill-upper is to lower the barrier to entry for Agent Skill testing. It automates the following tasks:
SKILL.md skills/skill-upper/SKILL.md104-107eval.yaml and case.yaml from templates while adapting to the user's language and requirements skills/skill-upper/SKILL.md113-118skill-up validate to ensure configurations are schema-compliant before execution skills/skill-upper/SKILL.md126-128skill-up run command with appropriate engine and environment overrides skills/skill-upper/SKILL.md130-132result.json and report.html to explain failures and benchmark deltas to the user skills/skill-upper/README.md13skill-upper is triggered when a user expresses intent to:
evals.json format to the skill-up YAML format skills/skill-upper/SKILL.md60skill-up init skills/skill-upper/SKILL.md59Sources: skills/skill-upper/SKILL.md1-62 skills/skill-upper/README.md1-22
The skill follows a strict sequence to ensure consistency and reliability during the evaluation process.
The following diagram illustrates how skill-upper bridges the gap between a user's natural language request and the underlying skill-up filesystem entities and CLI commands.
Diagram: Natural Language to Code Entity Mapping
Sources: skills/skill-upper/SKILL.md63-145 skills/skill-upper/README.md8-14 internal/cli/init.go18-32 internal/userconfig/loader.go4-7
skill-up binary is in the PATH using command -v skill-up. If missing, it provides installation instructions for macOS/Linux skills/skill-upper/SKILL.md65-88skill-up init to create ~/.config/skill-up/config.yaml or .skill-up.yaml for project-specific telemetry and runtime defaults skills/skill-upper/SKILL.md91-102 internal/cli/init.go57-64SKILL.md upward from the current working directory to establish the skill-root skills/skill-upper/SKILL.md104-107evals/ directory exists, it copies assets/eval.yaml.tmpl and assets/case.yaml.tmpl to the target directory skills/skill-upper/SKILL.md113-116skill-up validate [path] to catch path errors or schema mismatches skills/skill-upper/SKILL.md126-128 skills/skill-upper/references/cli.md66-72skill-up run with flags like --parallelism, --engine, or --model as requested by the user skills/skill-upper/SKILL.md130-136 skills/skill-upper/references/cli.md7-13grading.json, result.json) to summarize the outcome skills/skill-upper/SKILL.md141-145 skills/skill-upper/references/cli.md158-182Sources: skills/skill-upper/SKILL.md63-145 skills/skill-upper/references/cli.md1-157 internal/cli/init.go1-120
skill-upper bundles template files to ensure generated evaluations follow best practices.
| Asset | Target Path | Description |
|---|---|---|
eval.yaml.tmpl | evals/eval.yaml | Defines the global environment (none, opensandbox), engine, and case file list skills/skill-upper/assets/eval.yaml.tmpl1-50 |
case.yaml.tmpl | evals/cases/<id>.yaml | Defines a specific test case, including the prompt, expected output, and judge type skills/skill-upper/assets/case.yaml.tmpl1-62 |
The skill-up init command uses an internal template initTemplateContent to generate the user config file internal/cli/init.go125-171
Diagram: Configuration Hierarchy and Precedence
Sources: internal/userconfig/loader.go1-10 internal/cli/init.go125-171 internal/userconfig/apply_test.go70-81
skill-upper supports generating complex multi-turn evaluations using the turns and capture features of the skill-up engine skills/skill-upper/references/case-yaml.md40-64
case.yaml skills/skill-upper/assets/case.yaml.tmpl12-23pattern or jsonpath into variables skills/skill-upper/references/case-yaml.md50-63Sources: skills/skill-upper/references/case-yaml.md1-132 skills/skill-upper/assets/case.yaml.tmpl11-24 e2e/testdata/multi-turn-conversation/evals/cases/multi-turn-refinement.yaml8-30
The skill-upper skill is evaluated against itself to ensure it can correctly scaffold and run evaluations for different engines and judge types.
case.yaml to an existing Skill and update the cases.files list in eval.yaml skills/skill-upper/evals/cases/add-case-to-existing.yaml1-31evals/ directory exists skills/skill-upper/evals/cases/scaffold-new-evals.yaml1-23qodercli as the engine in eval.yaml skills/skill-upper/evals/cases/scaffold-with-qodercli-engine.yaml1-19script judge type skills/skill-upper/evals/cases/scaffold-with-script-judge.yaml1-18Sources: skills/skill-upper/evals/eval.yaml13-19 skills/skill-upper/evals/cases/add-case-to-existing.yaml1-31 skills/skill-upper/evals/cases/scaffold-new-evals.yaml1-23
The skill relies on a set of reference markdown files to provide accurate CLI and schema advice to the user:
references/cli.md: Command reference for skill-up run, validate, report, and init skills/skill-upper/references/cli.md1-108references/case-yaml.md: Reference for case.yaml fields including multi-turn capture and post_condition logic skills/skill-upper/references/case-yaml.md1-132Sources: skills/skill-upper/references/cli.md1-108 skills/skill-upper/references/case-yaml.md1-132
Refresh this wiki
This wiki was recently refreshed. Please wait 1 day to refresh again.