This page provides a technical deep dive into the bundled examples and reference implementations within the skill-up repository. These examples demonstrate the integration between the evaluation engine, various judge strategies, and multi-platform execution environments.
The code-stats skill serves as the primary reference for a real-world evaluable skill. It implements a dual-format configuration that allows it to be used by both the legacy skill-creator (JSON-based) and the skill-up Go framework (YAML-based).
The skill is organized to separate the skill definition, test cases, and evaluation logic.
Sources: examples/code-stats/README.md7-24 examples/code-stats/evals/eval.yaml21-25
The code-stats example maintains compatibility across two different evaluation paradigms:
evals/evals.json which contains a flat evals[] array examples/code-stats/README.md50evals/eval.yaml following the v1alpha1 schema examples/code-stats/evals/eval.yaml1For skill-up, the configuration defines a none environment type (host execution) and utilizes the qodercli engine examples/code-stats/evals/eval.yaml3-17
The evaluation relies on evals/fixtures/scripts/check-stats.sh. This script is executed after the agent completes its task. It validates the output based on several criteria:
| character examples/code-stats/README.md66Sources: examples/code-stats/README.md48-68 examples/code-stats/evals/eval.yaml38-41 skills/skill-upper/references/judge-types.md69-80
The repository includes an end-to-end fixture for Custom Engines, demonstrating how to integrate non-native agents via local or http transports. This is used to test the skill-up engine abstraction layer.
In the local transport mode, skill-up executes a command within the runtime environment. The e2e/testdata/custom-engine/evals/eval.yaml demonstrates the configuration:
${CUSTOM_AGENT_BIN} e2e/testdata/custom-engine/evals/eval.yaml15SessionInput JSON to a temporary file, passed as an argument e2e/testdata/custom-engine/evals/eval.yaml17agent.sh) emits a SessionResult JSON to stdout e2e/testdata/custom-engine/agent.sh18-28The http transport allows skill-up to communicate with a remote agent service.
eval-http.yaml specifies a POST method and an endpoint URL e2e/testdata/custom-engine/evals/eval-http.yaml11-18SessionResult which is parsed and passed to the judge docs/design/custom-engine.md104-110Sources: docs/design/custom-engine.md98-115 e2e/custom_engine_test.go146-164
The repository includes tools for debugging evaluation logic and managing large-scale context delivery, particularly for agent_judge and script_judge types.
To support both POSIX and Windows environments, skill-up provides equivalent logic in Shell and PowerShell. The judge-debug-eval.ps1 script demonstrates how to access the agent's final response via the EVAL_FINAL_MESSAGE environment variable examples/judge-debug-eval.ps15
Sources: examples/judge-debug-eval.ps11-18
The bundled documentation provides a decision tree for selecting the appropriate judge type:
Sources: skills/skill-upper/references/judge-types.md7-12
skill-up reportReference implementations generate structured artifacts that can be analyzed using the skill-up report command.
The following diagram illustrates how the engine translates agent sessions into the final report artifacts.
Sources: examples/code-stats/evals/eval.yaml43-47
In the code-stats example, the eval.yaml specifies:
json and html examples/code-stats/evals/eval.yaml44transcript is explicitly collected for post-hoc review examples/code-stats/evals/eval.yaml45-46Sources: examples/code-stats/evals/eval.yaml35-47 examples/code-stats/README.md43
Refresh this wiki