The skill-up framework produces structured, multi-format output artifacts designed for both human review and automated CI/CD consumption. It adopts an Anthropic-compatible directory layout, allowing results to be compared across iterations and benchmarked with or without specific "skills."
Evaluation outputs are organized within a workspace directory (defaulting to <skill-name>-workspace/). Each run can execute multiple iterations, with results isolated into numbered subdirectories internal/runner/runner.go121-126 internal/report/workspace.go6-21
The IterationWorkspace abstraction manages the following hierarchy:
<skill-name>-workspace/: The root output directory internal/report/workspace.go55-56
iteration-N/: Contains all artifacts for the $N^{th}$ run internal/report/workspace.go89-91
benchmark.json: Machine-readable summary of the iteration internal/report/workspace.go8-205 internal/report/reporter.go83-85benchmark.md: Human-readable Markdown summary internal/report/workspace.go9-211report.html: Visual report for browser-based inspection internal/report/workspace.go10<case-id>/: Directory for a specific test case internal/report/workspace.go11-96
eval_metadata.json: Metadata including the prompt and assertions internal/report/workspace.go12-200 internal/report/grading.go104-109with_skill/: Results when the agent has access to the skill internal/report/workspace.go13-101
outputs/: Contains response.md (agent's final message) internal/report/workspace.go14-182grading.json: Detailed pass/fail results for each assertion internal/report/workspace.go16-191 internal/report/grading.go26-30without_skill/: (Optional) Baseline results without the skill, generated if benchmark.enabled=true internal/report/workspace.go17-106The IterationWorkspace struct internal/report/workspace.go54-63 provides a high-level API for the runner to interact with this filesystem structure without manually joining paths.
| Method | Purpose |
|---|---|
NewIterationWorkspace | Initializes the root and iteration directories. Defaults to current directory if outputDir is empty internal/report/workspace.go68-86 |
EnsureDirs | Creates the with_skill hierarchy for a set of cases and validates Case IDs against path traversal internal/report/workspace.go45-127 |
EnsureDirsWithBaseline | Creates both with_skill and without_skill output structures internal/report/workspace.go130-140 |
WriteResponse | Persists the agent's final text to outputs/response.md internal/report/workspace.go168-182 |
WriteGrading | Serializes the AnthropicGrading result to grading.json internal/report/workspace.go185-191 |
WriteEvalMeta | Saves the prompt and assertions to eval_metadata.json internal/report/workspace.go194-200 |
Sources: internal/report/workspace.go1-215 internal/report/grading.go1-116
skill-up supports several reporting formats. While result.json is always written by the Runner internal/runner/runner.go189-191 others are generated via the Reporter interface internal/report/reporter.go12-14
| Format | Implementation | Primary Use Case |
|---|---|---|
| JSON | JSONReporter | The "source of truth" containing raw execution data internal/report/reporter_test.go87-90 |
| Benchmark | BenchmarkResult | Statistical summary of pass rates and token usage internal/report/reporter.go83-115 |
| HTML | HTMLReporter | Rich visualization of agent sessions and grading evidence internal/report/html.go16-23 |
| JUnit | JUnitReporter | Integration with CI systems like Jenkins or GitHub Actions internal/report/junit.go15-22 |
The following diagram illustrates how raw execution data from the Runner is transformed into artifacts by the IterationWorkspace and the Reporter implementations.
Evaluation Data Flow to Artifacts
Sources: internal/runner/runner.go95-180 internal/runner/runner.go191-205 internal/report/workspace.go168-200 internal/report/reporter.go11-27 internal/report/html.go26-46 internal/report/junit.go24-39
The framework bridges its internal evaluation logic with standard formats via the judge.Result and report.AnthropicGrading structures.
The RuleBasedJudge evaluates assertions internal/judge/rule_based.go33-67 and produces AssertionResult objects internal/judge/judge.go189-198 These are then converted to Anthropic-compatible fields internal/report/grading.go55-82
Internal judge.AssertionResult | Anthropic grading.json | Description |
|---|---|---|
Text | expectations[].text | Description of the assertion internal/report/grading.go34-66 |
Passed | expectations[].passed | Boolean success status internal/report/grading.go35-67 |
Evidence | expectations[].evidence | Concrete reason for the result internal/report/grading.go36-68 |
The judge package defines four primary outcomes for any evaluation internal/judge/judge.go16-28:
Judge Execution Flow
Sources: internal/judge/rule_based.go12-67 internal/judge/judge.go16-28 internal/judge/judge.go112-142
The report command allows users to regenerate artifacts from a previous run's result.json without re-executing the evaluation.
The report.Input struct acts as the unified data boundary for all reporters internal/report/reporter.go17-27 It aggregates:
SkillName, EngineName, ModelName internal/report/reporter.go18-21StartTime, EndTime, TotalTokens internal/report/reporter.go22-25CaseResult objects containing individual grading and turn data internal/report/reporter.go24-72The HTMLReporter uses an embedded HTML template internal/report/html.go35 internal/report/templates/report.html1-86 to create a single-page application. It converts internal results into embeddedCase and embeddedTurn structures suitable for JSON injection into the HTML's JavaScript layer internal/report/html.go81-105
Sources: internal/report/reporter.go1-81 internal/report/html.go1-246 internal/report/junit.go1-211 internal/report/templates/report.html1-86
Refresh this wiki