Evaluation and Judging

Relevant source files

• internal/agent/agent.go
• internal/evaluator/evaluator.go
• internal/evaluator/evaluator_test.go
• internal/judge/agent_judge.go
• internal/judge/agent_judge_test.go
• internal/judge/factory.go
• internal/judge/factory_test.go
• internal/judge/helpers_test.go
• internal/judge/judge.go
• internal/judge/judge_skill.go
• internal/judge/judge_skill_test.go
• internal/judge/rule_based.go
• internal/judge/rule_based_test.go
• internal/mcp/provisioner_test.go
• internal/report/templates/report.html

The evaluation system in skill-up is a multi-stage pipeline designed to verify agent performance against specific criteria. It orchestrates the lifecycle from workspace preparation and agent execution to automated grading and result aggregation.

Evaluation Pipeline Overview

The evaluation pipeline follows a structured sequence to transform a case definition into a graded result:

1. Environment Setup: The Runtime is prepared (e.g., starting a Docker container or initializing a local workspace).
2. Skill Installation: The Skill under test is installed into the environment.
3. Agent Execution: The agent engine (e.g., claude_code, codex) runs the task described in the case via agent.Agent.Run internal/agent/agent.go143
4. Expect Pre-checks: A fast, rule-based check of the agent's output and exit code occurs before formal judging.
5. Judge Evaluation: A detailed assessment using one of the supported judge types (Rule-based, Script, or Agent-based) via judge.Judge.Evaluate internal/judge/factory.go22-49
6. Artifact Collection: Files matching collect_artifacts patterns are downloaded from the workspace.

The judge.Input struct internal/judge/judge.go55-98 serves as the unified data boundary between the execution layer and the evaluation layer, carrying the transcript, workspace diffs, final messages, and exit codes to the judge.

Evaluator Pipeline Flow

Sources: internal/evaluator/evaluator.go179-237 | internal/judge/judge.go55-98 | internal/judge/judge.go112-142 | internal/judge/factory.go22-49

---

Grading Components

Judge Types

skill-up supports three distinct judging strategies, selectable via the judge.type field in configuration. The judge.NewJudge factory instantiates the correct implementation based on the config.JudgeConfig internal/judge/factory.go22-50

Judge Type	Description	Best For
Rule-based	Applies declarative assertions like OutputContains, ExitCode, or ToolCalled internal/judge/rule_based.go17-22	Deterministic output and side-effect validation.
Script	Executes a host-side or runtime script where exit code 0 indicates success internal/judge/factory.go31-35	Complex logic or environment-specific checks.
Agent Judge	Uses an LLM to grade performance against a list of criteria internal/judge/agent_judge.go55-81	Subjective or nuanced quality assessment.

For details, see Judge Types.

Judge Implementation Mapping

Sources: internal/judge/factory.go22-50 | internal/judge/rule_based.go17-30 | internal/judge/agent_judge.go55-81

---

Configuration and Artifacts

Writing Evaluations

Evaluations are defined using eval.yaml (global settings) and case.yaml (individual test cases). Case-level judge configuration takes precedence over global configuration via MergeJudgeConfig internal/judge/factory.go68-87 This allows specific cases to override models, thresholds, or even the judge type entirely.

For details, see Writing Evaluations (eval.yaml and case.yaml).

MCP Integration

The evaluation pipeline can provision Model Context Protocol (MCP) servers using the mcp.Provisioner internal/mcp/provisioner_test.go34-44 Judges can verify if specific MCP tools were called using rule_based assertions like tool_called internal/judge/rule_based.go194-220

For details, see MCP (Model Context Protocol) Integration.

Result Structure

Every evaluation produces a judge.Result struct internal/judge/judge.go112-142 which maps directly to the grading.json output:

• Status: The overall outcome (PASS, FAIL, SKIP, ERROR) internal/judge/judge.go17-28
• AssertionResults: A slice of AssertionResult containing the text, pass/fail status, and concrete evidence for each criterion internal/judge/judge.go189-198
• Summary: Aggregate statistics including passed, failed, and pass_rate internal/judge/judge.go205-210
• JudgeSession: For agent_judge, this preserves the agent.SessionResult of the LLM judge itself for debugging internal/judge/agent_judge.go189-190

Sources: internal/judge/judge.go17-28 | internal/judge/judge.go112-142 | internal/judge/factory.go68-87 | internal/judge/agent_judge.go189-190

---

Debugging and Recovery

The system includes robust error handling for the judging phase. If an agent_judge fails (e.g., due to a timeout), the system attempts to recover the SessionResult using SessionResultFromError internal/judge/judge.go176-182 so that partial logs and partial JSON outputs are still available in the final report. AgentJudge also includes logic to repair or extract JSON from the judge's response even if the agent process returned an error internal/judge/agent_judge.go150-160

The report.html template visualizes these results, displaying the status badges and evidence strings for each assertion internal/report/templates/report.html77-81

Sources: internal/judge/judge.go156-182 | internal/judge/agent_judge.go150-160 | internal/report/templates/report.html77-81