This page defines the core terminology, domain concepts, and technical abstractions used throughout the skill-up codebase. It serves as a reference for engineers to map conceptual evaluation workflows to specific Go types and implementation logic.
A Skill is the primary unit of extension for AI agents (like Claude Code or Codex). In the context of skill-up, it refers to the local directory containing a SKILL.md file and an evals/ directory.
SKILL.md README.md122-123evals/eval.yaml is under a directory with SKILL.md, skill-up installs the local skill automatically README.md169An Engine is the specific AI agent implementation being evaluated. skill-up supports built-in engines and custom extensions.
claude_code, codex, qodercli, and qwen_code (aliased as qwen) internal/config/validator.go40-42 CHANGELOG.md112-115local transport) or service (via http transport) that follows the SessionInput/SessionResult JSON contract internal/config/schema.go105-106 CHANGELOG.md121-128local (executing inside the runtime) and http (calling remote services with optional multipart file upload) internal/config/validator.go31-34 CHANGELOG.md121-135A Case is a single test scenario defined in a YAML file (usually under evals/cases/). It contains the input prompt or multi-turn conversation, environment requirements, and grading criteria.
input.turns, including post_condition gates and regex-based capture for variables CHANGELOG.md61-68 internal/config/validator.go124-142CaseConfig internal/config/schema.go237-254Validator ensures unique Case IDs and valid glob patterns internal/config/validator.go45-50 CHANGELOG.md83-91A Runtime is the isolated environment where the Agent Engine and the Skill execution occur.
none (host), docker (local container), and opensandbox (remote sandbox) internal/config/validator.go24-28Runtime interface, providing methods for Create, Exec, UploadFile, and DownloadFile.none runtime terminates process groups gracefully using SIGTERM before escalating to SIGKILL CHANGELOG.md93-102The following diagram bridges the gap between natural language evaluation concepts and the corresponding code entities.
Diagram: Natural Language Space to Code Entity Space
Sources: internal/config/validator.go16-34 internal/config/schema.go11-21 internal/config/validator.go40-42 internal/agent/agent.go133-148
| Term | Definition | Key Code Pointer |
|---|---|---|
| Agent Judge Skills | Reusable grading rubrics and evidence rules installed only into the judge agent via judge.skills. | CHANGELOG.md23-27 internal/config/schema.go149 |
| Baseline | An execution mode (without_skill) used to compare agent performance without the Skill installed. | internal/evaluator/evaluator.go4-5 README.md58 |
| Collect Artifacts | Glob patterns that select workspace files to download into the report directory regardless of success. | internal/config/schema.go128-134 docs/guide/writing-evals.md136-140 |
| Custom Engine | An extension point allowing any CLI tool or HTTP service to act as an agent. | internal/config/schema.go105-106 CHANGELOG.md121-128 |
| Expect Block | A set of rule-based pre-checks (e.g., must_contain) run before the main judge. | internal/config/schema.go257-261 README.md182-186 |
| Judge | The logic determining if a case passed (Rule-based, Script, or Agent-led). | internal/config/validator.go16-21 README.md51 |
| Judge Context | Materials (transcripts, diffs, file references) delivered to an agent_judge using profiles like minimal or standard. | CHANGELOG.md40-44 internal/config/schema.go150 |
| MCP | Model Context Protocol; used to provide tools/context to agents via stdio or http. Supports real and mocked modes. | internal/config/schema.go72-75 CHANGELOG.md11-19 |
| SessionResult | Standardized output from agent execution, including final message, tokens, and turns. | internal/evaluator/evaluator.go67-71 internal/agent/agent.go49-63 |
| Transcript | A recorded sequence of messages during an agent session, used for grading and reporting. | CHANGELOG.md50-54 internal/agent/agent.go60 |
| Workspace | The isolated directory where the skill is installed and commands are executed. | docs/guide/writing-evals.md136-140 internal/evaluator/evaluator.go118-122 |
The Evaluator coordinates the flow of data from the initial CaseConfig through the Runtime and Agent, finally reaching the Judge.
Diagram: Evaluation Data Flow and Implementation
Sources: internal/evaluator/evaluator.go1-32 internal/evaluator/evaluator.go67-106 internal/config/validator.go109-161 internal/agent/agent.go133-148
skill-up uses a multi-layered resolution strategy for API keys and environment variables.
Runtime interface method that allows seeding a persistent environment baseline internal/agent/agent.go109engine.custom.env are masked in responses and logs to prevent leakage CHANGELOG.md127-128SKILL_UP_PROMPT_INLINE_MAX_BYTES (default 32KB) CHANGELOG.md47-48deny_all or allow_declared egress policies for security internal/config/schema.go37-38 internal/config/validator.go77-78Sources: internal/config/schema.go24-39 CHANGELOG.md45-57 CHANGELOG.md121-135 internal/agent/agent.go1-22
Sources: internal/evaluator/evaluator.go1-177 internal/config/validator.go1-106 internal/config/schema.go1-161 CHANGELOG.md1-110 README.md1-205 docs/guide/writing-evals.md1-170 internal/agent/agent.go1-176