CLI Layer

Relevant source files

• docs/guide/cli-reference.md
• docs/zh/guide/cli-reference.md
• e2e/userconfig_test.go
• internal/agent/mcp_test.go
• internal/cli/command_integration_test.go
• internal/cli/import.go
• internal/cli/import_test.go
• internal/cli/init.go
• internal/cli/init_test.go
• internal/cli/judge_debug.go
• internal/cli/judge_debug_test.go
• internal/cli/root_test.go
• internal/cli/run.go
• internal/cli/run_test.go
• internal/credential/agent_init.go
• internal/credential/credential_test.go
• internal/credential/provider_query.go
• internal/credential/provider_query_test.go
• internal/evaluator/artifacts_collect.go
• internal/judge/e2e_test.go
• internal/observability/observability_test.go
• internal/report/template_helpers_test.go
• internal/userconfig/apply_test.go
• internal/userconfig/loader.go
• skills/skill-upper/SKILL.md
• skills/skill-upper/references/cli.md

The CLI layer serves as the entry point for skill-up, providing a command-line interface built on the spf13/cobra and spf13/pflag libraries internal/cli/run.go16-17 It orchestrates configuration loading, credential resolution, and the execution of evaluation pipelines.

Command Structure

The CLI is organized into a hierarchical structure of commands, each targeting a specific phase of the evaluation lifecycle.

Core Commands

Command	Purpose	Key Function
run	Executes evaluation cases against an Agent Engine and produces reports.	runEval internal/cli/run.go97
validate	Checks if eval.yaml and referenced cases are schema-compliant.	validateCmd internal/cli/validate.go15
list-cases	Displays a summary table of all evaluation cases in a skill.	runListCases internal/cli/list_cases.go26
report	Regenerates reports (HTML, JUnit, JSON) from an existing result.json.	runReport internal/cli/report.go42
import	Converts legacy formats (e.g., Anthropic evals.json) to skill-up YAML.	importCmd internal/cli/import.go28
init	Scaffolds a user configuration file (.skill-up.yaml).	runInit internal/cli/init.go44
debug judge	Runs a judge in isolation against provided turn data for debugging.	runJudgeDebug internal/cli/judge_debug.go127

Command Execution Flow

The following diagram illustrates the data flow from the CLI entry point through the core execution phases of the run command.

CLI Execution Pipeline (run command)

Sources: internal/cli/run.go123-166 internal/cli/run.go170-195 internal/cli/run.go215-250

Global Flags and Precedence

skill-up implements a multi-layer override system where CLI flags take the highest precedence during the construction of AgentInitParams internal/credential/agent_init.go181-200

Common Flags

• --config <path>: Explicitly point to a .skill-up.yaml user config internal/cli/root.go48
• -v, --verbose: Sets log verbosity using a custom verbosityValue type. -v for debug, -vv for trace internal/cli/run.go46-72 internal/cli/run.go116
• --api-key <string>: Overrides credentials for the model provider internal/cli/run.go111
• --model <string>: Overrides the model (supports provider/name format) internal/cli/run.go110
• --runtime: Overrides the environment.type (none, opensandbox, docker) internal/cli/run.go109
• --iteration: Total number of evaluation runs. 0 = auto (appends after the last existing iteration) internal/cli/run.go118
• --parallelism: Overrides cases.parallelism (range 1-256) internal/cli/run.go112

Precedence Rules

When resolving configuration (e.g., which model to use), the system follows this order:

1. CLI Flags: Highest precedence (marked as ValueSourceCLI) internal/credential/agent_init.go35-36 internal/credential/agent_init.go181-200
2. Environment Variables: Provider-scoped vars (e.g., ANTHROPIC_API_KEY, OPENAI_API_KEY) internal/credential/agent_init.go31-32 internal/credential/provider_query.go64-69 internal/credential/agent_init.go221-224
3. Credential Resolver: Entries from credentials.yaml internal/credential/agent_init.go33-34 internal/credential/agent_init.go202-210
4. Evaluation Config: Fields defined in eval.yaml internal/credential/agent_init.go25-26

Sources: internal/cli/run.go184-211 internal/credential/agent_init.go21-37 internal/credential/agent_init.go153-200 internal/credential/agent_init.go202-224

Implementation Details

The run Command Phases

The runEval function internal/cli/run.go123 divides the process into four distinct phases:

1. Phase 1: Configuration Loading: Uses config.Loader to discover the skill root via SKILL.md and load eval.yaml internal/cli/run.go139 It applies glob filters for --include-case-name and --exclude-case-name internal/cli/run.go180-184
2. Phase 2: Agent Setup: Initializes the credential.Resolver and creates the agent.Agent instance via loadCredentialsAndAgent internal/cli/run.go149 It uses ResolveModelRef to disambiguate whether a slashed model name is a provider/model pair or an opaque ID internal/credential/provider_query.go108-125
3. Phase 3: Evaluation: Calls executeEvaluation, which triggers the runner.Runner to manage iterations and parallelism internal/cli/run.go155
4. Phase 4: Summary: Aggregates results and prints a summary (Passed/Failed/Errored) to the UI internal/cli/run.go161-163

Internal Kwarg Handling

The CLI supports passing arbitrary key-value pairs to Runtimes and Engines via "kwargs":

• --runtime-kwarg (alias --rk): Passed to the environment setup internal/cli/run.go114
• --engine-kwarg (alias --ek): Passed to the Agent Engine (e.g., bypass_sandbox=true for Codex) internal/cli/run.go115

Import Command

The import command internal/cli/import.go24 parses Anthropic-style evals.json and generates case.yaml files. It uses agent_judge as the default judge for imported cases because expectations are typically natural language internal/cli/import.go125-129 It also computes a relative prefix from the skill root to ensure portable eval.yaml paths internal/cli/import.go156 internal/cli/import.go191-207

Public Wrapper Package

The pkg/skillup package provides a public wrapper for common operations, such as running an evaluation suite programmatically. This package simplifies the integration of skill-up into other Go projects by exposing high-level functions like Run pkg/skillup/skillup.go12

Code Entity Association

Sources: internal/cli/run.go1-166 internal/credential/provider_query.go108-125 internal/credential/agent_init.go74-87 internal/credential/provider_query.go52-74 internal/credential/provider_query.go22-26

Debugging and Special Commands

debug judge

Allows developers to test judge logic without running a full agent turn. It reads a JSON file containing judgeDebugInput (which includes transcript, expect, and judge config) and initializes a standalone judge via buildJudgeFromConfig internal/cli/judge_debug.go71-91 internal/cli/judge_debug.go156-159 The command executes the evaluateJudgeWithExpect pipeline (Expect → Judge) and outputs a grading.json result internal/cli/judge_debug.go165-179

report

Used to regenerate artifacts from a result.json. It supports json, junit, and html formats internal/cli/report.go42-80

Validation

The validate command checks schema compliance for the entire suite. Unlike run, which validates only selected cases, validate iterates over every case file referenced in eval.yaml internal/cli/validate.go31-36 docs/guide/cli-reference.md104-106

Artifact Collection

The CLI layer influences artifact collection via the evalCfg.Cases.Defaults.CollectArtifacts list internal/evaluator/artifacts_collect.go31-54 The evaluator uses a portable find . -type f command to list files while excluding the .git directory internal/evaluator/artifacts_collect.go116-117

Sources: internal/cli/judge_debug.go1-120 internal/cli/report.go1-80 internal/cli/validate.go1-42 internal/evaluator/artifacts_collect.go1-145