The Runtime module is responsible for provisioning and managing the execution environment where an agent is evaluated. Because skill-up runs on a host machine while the agent may require isolation (e.g., a specific OS, network restrictions, or a clean filesystem), all interactions with the execution environment are abstracted behind a common interface internal/runtime/runtime.go112-149
The framework follows a Cross-environment isolation principle: the skill-up program is isolated from the runtime environment and cannot directly read/write files or environment variables inside the runtime. All file access must go through the UploadFile / DownloadFile interfaces, and all environment-variable access must go through the Exec interface internal/runtime/README.md3-8
Every backend implementation must satisfy the Runtime interface, which defines the shared contract for lifecycle, transfer, and execution internal/runtime/runtime.go114-115
| Method | Purpose |
|---|---|
Create(ctx) | Provisions the environment (e.g., creates a container or temp dir) internal/runtime/runtime.go119 |
Exec(ctx, cmd, opts) | Executes a command inside the runtime and returns ExecResult internal/runtime/runtime.go138 |
UploadFile / UploadDir | Transfers files from the host to the runtime workspace while preserving permission bits internal/runtime/runtime.go125-132 |
DownloadFile / DownloadDir | Transfers files from the runtime workspace to the host internal/runtime/runtime.go133-136 |
MergeEnv(env) | Seeds the runtime's persistent environment baseline (e.g., PATH) internal/runtime/runtime.go139-145 |
Shell() | Reports the target platform.Shell (OS and Family) for command planning internal/runtime/runtime.go152 |
Sources: internal/runtime/runtime.go112-153 internal/runtime/README.md40-63
The following diagram illustrates how the Evaluator interacts with the Runtime interface and how the factory selects a backend based on the Config struct.
Runtime Selection and Execution Flow
Sources: internal/runtime/runtime.go216-229 internal/runtime/README.md92-130 internal/runtime/runtime.go138
The system supports three primary runtime types, configured via the Type field in the runtime Config internal/runtime/runtime.go166-167
The none runtime executes commands directly on the host machine. It uses a temporary directory (created via os.MkdirTemp) as the workspace internal/runtime/none.go44-65
Shell() returns the host's native platform.Host().Target internal/runtime/none_test.go31-36 It handles platform-specific shell routing via the platform package's HostShell abstraction.The docker runtime runs evaluations inside a local Docker container, providing container-level isolation for filesystem, process, and network internal/runtime/docker.go44-58
docker create, start, and rm. Workspace defaults to /workspace internal/runtime/docker.go26 It uses sh -c to execute commands to ensure compatibility with minimal images internal/runtime/README.md126The opensandbox runtime provisions remote sandboxes via the OpenSandbox SDK. This is used for secure evaluations that require network policies or remote provisioning internal/runtime/README.md108-110
OPENSANDBOX_API_KEY and supports NetworkPolicy configurations like allow_declared with FQDN egress internal/runtime/runtime.go182-183Environment variables are handled consistently across all runtimes. Values are forwarded literally; no shell expansion (like $VAR or ${VAR}) is performed by the runtime itself to ensure consistency between host and containerized environments internal/runtime/runtime.go23-27
Environment Merging Logic
Sources: internal/runtime/runtime.go23-38 internal/runtime/runtime.go61-73
Each evaluation creates a specific directory structure to separate agent outputs from the runtime's internal workspace.
| Path | Description |
|---|---|
{workspace}/ | The root directory inside the runtime where the agent works internal/runtime/README.md14 |
outputs/ | A special subdirectory for agent artifacts like transcript.jsonl or stdout.json internal/runtime/README.md15-19 |
iteration-N/ | The host-side directory where skill-up downloads results internal/runtime/README.md23-31 |
Artifacts are collected at two primary stages: after agent execution (from runtime:outputs/) and after judge execution (from runtime:{workspace}/) internal/runtime/README.md33-38
Sources: internal/runtime/README.md9-38