The Custom Engine extension point allows developers to integrate any agent executor—whether a local CLI, a script, or a remote service—into the skill-up evaluation pipeline. By implementing a standard JSON contract, a custom engine becomes interchangeable with built-in agents like claude_code or codex.
The CustomAgent implementation manages the lifecycle of a custom engine run. Unlike built-in agents that have hardcoded CLI behaviors, the CustomAgent relies on a customTransport to execute the agent and a JSON-based protocol to exchange data internal/agent/custom.go65-79
The following diagram illustrates how skill-up bridges the gap between the evaluation configuration and the external agent process.
Diagram: Custom Engine Local Transport Flow
Sources: internal/agent/custom.go128-130 internal/agent/custom_local.go26-111 internal/runtime/runtime.go129
Integration requires adherence to two core data structures: SessionInput and SessionResult.
The framework writes this JSON to a location specified by ${input_file} (defaulting to inputs/messages.json internal/agent/custom.go30). It includes:
messages: The full conversation history (transcript).kwargs: Key-value pairs passed from eval.yaml or case.yaml.env: Environment variables (secrets are scrubbed unless explicitly injected).The agent must return a JSON object (either via stdout or a file) containing:
exit_code: (Integer) 0 for success, non-zero for failure.final_message: (String) The agent's final answer to be graded.turns: (Integer) Number of steps taken.artifacts: (Object) Files or inline content to be collected for the report.Sources: internal/agent/custom.go28-31 docs/design/custom-engine.md135-143 internal/agent/custom_local.go89-90
Custom engines are defined in eval.yaml under the engine.custom block. skill-up supports dynamic variable injection using ${variable} syntax. The ParseTemplateToken function handles the ${X:-default} and ${X?msg} grammar internal/customengine/template_token.go32-40
| Variable | Description |
|---|---|
${workspace} | Absolute path to the current execution workspace. |
${input_file} | Path where skill-up writes the SessionInput JSON. |
${output_file} | Path where skill-up expects the SessionResult JSON. |
${api_key} | Resolved credential for the configured provider. |
${kwargs.<key>} | Values defined in the engine.custom.kwargs map. |
Sources: internal/agent/custom_local.go37-41 docs/design/custom-engine.md180-199 internal/config/customengine.go129-148
docs/design/custom-engine.md73-93
The response_format field determines how skill-up interprets the agent's output:
session_result (Default): The engine expects a valid JSON matching the SessionResult struct. If the engine exits with code 0 but provides no JSON, the run is marked as a failure with the error custom engine returned a result without exit_code internal/agent/custom.go28 internal/agent/custom.go32text: The entire stdout of the process is captured as the final_message. The process exit code is used as the exit_code. This is useful for wrapping simple scripts that don't support JSON internal/agent/custom.go29The http transport allows calling remote agent services. It POSTs a JSON body (defaulting to the SessionInput) and expects a SessionResult JSON in response internal/agent/custom_http.go15-20
engine.custom.http.request_body. If set to ${session_input}, the full JSON structure is injected as an object rather than a string internal/agent/custom_http.go177-184Authorization: Bearer ${api_key} internal/agent/custom_http.go109-115engine.custom.http.files. The buildMultipartBody function assembles a payload field for the JSON and files parts for workspace content internal/agent/custom_http_files.go27-54url field to prevent leakage in logs/proxies/traces internal/agent/custom_http.go51-53Sources: internal/agent/custom_http.go34-84 internal/agent/custom_http.go129-175 internal/agent/custom_http_files.go27-54
skill-up prevents accidental secret leakage by validating template variables. The CustomAgent uses maskAPIKey to scrub credentials from stderr and error messages internal/agent/custom_http.go139-144 Additionally, ResolveCustomEngineConfig rejects sensitive environment variables or credentials used directly in command-line templates internal/config/customengine.go166-174 isSensitiveTemplateVar identifies variables like ${api_key} or ${kwargs} that must not be rendered into command lines internal/config/customengine.go87-98
To prevent "ghost" results from previous runs or fixtures, the localTransport performs a pre-run cleanup. If a specific output_file is configured, skill-up attempts to delete it before executing the agent internal/agent/custom_local.go53-56
__skill_up_stale_output_cleared__ is logged internal/agent/custom.go35output_file points to the same path as input_file to avoid self-deletion internal/agent/custom_local.go54-56The framework enforces limits on inline artifacts and downloads to prevent resource exhaustion:
maxInlineArtifactBytes: Caps a single inline file at 50MB internal/agent/custom.go40-43maxHTTPResponseBytes: Caps HTTP response bodies at 64MB internal/agent/custom.go47-50maxHTTPUploadBytes: Caps total multipart upload size at 256MB internal/agent/custom_http_files.go21maxArtifactDownloadBytes: Caps external URL artifact downloads at 256MB internal/agent/custom.go63The CustomAgent struct in internal/agent/custom.go implements the Agent interface but overrides lifecycle methods like InstallMCP and CheckCredentials to be no-ops since custom engines manage their own environment and credentials internal/agent/custom.go77-100
Diagram: Entity Relationship
Sources: internal/agent/custom.go77-85 internal/agent/custom.go128-130 internal/agent/custom_local.go17-19 internal/agent/custom_http.go25-27
Refresh this wiki
This wiki was recently refreshed. Please wait 1 day to refresh again.