Observability and Telemetry

Relevant source files

• internal/agent/mcp_test.go
• internal/cli/command_integration_test.go
• internal/cli/import_test.go
• internal/cli/root_test.go
• internal/observability/observability_test.go
• internal/report/template_helpers_test.go

The skill-up framework integrates with OpenTelemetry (OTel) to provide deep visibility into the evaluation lifecycle. This includes tracing the execution of agent runs, measuring runtime performance, and collecting low-cardinality metrics. Telemetry is disabled by default and is activated via standard OTEL_* environment variables.

Implementation Overview

The observability subsystem is encapsulated in the internal/observability package. It manages the lifecycle of OTel SDK components, including the TracerProvider and MeterProvider. The system is designed to be "lazy"—if no exporters are configured, it remains a no-op to avoid performance overhead internal/observability/observability_test.go66-90

Data Flow

1. Initialization: Init is called during CLI startup. It parses environment variables and configures the sdk.Resource, TracerProvider, and MeterProvider internal/observability/observability_test.go74-86
2. Resource Attribution: Global attributes (e.g., service name, version) are attached to a Resource. The system supports standard OTEL_RESOURCE_ATTRIBUTES and SKILL_UP_RESOURCE_ATTRIBUTES for overrides internal/observability/observability_test.go168-214
3. Instrumentation: Spans are created for major operations, such as evaluation runs, individual cases, and agent execution.
4. Propagation: Trace context (specifically TRACEPARENT) is injected into the environment of child processes (Agent CLIs) via observability.AgentEnv to ensure distributed traces are linked internal/observability/observability_test.go216-220
5. Shutdown: The shutdown function returned by Init ensures all buffered spans and metrics are flushed to the exporter (OTLP or Console) before the CLI exits internal/observability/observability_test.go74-83

Code Entity Map

The following diagram bridges the observability concepts to the specific entities in the internal/observability package.

Observability System Architecture

Sources: internal/observability/observability_test.go74-148 internal/observability/observability_test.go168-220

Configuration

Telemetry is configured using standard OpenTelemetry environment variables. The framework supports both grpc and http/protobuf protocols internal/observability/observability_test.go150-166

Environment Variable	Description
OTEL_EXPORTER_OTLP_ENDPOINT	The base URL for the OTLP collector internal/observability/observability_test.go38-41
OTEL_TRACES_EXPORTER	Set to otlp to enable tracing; none to disable internal/observability/observability_test.go43-47
OTEL_METRICS_EXPORTER	Set to otlp or console to enable metrics internal/observability/observability_test.go49-64
OTEL_RESOURCE_ATTRIBUTES	Key-value pairs for resource metadata internal/observability/observability_test.go169-170
SKILL_UP_RESOURCE_ATTRIBUTES	Skill-up specific overrides for resource attributes internal/observability/observability_test.go171-172
SKILL_UP_TRACE_TOPOLOGY	Set to single to disable linked trace topologies internal/observability/observability_test.go72-89

Agent-Specific Telemetry

For agents that support their own internal telemetry (like Claude Code), skill-up automatically triggers these features when global tracing is enabled:

• Claude Code: The framework propagates the parent trace ID to ensure seamless spans between the Go runner and the Node.js agent.
• Environment Injection: observability.AgentEnv ensures that variables like OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_LOG_TOOL_DETAILS, and the OTLP endpoint are passed to agent processes internal/observability/observability_test.go216-220
• CLAUDE_CODE_ENABLE_TELEMETRY: This specific flag is often mapped to enable the internal OTel instrumentation within the Claude Code agent.

Sources: internal/observability/observability_test.go30-64 internal/observability/observability_test.go216-220

Tracing and Propagation

skill-up acts as the root of the trace. When an agent (like claude-code or codex) is executed via a Runtime, the framework propagates the trace context.

Trace Propagation Flow

Sources: internal/observability/observability_test.go216-220

Resource Attributes

The framework attaches specific skill_up.* resource attributes to help filter telemetry in backend systems. If SKILL_UP_RESOURCE_ATTRIBUTES is provided, it takes precedence over standard OTEL_RESOURCE_ATTRIBUTES internal/observability/observability_test.go198-214

• service.name: Defaults to skill-up unless OTEL_SERVICE_NAME is set internal/observability/observability_test.go169-184
• service.version: The version string passed to Init internal/observability/observability_test.go173-185
• telemetry.component: Often set to cli via resource attributes internal/observability/observability_test.go189
• skill_up.eval_id: Injected into spans to identify the specific evaluation run.

Metrics

When OTEL_METRICS_EXPORTER is active, the system exports low-cardinality metrics. Supported exporters include otlp (gRPC/HTTP) and console (for local debugging) internal/observability/observability_test.go106-126

1. Metric Exporter Validation: The newMetricExporter function validates the environment configuration, ensuring that the protocol (e.g., grpc, http/protobuf) matches the exporter type internal/observability/observability_test.go128-148
2. Lifecycle: Metrics are flushed during the ShutdownFunc execution to ensure no data is lost upon CLI termination internal/observability/observability_test.go106-126

Sources: internal/observability/observability_test.go106-148

Lifecycle Management

The observability.Init function is the primary entry point. It returns a context.Context (potentially with tracing info) and a ShutdownFunc internal/observability/observability_test.go74-83

Protocol Normalization

The framework supports flexible protocol strings and normalizes them to match OTel SDK expectations internal/observability/observability_test.go150-166:

• grpc → grpc
• http → http/protobuf
• http/protobuf → http/protobuf

Error Handling

If an unsupported protocol or exporter is requested (e.g., OTEL_TRACES_PROTOCOL=ftp), Init returns a descriptive error and a nil shutdown function to prevent resource leaks internal/observability/observability_test.go92-104

Sources: internal/observability/observability_test.go74-104 internal/observability/observability_test.go150-166