This page covers the parallel data generation pipeline used to run the Hermes agent across large prompt datasets, producing trajectory files for model training. It describes BatchRunner, toolset distributions, the checkpointing system, output formats, and the post-processing pipeline.
For details on the trajectory compression algorithm (token budgeting, summarization, TrajectoryCompressor), see Data Generation and Trajectories. For the agent conversation loop that each worker invokes, see Core Agent.
The batch processing system runs AIAgent in parallel across a JSONL prompt dataset, saves per-prompt conversation trajectories, and aggregates tool usage and reasoning statistics. It is the primary mechanism for generating supervised fine-tuning data.
Entry point: batch_runner.py batch_runner.py1-21 invoked via python batch_runner.py or imported as BatchRunner.
Supporting modules:
toolset_distributions.py — defines named probability distributions over toolsets toolset_distributions.py1-20model_tools.py — provides TOOL_TO_TOOLSET_MAP for tool discovery and statistics normalization batch_runner.py61-65 model_tools.py14toolsets.py — defines the core toolset groupings used in distributions toolset_distributions.py23 toolsets.py95-189trajectory_compressor.py — post-processes trajectories to fit within context windows trajectory_compressor.py1-31mini_swe_runner.py — a specialized runner for SWE benchmark data collection using Hermes execution environments mini_swe_runner.py1-27Sources: batch_runner.py1-21 toolset_distributions.py1-20 batch_runner.py61-65 toolsets.py95-189 trajectory_compressor.py1-31 mini_swe_runner.py1-27
The BatchRunner utilizes multiprocessing.Pool to parallelize agent execution batch_runner.py41 Each worker process instantiates an AIAgent batch_runner.py49 with specific configuration overrides to ensure session isolation, such as skipping persistent memory files (MEMORY.md) and local context files to prevent data contamination batch_runner.py334-457
BatchRunner Pipeline
Sources: batch_runner.py41-49 batch_runner.py334-457 batch_runner.py125-228
For details, see Batch Runner.
toolset_distributions.py defines named probability configurations toolset_distributions.py1-20 Each distribution maps toolset names to an independent inclusion probability (0–100%). On each prompt, sample_toolsets_from_distribution() rolls independently for each toolset to decide its inclusion in the current run toolset_distributions.py50-54
Distribution Sampling Logic
Sources: toolset_distributions.py50-54 batch_runner.py50-54
For details, see Toolset Distributions.
The runner tracks completed prompts to support fault-tolerant resumption. Resume logic matches on prompt text content and indices to ensure the run picks up exactly where it left off batch_runner.py8-12
checkpoint.json stores progress, run metadata, and aggregated statistics batch_runner.py8-12try/except blocks in the worker to allow for retries or logging without crashing the entire batch batch_runner.py440-457Sources: batch_runner.py8-12 batch_runner.py440-457
_extract_tool_stats)The runner walks the messages list to increment usage counts batch_runner.py125-158 Success is determined by the absence of an "error" field in tool responses, with special handling for terminal tool responses (nested content) and JSON success flags batch_runner.py160-188 Stats are normalized via _normalize_tool_stats() using ALL_POSSIBLE_TOOLS (derived from TOOL_TO_TOOLSET_MAP) to ensure a consistent schema for HuggingFace datasets batch_runner.py65-98
_extract_reasoning_stats)The system identifies reasoning via scratchpad tags or native thinking tokens extracted from the assistant messages batch_runner.py195-228 Trajectories without reasoning are typically discarded to maintain dataset quality batch_runner.py387-393
Sources: batch_runner.py125-158 batch_runner.py160-188 batch_runner.py65-98 batch_runner.py195-228 batch_runner.py387-393
For details, see Data Generation and Trajectories.
The batch processing system bridges high-level tool configuration defined in hermes_cli/tools_config.py with the runtime toolset resolution in toolsets.py.
Tool Mapping and Configuration
Sources: batch_runner.py50-55 toolsets.py31-73 model_tools.py12-21 trajectory_compressor.py83-124 mini_swe_runner.py68-110 hermes_cli/tools_config.py55-83
Basic batch run:
Resuming an interrupted run:
Compressing generated trajectories:
Running SWE tasks with sandboxed environments:
Sources: batch_runner.py14-21 trajectory_compressor.py16-31 mini_swe_runner.py16-27
Refresh this wiki
This wiki was recently refreshed. Please wait 4 days to refresh again.