Protocol Buffer Generation

Relevant source files

• .protoc-manifest.json
• cmd/generate/main.go
• deps.go
• generator/update_protos.bash

This page describes the automated pipeline used to synchronize, preprocess, and compile Dota 2 Protocol Buffers into Go code. The system utilizes a specialized generation tool to ensure builds are reproducible and compatible with the library's internal architecture.

---

Generation Pipeline Overview

The generation process is orchestrated by a standalone Go command located in cmd/generate. This tool manages the lifecycle of the protobuf definitions from the upstream source to the final generated Go files in the protocol/ directory.

High-Level Workflow

The pipeline consists of several distinct steps executed in sequence by the run function in cmd/generate/main.go cmd/generate/main.go51-106:

1. Submodule Update: Synchronizes the generator/Protobufs git submodule to fetch the latest .proto definitions from the upstream Valve repository cmd/generate/main.go67-72
2. Preprocessing: Modifies the raw upstream .proto files via preprocessProtos to make them compatible with the Go protobuf compiler and the protocol package structure cmd/generate/main.go74-77
3. Dependency Vendoring: Ensures that required protobuf include files (like google/protobuf/descriptor.proto) are available locally in the vendor/ directory via vendorProtobufIncludes cmd/generate/main.go79-85
4. Protobuf Compilation: Runs the protobuf compiler to generate .pb.go files cmd/generate/main.go87-90
5. API Generation (apigen): Executes the custom apigen tool via runApigen to generate high-level Client API and event handlers based on the compiled messages cmd/generate/main.go92-97
6. Formatting: Runs goimports via runAptre to ensure all generated code follows standard Go formatting and import conventions cmd/generate/main.go99-102

Code Entity Mapping: Generation Command

Sources: cmd/generate/main.go22-38 cmd/generate/main.go51-106 cmd/generate/main.go123-191

---

Preprocessing Logic

Upstream Dota 2 protobufs are often incompatible with the standard Go protoc-gen-go plugin due to missing syntax declarations, package names, or specific message structures that cause circular dependencies or syntax errors in the Go ecosystem.

Transformations

The preprocessProtos function and its helper normalizeProto apply several automated fixes to the source files cmd/generate/main.go123-191:

• Syntax & Package: Prepends syntax = "proto2"; and package protocol; to every file cmd/generate/main.go227
• Pointer Fixups: Removes leading dots from type references (e.g., optional .CMsg becomes optional CMsg) to satisfy Go's package scoping cmd/generate/main.go228-231
• Dependency Redirects: Rewrites imports for google/protobuf/valve_extensions.proto to the local valveextensions.proto cmd/generate/main.go232
• Specific File Overrides:
  • steammessages_steamlearn.steamworkssdk.proto: Removes problematic dots from (.CMsgSteamLearn generator/update_protos.bash63
  • dota_gcmessages_common.proto: Removes the CDotaMsgStructuredTooltipProperties message block which is known to cause issues generator/update_protos.bash66
• Custom Option Overrides: Adjusts field IDs for custom options (like maximum_size_bytes or service_description) to avoid collisions with standard extensions generator/update_protos.bash68-78

Source File Selection

The pipeline targets a specific subset of the Valve protobuf repository defined in sourceProtoGlobs cmd/generate/main.go22-38:

Glob Pattern	Purpose
dota_gcmessages_*.proto	Game Coordinator specific messages
dota_client_enums.proto	Core game enumerations
gcsdk_gcmessages.proto	GC SDK and SOCache messages
steammessages.proto	Base Steam protocol messages
econ_*.proto	Economy and item-related messages
events.proto	GC event definitions

Sources: cmd/generate/main.go22-38 cmd/generate/main.go226-233 generator/update_protos.bash25-41 generator/update_protos.bash63-78

---

Tooling and Dependencies

The generation process relies on a specific set of tools and vendored dependencies to ensure consistency across environments.

Protobuf Vendoring

Because the generation process requires standard Google protobuf definitions, the vendorProtobufIncludes function copies the necessary .proto files from the github.com/aperturerobotics/protobuf module into the local vendor directory cmd/generate/main.go193-203 This ensures that protoc can find imports like google/protobuf/descriptor.proto without external dependencies deps.go5-8

Manifest and Tracking

The library uses a .protoc-manifest.json file to track the state of generated files. This manifest includes:

• toolVersions: Records the versions of protoc, protobuf-go-lite, and starpc used during generation \\.protoc-manifest.json4
• hashes: Stores hashes of the input proto files and the flags used, allowing the system to detect if regeneration is necessary \\.protoc-manifest.json3-7
• generatedFiles: A comprehensive list of all .pb.go files that should exist in the protocol/ package \\.protoc-manifest.json9-80

Data Flow: Source to Go Entity

Sources: cmd/generate/main.go193-203 deps.go1-9 \\.protoc-manifest.json1-129 cmd/generate/main.go94

---

Output Structure

The final output is stored in the protocol/ package. Each .proto file results in a corresponding .pb.go file containing Go structs, enums, and descriptor data.

Source File	Generated File	Entities (Examples)
networkbasetypes.proto	networkbasetypes.pb.go	CNETMsg_Tick, CSVCMsg_UserMessage
dota_gcmessages_msgid.proto	dota_gcmessages_msgid.pb.go	EDOTAGCMsg (The primary message ID enum)
gcsystemmsgs.proto	gcsystemmsgs.pb.go	EGCBaseMsg, ESOMsg

The protocol/ directory also contains a protocol.go file, which is a simple package header generated during the preprocessing phase to ensure the directory is recognized as a valid Go package even before compilation cmd/generate/main.go187-190

Sources: cmd/generate/main.go187-190 \\.protoc-manifest.json9-80 generator/update_protos.bash81-82