microfat

Microfat

Microfat combines multiple CPU microarchitecture-specific ELF binaries (v1, v2, v3, v4 or v8.0..v9.5) into a single, self-dispatching Linux executable with no persistent launcher process, automatic container resource tuning (GOMEMLIMIT & GOMAXPROCS), and payload integrity verification.

Key Highlights

• 🚀 Dynamic Hardware Dispatch: Automatically probes host CPU instruction extensions (AVX2, FMA, BMI2, AVX-512, SVE, SVE2) and selects a compatible variant according to the configured policy.
• ⚡ Zero Persistent Process Overhead: Replaces the launcher through syscall.Exec using a verified existing cache entry or sealed Linux memfd (no wrapper daemon, PID 1 preserved in containers).
• 🛡️ Container Auto-Tuning: Automatically parses Linux cgroup v1 & v2 limits to set a soft Go-runtime memory budget (GOMEMLIMIT) and CPU parallelism (GOMAXPROCS); neither guarantees freedom from OOM kills or CPU throttling.
• ✂️ Flexible Lifecycle Modes:
  • Universal Fat Binary: Distribute one Linux executable per architecture, selecting among the included compatible variants.
  • Trimmed Fat Binary (--microfat:trim / microfat trim): Discard unneeded variants on disk while retaining launcher auto-tuning and cache/memfd dispatch.
  • Raw Native ELF (--microfat:optimize): Permanently specialize to raw uncompressed ELF machine code with no microfat launcher or decompression stage; normal ELF startup still applies.
• 🔒 Payload Integrity Verification: 56-byte trailer with SHA-256 index hashing and variant checksum validation.
• 📦 Shared Inter-Variant Dictionary: Multi-variant compression with trained Zstandard dictionaries; measure size savings on your own variants.

[!NOTE] Payload Integrity vs Producer Authenticity: Embedded SHA-256 digests provide payload integrity verification; they do not authenticate the producer of the fat binary. To establish provenance and origin authenticity in production pipelines, sign fat executables with external tools such as Sigstore Cosign or GPG. See SECURITY.md for security architecture details.

Documentation Guide

Explore the specialized deep-dive documentation in the docs/ and examples/ directories:

Checking for updates

Starting with v0.3.0, microfat update --check reports the latest stable release without changing the installation. Use microfat update to authenticate and apply an update to an installer-owned installation, or brew upgrade microfat for Homebrew. Ordinary commands never check in the background. See update options and recovery.

Quick Start

1. Installation

Homebrew on Linux

The official shared tap is EpicBlackWolfZ/homebrew-tap. The authenticated v0.3.0 recipe is available and its public installation passed on native amd64 and arm64. The Linux cask installs the CLI and both launcher stubs together:

brew install --cask EpicBlackWolfZ/tap/microfat
brew upgrade --cask microfat
brew uninstall --cask microfat

Supported platforms are Linux amd64 and arm64. Qualification uses Homebrew 7.0.2 with Linux cask support. See Homebrew installation and recovery and the v0.3.0 acceptance record.

Verified Linux installer and release archives

The v0.3.0 installer installs the CLI and both companions coherently into user-owned generation storage. It authenticates a pinned verifier and native helper before downloading and verifying the selected release. Public bootstrap installation passed in fresh nonroot Linux containers on native amd64 and arm64 without Go or preinstalled Cosign. Latest/pinned installation, reinstall, companion discovery, memfd/cache execution and conservative uninstall passed with authenticated product hashes.

See installation and recovery for trusted source-helper use, pinned/latest versions, custom destinations, manual-install migration and uninstall. See the release verification contract before executing any downloaded program. The release artifact guide describes the two architecture archives, each containing the CLI plus both launcher stubs. The v0.3.0 acceptance record identifies each qualification boundary.

Via Go Toolchain

go install github.com/EpicBlackWolfZ/microfat/cmd/microfat@latest
go install github.com/EpicBlackWolfZ/microfat/cmd/microfat-stub@latest

2. Detect Host CPU Capabilities & Environment Readiness

# Check CPU microarchitecture level
microfat detect

# Comprehensive host environment verification (CPU, memfd, cache, cgroups)
microfat doctor

# Machine-readable JSON output for CI/CD gating
microfat doctor --json

Illustrative output (values depend on the host and installed version):

=== Microfat Host Environment Doctor ===

[✔] Host CPU Microarchitecture
    • OS/Arch:        linux/amd64
    • Detected Level: v3
    • Key Features:   cx16, popcnt, sse3, ssse3, sse4.1, sse4.2, avx, avx2, bmi1, bmi2, fma, osxsave, f16c, lzcnt, movbe
    • AVX-512 Status: not present (no downclock risk)

[✔] In-Memory Execution (memfd_create)
    • Kernel Support: Available (Linux 6.8.0-generic)
    • Seccomp Filter: Permitted

[✔] Disk Cache Execution Fallback
    • Resolved Path:  /home/deployer/.cache/microfat
    • Permissions:    0700 (read/write OK)

[✔] Container Resource Limits (cgroup v2)
    • Memory Limit:   2147483648 B (2.00 GiB)
    • CFS CPU Quota:  4.00 cores
    • Auto GOMEMLIMIT: 1932735283 B (~1.80 GiB)
    • Auto GOMAXPROCS: 4

[✔] Toolchain & Version Metadata
    • Version:        v0.2.1
    • Commit:         7cba82c
    • Build Date:     2026-09-02T20:00:00Z

Summary: Environment is fully ready for high-performance Microfat dispatch!

3. Compile & Package a Fat Binary

Packaging targets Linux amd64 or arm64. Direct --os and manifest target_os accept only Linux (case-insensitively), with Linux as the default. ELF validation rejects conflicting OSABI values and embedded Go GOOS settings in both payloads and custom launcher stubs. Generic SysV ELF (ELFOSABI_NONE) remains accepted because Linux binaries commonly use it; this value alone does not establish operating-system compatibility or the availability of a required loader or libc. --skip-elf-validation explicitly bypasses ELF evidence checks but does not enable other target systems.

AMD64 (x86_64) Packaging

# 1. Compile variants with Go microarchitecture targets
GOAMD64=v1 go build -o bin/app_v1 main.go
GOAMD64=v3 go build -o bin/app_v3 main.go
GOAMD64=v4 go build -o bin/app_v4 main.go

# 2. Package with microfat
microfat pack \
  --stub bin/microfat-stub \
  --name myapp \
  -v v1=bin/app_v1 \
  -v v3=bin/app_v3 \
  -v v4=bin/app_v4 \
  -o bin/myapp

ARM64 (aarch64) Packaging

# 1. Compile ARM64 variants (Graviton 2/3/4, Apple Silicon, Ampere)
GOOS=linux GOARCH=arm64 GOARM64=v8.0 go build -o bin/app_arm64_v8.0 main.go
GOOS=linux GOARCH=arm64 GOARM64=v8.2 go build -o bin/app_arm64_v8.2 main.go
GOOS=linux GOARCH=arm64 GOARM64=v9.0 go build -o bin/app_arm64_v9.0 main.go

# 2. Package universal ARM64 fat binary with shared dictionary compression
microfat pack \
  --stub bin/microfat-stub-arm64 \
  --name myapp \
  --arch arm64 \
  --dict \
  -v v8.0=bin/app_arm64_v8.0 \
  -v v8.2=bin/app_arm64_v8.2 \
  -v v9.0=bin/app_arm64_v9.0 \
  -o bin/myapp-arm64

Declarative PGO Matrix Packaging (microfat pgo-pack)

Automate variant compilation and Profile-Guided Optimization packaging in a single step using a YAML/JSON manifest:

# pgo.yaml
name: myapp
package: ./cmd/myapp
output: bin/myapp
stub: bin/microfat-stub
variants:
  - level: v1
    pgo: "off"
  - level: v3
    pgo: profiles/v3.pgo
  - level: v4
    pgo: profiles/v4.pgo

microfat pgo-pack --manifest pgo.yaml

4. Try the Demonstration & Benchmark Suite

# Build and run the AMD64 demo workloads:
task demo

# Build the ARM64 demo fat binary:
task demo-arm64

# Run the standard benchmark suite (~110ms):
task bench

# Run the heavy sustained compute benchmark suite (~500ms):
task bench-heavy

# Run the ultra sustained compute benchmark suite (5-15s per run):
task bench-ultra

Standalone Container Tuning (runtimeinit)

For Go services running directly in Docker or Kubernetes without the fat binary wrapper, microfat provides both zero-code startup auto-tuning via runtimeinit/autoload and programmatic tuning via runtimeinit:

Zero-Code Startup Auto-Tuning

Add a blank import to auto-tune debug.SetMemoryLimit (90% container ceiling) and runtime.GOMAXPROCS (CFS quota) before main() executes:

package main

import (
	_ "github.com/EpicBlackWolfZ/microfat/runtimeinit/autoload"
)

func main() {
	// GOMEMLIMIT and GOMAXPROCS are auto-configured
}

Programmatic Auto-Tuning

For explicit control, custom profiles, or headroom overrides:

package main

import (
	"github.com/EpicBlackWolfZ/microfat/runtimeinit"
)

func main() {
	res := runtimeinit.AutoTune(
		runtimeinit.WithProfile(runtimeinit.ProfileLatencyCritical),
		runtimeinit.WithMemoryRatio(0.85),
	)
	_ = res
}

To keep memory and GC tuning while preserving Go's native CPU adaptation, set MICROFAT_CPU_POLICY=native for the launcher or autoload, or use the programmatic option:

res := runtimeinit.AutoTune(runtimeinit.WithCPUPolicy(runtimeinit.CPUPolicyNative))

The default remains static (floor-rounded quota, minimum one CPU). Native mode preserves explicit CPU settings and does not re-enable adaptation disabled by an earlier setter or GODEBUG. See CPU policies and Go-version caveats.

Dry-Run Simulation (WithDryRun)

To simulate container resource tuning and inspect computed parameters without mutating active Go runtime state (debug.SetMemoryLimit, runtime.GOMAXPROCS, debug.SetGCPercent):

res := runtimeinit.AutoTune(
	runtimeinit.WithDryRun(true),
	runtimeinit.WithProfile(runtimeinit.ProfileLatencyCritical),
)
// res.DryRun is true, res.MemLimitApplied is false, res.GOMEMLIMIT and res.GOMAXPROCS are populated

Locating Original Executable & Sibling Assets (runtimeinit.Executable)

When running packaged inside a microfat binary, os.Executable() identifies the extracted payload: a content-addressed cache path in cache mode or memfd:microfat_payload (deleted) in memfd mode. Use runtimeinit.Executable() to retrieve the path to the original fat binary for locating neighboring configuration files, assets, plugins, or sibling CLI binaries:

exePath, err := runtimeinit.Executable()
if err != nil {
	log.Fatalf("cannot determine executable path: %v", err)
}
configPath := filepath.Join(filepath.Dir(exePath), "config.yaml")

[!NOTE] Authenticity Boundary: MICROFAT_ORIGINAL_EXE and runtimeinit.Executable() are informational asset-resolution hints. They must NOT be used as cryptographic identity, access-control tokens, or trusted binary origin claims.

Runtime Meta-Commands

Fat executables using the full launcher support reserved meta-commands for diagnostics and disk specialization:

# View host capabilities, cgroup limits, and embedded variants
./myapp --microfat:info

# Pre-extract selected compatible variant into cache (avoids decompression on a warm cache hit)
./myapp --microfat:prewarm

# Trim unneeded variants in-place (keeps stub & cgroup auto-tuning, reduces artifact size)
./myapp --microfat:trim

# Extract trimmed single-variant fat binary to a target path
./myapp --microfat:trim-to /usr/local/bin/myapp

# Permanently replace with raw uncompressed native ELF
./myapp --microfat:optimize

# Extract raw uncompressed native ELF to a target path
./myapp --microfat:optimize-to /usr/local/bin/myapp

Environment Variable & Policy Configuration

The default auto mode performs a read-only lookup before creating a memfd. A valid hit executes the same descriptor whose size, owner, mode and SHA-256 were checked, without decompressing the source payload or dictionary. An absent directory/entry or corrupt safe regular entry remains untouched and uses sealed memfd. A symlink, special file, foreign owner or unsafe entry mode is rejected, as is an entry whose metadata cannot be inspected after a failed open. Cache hashing, launcher work and normal ELF startup still have a cost.

If memfd creation, sealing or execution fails after a miss, auto may materialize and verify a disk cache entry. If an existing verified cache entry cannot execute, auto tries sealed memfd once; failure reports both attempts without retrying the cache. Corruption found during extraction terminates dispatch. Explicit cache mode and prewarming can populate the cache; a successful cold auto memfd launch does not. Read-only roots therefore work with a usable warm cache or permitted sealed memfd, while cold cache fallback needs writable storage.

Cache execution trusts same-UID writers: pinning the descriptor prevents pathname substitution, but the cached inode remains mutable. Use explicit memfd when immutable extracted storage is required, and authenticate the complete distribution externally before execution. See the execution policy and security boundary. The startup measurements compare complete cold and warm processes across both formats, stub profiles and all codecs with identical tuning.

Compression Profiles & CLI Integration

microfat provides three compression profiles to balance cold-start startup overhead against disk size and network transfer bandwidth:

Startup latency and compression ratios depend on the payload, host and execution mode.

For full benchmarks and sizing recipes, see the Advanced Optimizations Guide.

Programmatic CLI Integration

External Go applications integrate through the supported CLI. Packages under internal/ are private to this repository; a public Go packaging API is not provided.

Run this program with four arguments: the CLI path, the matching launcher stub path, a native baseline payload path, and the output path. The example targets Linux AMD64; use the matching architecture and variant level for other targets.

package main

import (
    "log"
    "os"
    "os/exec"
)

func main() {
    if len(os.Args) != 5 {
        log.Fatal("usage: pack-example CLI STUB PAYLOAD OUTPUT")
    }
    cmd := exec.Command(os.Args[1], "pack", "--stub", os.Args[2],
        "--os", "linux", "--arch", "amd64",
        "-v", "v1="+os.Args[3], "-o", os.Args[4])
    cmd.Stdout, cmd.Stderr = os.Stdout, os.Stderr
    if err := cmd.Run(); err != nil {
        log.Fatalf("packaging failed: %v", err)
    }
}

Use absolute input paths when invoking this example from another working directory. The E2E suite compiles and executes this exact snippet outside the module, then inspects, verifies and launches its output. Measure process startup separately from sustained throughput; compare ISA specialization with identical runtime settings, and evaluate tuning on/off as a separate experiment.

GoReleaser Integration

Add microarchitecture matrix builds to your .goreleaser.yaml for both AMD64 and ARM64:

version: 2

builds:
  - id: myapp-amd64
    main: ./main.go
    binary: myapp
    goos: [linux]
    goarch: [amd64]
    goamd64: [v1, v3, v4]

  - id: myapp-arm64
    main: ./main.go
    binary: myapp
    goos: [linux]
    goarch: [arm64]
    goarm64: [v8.0, v8.2, v9.0]

Development & Developer Workflow

All daily development operations use Task through the root Taskfile.yml. Install the version in the shared tool pins with source .config/tool-versions.env && go install github.com/go-task/task/v3/cmd/task@v${TASK_VERSION} and put $(go env GOPATH)/bin on PATH. See contributor setup for required validation tools.

# View all targets and descriptions (respects NO_COLOR=1 and COLOR=0)
task help

# Run complete verification pipeline: tidy-check, fmt-check, lint, vuln, test, coverage gate, build
task all

# Format code across the repository with $(GOFMT) -s
task fmt

# Verify all Go source files are formatted cleanly (non-mutating)
task fmt-check

# Run Go API modernizations ('go fix'), linter auto-fixes, and format code
task fix

# Verify Go module dependencies are tidy (non-mutating 'go mod tidy -diff')
task tidy-check

# Run tests with Go 1.27 goroutine leak detection enabled (MICROFAT_TEST_LEAKS=1)
task test-leaks

# Check for Go 1.27 goroutine leaks on the running benchmark workload
task check-leaks

# Run regression tests for developer workflows (port safety, formatting, TTY detection)
task test-dx

# Open interactive browser UI for pprof profiles (heap, cpu, goroutine, allocs, mutex, block, goroutineleak)
# (block and mutex sampling rates are enabled only when PROFILE=block or PROFILE=mutex)
task pprof PROFILE=heap
task pprof PROFILE=block PORT=6061 HTTP_PORT=8081

# Verify finished fat binary stub commands (info, optimize-to, trim-to, prewarm)
task demo-check

Terminal Detection & Port Safety:

• Piped or redirected stdout automatically disables ANSI escapes and uses ASCII markers ([OK], [FAIL]). Use NO_COLOR=1 or COLOR=0 to force disable colors.
• Profiling targets check port occupancy before spawning servers and fail cleanly without terminating unrelated processes. Cleanup sends graceful SIGTERM before SIGKILL strictly to target-spawned child processes.
• Developer workflow regressions (test-dx, test-leaks, check-leaks, demo-check) are CI-enforced in GitHub Actions.

License

Apache 2.0