base32

base32

A drop-in fast path for standard base32 (StdEncoding, RFC 4648, padded), byte-identical to encoding/base32 (and, on decode, with identical CorruptInputError offsets). Both encoding and decoding run a SIMD kernel generated by go-asmgen; the short tail, padding and all error reporting reuse the standard library, so output matches exactly.

s := base32.EncodeToString(data)      // same bytes as encoding/base32.StdEncoding
b, err := base32.DecodeString(s)      // same bytes AND same error offsets

The encode fast path covers five ISAs across six architectures. ppc64le, s390x and (on Go 1.27+) arm64 run the full kernel — the same algorithm amd64 uses — because POWER (VSX), Z (vector facility) and NEON each provide the per-lane variable shift / integer vector multiply the amd64 path relies on. On arm64 those ops (VUMULL, VUSHL, VTBL) were only added to the Go assembler in Go 1.27, which is precisely what blocked the NEON port until now (see below); on stable Go (≤ 1.26) arm64 encode falls back to encoding/base32. The ppc64le and s390x kernels are qemu-validated (byte-and-error-identical to encoding/base32 via the exhaustive + fuzz differential tests under QEMU power9 / qemu); the arm64 NEON kernel is validated on native arm64 with the gotip (1.27-devel) toolchain. ppc64le is now natively measured on real POWER9 silicon (GCC Compile Farm, https://portal.cfarm.net/ , VSX, Go 1.26.4, 2026-06-26): SIMD decode runs ~5.5× the stdlib scalar decoder (621 vs 113 MB/s) — a real VSX kernel (VSRH) on hardware where arm64 stable can't run one. s390x is now natively measured on real IBM z15 (VXE2) (2026-07-03, -count=6): SIMD decode runs ~8.4× and encode ~3.4× the stdlib scalar baseline.

Algorithm

Per 5-byte group → 8 chars (base32 packs 5-bit groups, so 5 input bytes = 40 bits = 8 output chars):

1. PSHUFB spreads the input into eight 16-bit lanes, each holding the big-endian 16-bit window that contains one output char's 5 bits.
2. A single per-lane unsigned high-multiply (PMULHUW) by lane-specific powers of two shifts every char's 5-bit field down to bits [4:0]; PAND 0x1f isolates it (value 0..31).
3. A second PSHUFB packs the eight values into the low 8 bytes, then a two-range ASCII map turns value v into its char — v<26 -> 'A'+v, v>=26 -> '2'+(v-26) — built as v + 65 - (PCMPGTB(v,25) & 41).

The AVX2 path processes two groups at once (one per 128-bit lane, placed via VINSERTI128 at src+0 / src+5), 2×-unrolled, with a VPERMQ gather and a full 16-byte store per group-pair. Constants come from go-asmgen's emit.File.Data. Verified against encoding/base32 (table + exhaustive + fuzz, on real AVX2 hardware).

ppc64le (VSX) and s390x (vector facility)

Both run the same five-step kernel as amd64; only the spelling changes.

• ppc64le loads with LXVB16X (big-endian element semantics: memory byte 0 = element 0's MSB), spreads with VPERM, and replaces PMULHUW with a direct per-lane variable right shift VSRH — field >> p is exactly what the multiply-by-2^(16-p)-then-take-high-word trick computes, and VSX exposes the variable shift natively. A second VPERM packs, VAND masks, and the ASCII map uses VCMPGTUB/VADDUBM/VSUBUBM. VSX is baseline on POWER8+, so there is no runtime dispatch. (VSX↔VMX aliasing: load into VS(32+k), do arithmetic on Vk.)
• s390x is the one shipped non-amd64 arch with a genuine vector integer multiply-high (VMLHH), so it reproduces the amd64 PMULHUW step almost instruction-for-instruction: VL → VPERM spread → VMLHH by 2^(16-p) → VPERM pack → VN mask → VCHLB/VAB/VSB ASCII map → VST. s390x is big-endian, but VL/VST and the VPERM control vectors are laid out in big-endian lane order (lane 0 = lowest address), which matches the amd64 byte layout because amd64's 16-bit windows are themselves big-endian — so no scan-direction reversal is needed. The cross-lane shuffle is pinned by a position-dependent qemu test. The vector facility is baseline on z13+.

Both kernels emit a full 16-byte store per 5-byte group (only the low 8 chars are kept); the dispatcher caps the group count so the final store stays in bounds and hands the tail to encoding/base32.

arm64 (NEON) — Go 1.27+

The NEON encode kernel is a faithful port of the amd64 five-step path, and is the concrete demonstration of the three NEON ops the released Go arm64 assembler does not expose but Go master / Go 1.27 does — the same gap ppc64le (VSRH) and s390x (VMLHH) already worked around:

• VUMULL / VUMULL2 — the integer widening vector multiply (16×16 → 32). Released Go only assembled the polynomial VPMULL; the integer multiply mnemonics landed upstream in Go 1.27. The kernel multiplies each 16-bit window by 2^(16-p) and keeps the high 16 bits (multiply-high == amd64 PMULHUW).
• VUSHL — the per-lane register-variable shift (one count per lane, taken from a vector register), used to take the high half of each 32-bit product.
• VTBL — table-lookup permute, used twice: once to spread the 5-byte group into eight big-endian 16-bit windows, once as the 32-entry base32 alphabet LUT (a two-register table covering indices 0..31, replacing the two-range ASCII arithmetic the amd64 path uses).

The chain is VLD1 → VTBL spread → VUMULL/VUMULL2 → VUSHL (−16) → VXTN → VAND 0x1f → VTBL pack → VTBL alphabet → VST1. NEON is baseline on arm64, so there is no runtime dispatch. The kernel is gated //go:build arm64 && go1.27; on stable Go (≤ 1.26) the build falls back to the scalar encoding/base32 path (encode_generic.go). Validated byte-and-error-identical to encoding/base32 (table + exhaustive + FuzzEncode) on native arm64 under gotip, where it measures ~2.1× the stdlib scalar encoder.

Decode

Decode is the exact inverse of encode, per 8-char block → 5 bytes:

1. Validate + map ASCII → 5-bit value, per lane, with two unsigned range checks: az = c in 'A'..'Z' and tw = c in '2'..'7'. On amd64 these use the PSUBUSB-then-compare-zero trick (PCMPGTB is signed); ppc64le uses VCMPGTUB+VNOR, s390x uses VCHLB+VNO. value = ((c-65) & az) | ((c-24) & tw); lanes matching neither range are invalid.
2. Spread the eight values into the low byte of each 16-bit lane, then a per-lane left shift by p places each value into its big-endian 16-bit output window — the inverse of the encoder's right shift. amd64 uses PMULLW (multiply low word by 2^p), s390x uses VMLHW (vector multiply low, halfword), ppc64le uses the native variable shift VSLH.
3. Three PSHUFB/VPERM gathers + OR scatter each window's high/low byte into the five output bytes (each output byte takes up to three overlapping contributions).

A block is decoded only if all eight chars are valid; the kernel counts the blocks it consumes and stops before the first block containing a non-alphabet char, and always reserves the final block. The remainder — the final block, any padding (=), the short tail, and everything from the first invalid char — is handed to encoding/base32, with the CorruptInputError offset shifted back over the SIMD-consumed prefix. This keeps decode byte-and-offset identical to base32.StdEncoding.Decode (RFC 4648 padding + error semantics) while still vectorising the common all-valid bulk. The AVX2 path decodes two blocks at once, one per 128-bit lane. Validated against encoding/base32 (table + exhaustive + fuzz) on real AVX2 hardware and under QEMU power9 / qemu for VSX / Z.

Performance

There is no pre-existing pure-Go SIMD base32 to compare against (encoding/base32 ships zero assembly; pkg.go.dev / GitHub turn up only alternative-alphabet, Crockford, or CLI packages — none vectorised). So the comparison is purely this package vs the stdlib scalar encoder.

Encode throughput, 1 MiB buffer, native amd64 (GitHub Actions, AMD EPYC, GOAMD64=v1, -count=6, median MB/s; CI-measured because the dev box is arm64):

(A local emulated QEMU Haswell VM showed only ~1.4×, but QEMU's TCG does not model out-of-order execution — the native figures above are representative.)

Ratio re-confirmation (as of 2026-06-14): re-benched -count=6 medians on a QEMU x86_64 lima VM (so absolutes are TCG-low, but the SSE-vs-stdlib ratio holds): forced-SSE ~243 MB/s vs stdlib ~181 MB/s = ~1.34× — SIMD still beats the scalar encoder even on the worst-case in-order TCG model. (Forced-AVX2 is ~parity under TCG, ~185 MB/s: 32-byte ops gain nothing without OoO execution — exactly why the native EPYC numbers above are the representative ones.) On native arm64 under gotip / Go 1.27 the NEON encode kernel engages and measures ~7400 MB/s vs the stdlib scalar encoder ~3530 MB/s ≈ 2.1× (-count medians on an Apple-silicon dev box; see the arm64 section above). On stable Go (≤ 1.26) arm64 encode is an alias of encoding/base32 (stdlib parity by construction). Verdict: SIMD encode wins on amd64 and on arm64/go1.27+; arm64 on stable Go = stdlib fallback.

The speedup is somewhat below base64's because base32's 5-bit grouping forces an 8-byte store per 5-byte group (vs base64's full 16-byte store per 12 bytes) and a longer serial extract chain — inherent to the format.

Decode throughput (1 MiB, AVX2 path) was re-benched -count=4 on the same QEMU x86_64 lima VM (TCG, in-order — absolutes are low, but the SIMD-vs-stdlib ratio holds): ~108 MB/s vs encoding/base32 ~36 MB/s ≈ 3×, and the SIMD decode is allocation-free (0 B/op) where the stdlib path allocates per call in this harness. Native EPYC numbers (representative, with out-of-order execution) are pending a CI bench run; expect a higher ratio there, as with encode. On native arm64 / loong64 / riscv64, decode is an alias of encoding/base32 (stdlib parity by construction).

• ppc64le: full SIMD encode (above), now natively measured on real POWER9 (GCC Compile Farm, VSX, Go 1.26.4, 2026-06-26): SIMD decode ~5.5× the stdlib scalar decoder (621 vs 113 MB/s), a real VSX kernel where arm64 stable can't run one.
• s390x: full SIMD encode (above), now natively measured on real IBM z15 (VXE2) (2026-07-03, -count=6): SIMD decode ~8.4× and encode ~3.4× the stdlib scalar baseline. POWER and Z supply the per-lane variable shift / multiply-high natively, so they run the whole kernel — and as of Go 1.27 arm64 NEON does too (the ops it was missing finally landed upstream).
• arm64 (Go 1.27+): full NEON SIMD encode (above), validated on native arm64 under gotip, ~2.1× the stdlib scalar encoder.
• riscv64: now natively measured on a real SpacemiT X60 (RVV 1.0, GCC Compile Farm, Go 1.26.4, 2026-06-26). Honest result: scalar parity — decode 27.4 vs stdlib 27.6 MB/s, encode 155 vs 155 (encode falls back to encoding/base32 here, and decode is an alias of the stdlib decoder). No RVV win on this low-power, in-order core (currently the only widely-available RVV 1.0 silicon); the SIMD encode wins stay on amd64, ppc64le and arm64 (1.27).

llvm-mca cycle-model estimate (historical — both ppc64le and s390x now measured)

Static analysis, NOT a hardware measurement — retained for historical comparison. Both non-amd64 SIMD arches here are now natively measured: ppc64le on real POWER9 silicon (GCC Compile Farm, VSX, Go 1.26.4, 2026-06-26 — SIMD decode ~5.5× the stdlib scalar decoder, 621 vs 113 MB/s) and s390x on real IBM z15 (VXE2), 2026-07-03, -count=6 — SIMD decode ~8.4×, encode ~3.4× the stdlib scalar baseline. The cycle-model rows below are superseded by those hardware numbers and kept only for reference. The committed inner loops (one 16-byte vector iteration emits 8 base32 chars from 5 input bytes) were extracted from encode_ppc64le.s / encode_s390x.s and fed to llvm-mca (LLVM 22, which has production PowerPC and SystemZ backends):

llvm-mca -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 <loop.s>
llvm-mca -mtriple=s390x-unknown-linux-gnu -mcpu=z14  <loop.s>

A representative scalar base32 inner loop processing the identical 5-in/8-out group (per-char shift → mask → table load → store) was modelled the same way for an apples-to-apples ×scalar ratio:

Caveats: Block RThroughput is a steady-state throughput ceiling (no front-end, cache, branch-mispredict or dependency-stall modelling); the scalar baseline is a hand-modelled idealised loop (table-driven, no bounds checks), so the real Go scalar fallback would be slower — i.e. these ×scalar figures are conservative lower bounds for the SIMD win. z14's wider vector pipeline + the single-shot VMLHH multiply (vs POWER's VSRH variable-shift dependency chain) is why Z models meaningfully faster than POWER here. Every instruction in both loops is modelled by llvm-mca (no unmodelable op). Treat these as ordering/ballpark estimates only. The ppc64le estimate has now been superseded by a native POWER9 measurement (above), and the s390x estimate by the native IBM z15 (VXE2) measurement (2026-07-03): SIMD decode ~8.4×, encode ~3.4× the stdlib scalar baseline.

• arm64: on stable Go (≤ 1.26) encode falls back to encoding/base32 — the per-char 5-bit fields need a per-lane variable shift and an integer vector multiply, and the released Go arm64 assembler exposed neither a register-form VUSHL nor the integer VUMULL/VMUL, so the multiply-shift trick could not be expressed. Those mnemonics were upstreamed in Go 1.27, so on Go 1.27+ a //go:build arm64 && go1.27 NEON kernel runs the full encode (above). The exact ops ppc64le (VSRH) and s390x (VMLHH) used to work around the gap are now natively available on arm64 too.
• loong64 / riscv64: encode falls back to encoding/base32.
• decode runs the same SIMD coverage on amd64 (AVX2+SSE), ppc64le (VSX) and s390x (vector facility); arm64/loong64/riscv64 use the scalar stdlib decode (there is no NEON decode kernel — the arm64 fast path is encode-only). Error and padding semantics stay exactly identical to the stdlib because every block that isn't a full 8 valid-alphabet chars (and the final block) is delegated to encoding/base32, with CorruptInputError offsets shifted to match.

Coverage

Six SIMD targets, validated on seven architectures. SIMD acceleration stays on the six targets above (amd64, ppc64le, s390x, arm64, loong64, riscv64). On top of those, a seventh architecture — ppc64 (big-endian) — is now build+test validated on real POWER9 silicon (GCC Compile Farm): the portable/generic fallback path is proven bit-exact on a big-endian target distinct from s390x's vector kernel. ppc64 carries no SIMD kernel; it exercises the scalar encoding/base32 fallback, confirming byte-identical output on a big-endian non-vector arch.

The CI gate enforces 100% coverage of the Go code on each arch job: native amd64 + native arm64 (stable), a native arm64 / gotip (Go 1.27-devel) job that compiles and covers the //go:build arm64 && go1.27 NEON kernel, plus emulated ppc64le + s390x (cross-compiled test binary run under QEMU power9 / qemu in a debian:trixie container, coverage profile extracted from the binary). On stable arm64 the !amd64 && !ppc64le && !s390x && !(arm64 && go1.27) generic fallback compiles and is measured; on the gotip job the NEON path is measured instead. Coverage is of the Go statements only: the generated .s SIMD kernels are not measured by go test -cover — they are validated by differential tests against the scalar encoding/base32 reference plus fuzzing (FuzzEncode/FuzzDecode, run on the gotip job so the NEON kernel is fuzzed too).

License

BSD-3-Clause.