Making SAM3 8x Faster — What the Profiler Actually Showed

SAM3 — Meta’s third iteration of Segment Anything — is released, and suddenly you need it running fast enough to actually use. Someone grabs the inference notebook from the repo, turns it into a script, and a few hours later: it runs, it segments, results look good. Shipped. Except… it eats 87GB of VRAM on an H200, it’s slow, and the GPU bill is developing opinions of its own.

That’s a familiar situation with any new model release. A working baseline, built fast — doing exactly what vibe-coded baselines do best: running correctly (mostly) and inefficiently. No shame in that. But “it works” and “it runs efficiently” are two very different things.

So here’s where I ended up after a few rounds of profiling and optimization — without touching ONNX, TensorRT, or any quantization.
8x throughput at the default batch size. Memory down from 87GB to 23GB — enough to move off H200s entirely and onto a much cheaper GPU class. More prompts inferred in a single forward pass, and no drop in detection performance.