When the AI Model Dies, the Stream Stays Up: Rust + GStreamer Failover for Live Video.
I built “Braidpipe” so a crashed, stalled, or slow model costs you the effect, never the broadcast. Here’s how it works, what it costs, and where it still falls short.
At 30 frames per second, a frame gets 33.3 milliseconds before the next one shows up. Put an AI model inside that path and every ordinary software problem turns into a broadcast problem. An unhandled exception, one slow inference, a queue that fills up, a process restart. Any of them can freeze the picture or drop the publisher connection. The worst thing that happens with AI on a live stream isn’t a missing bounding box. It’s dead air.
I started braidpipe in July with one rule written at the top of the design notes before any code existed: losing inference should mean losing the enhancement, not the broadcast. Everything else in the project is a consequence of taking that sentence seriously.
braidpipe is an Apache-2.0 middleware written in Rust on top of GStreamer, with a small Python worker SDK. It ingests a live stream, hands every frame to an AI or computer-vision process, re-encodes the result, and streams it out. If that process crashes, hangs, never starts, or just misses its deadline, the original frames keep flowing to the viewer.
This post is about the two design decisions that make that hold, the measurements that show what it costs, and the honest boundary of the word “guaranteed”.
Rule one: the model doesn’t get to own the stream
The obvious AI-video pipeline is one chain:
decode → Python inference → encode → publishIt’s simple. It also welds together two systems with completely different timing behaviour. A media pipeline wants bounded work per frame, continuous timestamps, and predictable backpressure. A model runtime allocates memory, warms up, downloads weights, stops for garbage collection, and takes wildly different time from one frame to the next.
When those two share a critical path, Python quietly inherits responsibility for stream continuity. RuntimeError Somewhere in inference code becomes a black screen for the audience. I didn't want a stream whose uptime depended on a try/except I'd forgotten to write.
So braidpipe flips the ownership. A Rust daemon owns the media path for the whole life of the stream. Python is a replaceable worker that’s allowed to contribute frames only while it’s healthy and on time.
┌─────────────────────────────────────────────────────────┐
SRT / UDP / RTP │ braidpipe │
NDI / camera / file │ │
──────────► │ decode ──┬── queue ─────────┐ │
│ │ ▼ │
│ │ input-selector ──► encode ──► sink│──────► SRT / RTMP
│ │ ▲ │ UDP / display
│ └── appsink ──┐ appsrc │
│ │ │ │
└─────────────────────────┼────┼──────────────────────────┘
shared memory + Unix datagrams
▼ │
┌───────────┴────┐
│ Python worker │
│ (numpy/cv2/…) │
└────────────────┘A GStreamer tee splits the decoded video into two. The passthrough branch carries raw frames straight to an input-selector. The AI branch leaves GStreamer through anappsink, crosses into the worker over shared memory, and comes back through an appsrc. The selector decides, frame by frame, which of the two the viewer sees.
That’s the entire availability boundary. The passthrough branch has no dependency on Python at all, so there’s nothing Python can do to take it down.
Rule two: guarantee at two levels, not one
One late frame and a dead worker are different problems, and I ended up handling them at different levels. Trying to cover both with a single mechanism was my first design, and it was wrong in both directions: too twitchy for jitter, too slow for a crash.
Level 1: every frame has a deadline
For each frame, the relay writes RGB pixels into a shared-memory slot, pokes the worker with a small datagram, and waits for an acknowledgement. The budget is 1.5 frame periods, so 50 ms at 30 fps.
If the ack arrives in time, the processed pixels go back into GStreamer. If the deadline passes, or shared memory is full, or the worker says it failed, the relay pushes the original, untouched frame downstream and takes the slot back. The heart of the loop is short enough to show in full:
// crates/braidpipe/src/relay.rs (abridged)
let deadline = Duration::from_secs_f64(1.5 / f64::from(fps));while let Some(frame) = rx.recv().await {
frame_id += 1;
// `data` doubles as the output buffer: on any AI failure it still
// holds the original pixels, which then pass through unchanged.
let mut data = frame.data; match roundtrip(&shm, &bridge, frame_id, &mut data, deadline).await {
Ok(_) => bridge.record_success(),
Err(e) => {
bridge.record_failure();
debug!(%e, frame_id, "AI roundtrip failed; passing frame through unchanged");
}
} ai_src.push_buffer(gst::Buffer::from_mut_slice(data))?;
}
I like that the failure path is the absence of a copy. A slow inference costs one un-enhanced frame, never a stalled pipeline. The rule underneath applies to most live systems:
When stale work has no value, degrade the result instead of extending the queue.
The control channel is Unix datagrams for the same reason. A late notification can just be dropped. Reliability belongs to the stream, not to per-frame work that has already expired.
Level 2: a watchdog for sustained failure
The relay also reports every success and failure to a health counter. After 30 consecutive failed round trips, about a second at 30 fps, the watchdog marks the worker unhealthy and flips input-selector to the passthrough branch. The first successful round trip makes the AI branch eligible again.
One detail I’m particularly happy with: the AI branch starts unselected. It has to earn its place with a first good frame. A worker that never attaches properly can’t take the output down during startup, and a worker that dies is simply never heard from again. The stream doesn’t notice, and honestly, neither did I the first time. I had to go check the logs to confirm the kill had worked.
There’s also one GStreamer detail that matters more than the state machine, and it cost me most of an afternoon. Both branch queues have to be leaky=downstream. Without that, buffers pile up on the inactive selector pad until the queue blocks thetee, and the safety branch stalls the branch it was supposed to protect. Failover isn't just a watchdog. Backpressure policy is part of the availability design, whether you planned it or not.
Fast: what the isolation actually costs
The obvious objection to process isolation is latency. So I measured it, without OCR and without a second clock. A worker stamps the wall clock into every frame as a row of big black-and-white cells, a probe reads the cells back out of the decoded output, and the difference is a true one-way number on one machine. Big cells matter: H.264 smears a thin line, but block-coded black and white survives any bitrate worth streaming. The probe decoded 100% of frames.
M-series Mac, 1280×720 at 30 fps,x264enc tune=zerolatency speed-preset=ultrafastmedians over about 700 frames:
Braidpipe’s own contribution is about 6 ms. Everything else is the encoder and the transport. The IPC is not where the time goes, which is what I hoped but hadn’t proven until this.
Get Mohammadtaghi Yousefi’s stories in your inbox
Join Medium for free to get updates from this writer.
Two things from the same harness surprised me:
sync=falseOn the sink was worth 48 ms by itself. A syncing sink holds every buffer until its running time plus the pipeline latency. A live source already paces the pipeline, so the clock has nothing to add and only delays. Median went from 89.0 ms to 41.1 ms. I'd been leaving that on the table for weeks.- Hardware encoding was not the low-latency choice. VideoToolbox measured 46.5 ms at the receiver against 40.5 ms for ultrafast x264. Hardware offload saves CPU, which is a different property. Measure the one you actually need.
The harness doubles as a failover test with numbers attached. Give the worker a fake 60 ms per-frame cost, well past the 50 ms budget, and every stamped frame disappears because the AI branch never makes its deadline. The output still arrives at a 33.33 ms median interval. That is 30 fps exactly. Nothing downstream can tell the worker failed.
Light: no broker, no framework, a few megabytes
Raw video is heavy. A 720p RGB frame is about 2.64 MiB, so serialising every frame through a message queue would move roughly 79 MiB per second before the return trip. I looked at brokers early on and walked away for exactly this reason. braidpipe uses a fixed four-slot shared-memory ring instead: about 10.5 MiB at 720p, about 23.7 MiB at 1080p, and bounded rather than growing with a backlog.
The worker never receives pixels over a socket. It gets a frame ID and a slot index, and NumPy builds a view straight over the slot:
import braidpipedef process(frame): # (H, W, 3) uint8, RGB, a view over shared memory
frame[:, :, 0] //= 2 # your inference here; mutate in placeif __name__ == "__main__":
braidpipe.run(process)
Everything else the contract needs- the handshake, the per-frame datagram, releasing the slot, the ack, and reporting an exception as a failed frame so the stream falls back instead of dying- is the SDK’s job. The SDK depends on NumPy and nothing else. That was a deliberate line I refused to cross.
The rest of the footprint:
- The release daemon is one 3.4 MB binary. No runtime, no sidecar, no broker.
- Shared memory is anonymous, and its file descriptor travels over the control socket, so the Docker setup needs no
/dev/shmmount and noipc: host. Daemon and worker share a single socket volume. - The Python view really is zero-copy. The Rust side isn’t: each frame gets copied from the GStreamer buffer into the slot and back. Calling the whole path “zero-copy” would hide the next optimisation, so I don’t.
- When all four slots are full, the relay doesn’t allocate a fifth or wait. It emits a passthrough frame and bumps a counter you can alert on.
Slow models belong off the frame loop
Most real models don’t finish in 50 ms on a CPU. Dropping the input frame rate is one answer, but it’s rarely the right one. The SDK ships a background-model pattern instead:
model = braidpipe.BackgroundModel(infer, initial=[])def process(frame, ctx):
if ctx.frame_id % 3 == 0:
model.submit(frame) # copied to an inference thread, dropped if busy
draw(frame, model.latest()) # annotate every frame with the newest resultThis separates two rates that get confused constantly. The video rate has to stay continuous. The inference rate can be lower and can wobble. Detections lag the picture by a frame or two, and no frame ever misses its deadline. For detection, tracking, and telemetry overlays, that trade is almost always the right one.
Prove it in 60 seconds
The most convincing demo is the destructive one.
git clone https://github.com/emdiple/braidpipe.git && cd braidpipe
cargo build --release
python3 -m venv .venv && .venv/bin/pip install -e python/ opencv-python# a visible worker: a Canny edge transform on the left half of the frame
cargo run -p braidpipe --release -- --python-script examples/worker_edges.pyWait for the log line,Python worker active pid=…then kill -9 that PID. The edge effect vanishes about a second later. The video keeps moving. The publisher stays connected. That's the property the whole architecture exists to protect.
A small trap here that I fell into: don’t pkill -f worker.py. The daemon's own command line contains that path, so you kill the daemon too and prove nothing. Use the PID from the log.
The same daemon does a real broadcast shape, SRT in and SRT out, with the encoder built from a latency preset:
cargo run -p braidpipe --release -- \
--uri 'srt://0.0.0.0:9000?mode=listener' \
--preset lowlatency --output 'srt://0.0.0.0:8891?mode=listener'In the bundled Docker Compose stack, the worker is its own container with restart: unless-stopped. A hard worker crash there measured about 365 ms of passthrough before the restarted worker said hello again and the AI branch came back. The daemon also exposes Prometheus metrics for the active branch, deadline misses, queue depth, slot occupancy, round-trip histograms, output fps, A/V skew, and SRT transport stats, with a Grafana dashboard and alert rules in the repo. The events that drive failover are the same events operators get to see.
What “guaranteed” honestly means
No user-space program can promise that a camera, host, network, encoder, or power supply will never fail. braidpipe’s guarantee is deliberately narrow:
While the source and the Rust/GStreamer media path are healthy, failure or lateness in the AI worker does not interrupt the output stream.
That’s failure containment, not total availability. The current limits are just as explicit: one stream and one worker per daemon, full-frame RGB only, two Rust-side copies per frame, no built-in worker respawn, and an uncompressed remote-worker transport that’s LAN-only in practice. I’d rather list those than print “production-ready” on the README. They also tell a contributor exactly where the next useful pull request is.
The pattern, in five lines
Rust and GStreamer fit this problem because the long-lived media path gets explicit ownership, bounded memory, and predictable state transitions, while Python stays where it’s strongest, which is the model ecosystem. But the lesson underneath doesn’t care about the language.
- Keep the essential service outside the optional computation’s failure domain.
- Put a deadline on work whose value expires.
- Build the degraded path first, and make the good path earn its way in.
- Bound every queue and choose leak behaviour on purpose.
- Measure end-to-end, then state guarantees only inside the tested boundary.
AI can enhance a live stream. It should never be allowed to hold the stream hostage.
If this failure model matches something you run, try the kill test and tell me where it breaks on your hardware. braidpipe is on GitHub under Apache-2.0. Issues, measurements from other machines, and focused pull requests are the most valuable things you can send me. And if you want to follow the engineering as it develops, a star tells me the problem is worth continuing to solve.
