Building a Distributed HLS Video Streaming System with Django, RabbitMQ, and FFmpeg

Vishwajeet

10 min read

·

Jun 19, 2026

--

Listen

Share

Have you ever wondered what actually happens after you hit upload on YouTube?

link to repo

From the outside world, it looks simple like you upload a file, wait, and your video is playable at 144p, 360p, 720p, and 1080p. But behind that waiting spinner is a surprisingly deep backend problem.

Press enter or click to view image in full size

The naive approach is straightforward: user uploads a video, your server processes it, done. But try doing that synchronously inside a web request and you’ll quickly run into walls. Transcoding a video is CPU-intensive and can take minutes. Your web server will hang, the request will time out.

In this article, I’ll walk you through how HLS streaming actually works, what transcoding is and why FFmpeg is the workhorse behind almost every video platform, and how I built a distributed pipeline using Django, RabbitMQ, and Celery to handle it all asynchronously.

HLS from First Principles

HLS — HTTP Live Streaming — was invented by Apple in 2009 and has since become the dominant standard for video delivery on the web. If you’ve watched a video on any major platform in the last five years, you’ve almost certainly used HLS without knowing it.

The core idea is elegant: “instead of serving a video as one giant file, you split it into small chunks — typically 6 to 10 seconds each — called segments. These segments are plain .ts (MPEG Transport Stream) files. The player downloads and plays them back-to-back, giving the illusion of a continuous stream.”

But how does the player know which segments exist and in what order? That’s the job of the .m3u8 playlist file. It's a plain text manifest that lists every segment, their durations, and their URLs. When a player opens a video, it first fetches the .m3u8, reads the segment list, then starts fetching and playing segments one by one.

The beauty of this design is that it runs entirely over HTTP. There’s no special streaming protocol, no persistent connection, no stateful server. Every segment request is just a plain HTTP GET. This means HLS works over any web server, any CDN, and through any firewall — which is a big reason it won the format wars.

This also enables one of HLS’s most powerful features: adaptive bitrate streaming. Instead of one playlist, you generate multiple — one for each quality level (say, 144p, 360p, and 720p). A master playlist then points to all of them. The player monitors the user’s available bandwidth in real time and switches between quality playlists on the fly. Slow connection? Drop to 144p. Strong WiFi? Jump to 720p. The user barely notices.

Compare this to serving a plain MP4: the server has to support byte-range requests, the player can’t easily switch quality mid-stream, and you get none of the CDN-friendliness of small discrete files. DASH (Dynamic Adaptive Streaming over HTTP) is a comparable alternative — technically very similar to HLS — but HLS has broader native support, especially on Apple devices and Safari, making it the safer default for most use cases.

Transcoding & FFmpeg

When a user uploads a video, you have no idea what you’re getting. It could be an .mkv shot on an Android phone, a .mov from an iPhone, or a .avi from 2003. Different codecs, different containers, different resolutions. You can't just serve that raw file — browsers won't know how to play half of them, and even if they could, serving a 2GB raw upload directly is a terrible idea.

This is what transcoding solves. Transcoding is the process of taking a video file and re-encoding it into a target format, resolution, and bitrate. In our case, that means taking whatever the user uploaded and producing three clean outputs: 144p, 360p, and 720p — each as a set of HLS segments ready to stream.

The tool that does this heavy lifting is FFmpeg. FFmpeg is a free, open-source command-line tool that can decode and encode virtually any video or audio format in existence. It’s not an exaggeration to say that FFmpeg powers a significant portion of the internet’s video infrastructure — YouTube, Facebook, and countless streaming platforms all rely on it internally. It’s the Swiss Army knife of media processing.

Here’s the FFmpeg command at the heart of this project:

ffmpeg -i input.mp4 \
  -vf scale=w=256:h=144 -b:v 200k -hls_time 6 -hls_playlist_type vod -f hls output_144p.m3u8 \
  -vf scale=w=640:h=360 -b:v 800k -hls_time 6 -hls_playlist_type vod -f hls output_360p.m3u8 \
  -vf scale=w=1280:h=720 -b:v 2800k -hls_time 6 -hls_playlist_type vod -f hls output_720p.m3u8

Let’s break down what each flag is doing:

• -i input.mp4 — the input file, whatever the user uploaded
• -vf scale=w=...:h=... — scales the video to the target resolution
• -b:v — sets the target video bitrate. Lower bitrate = smaller file, lower quality. 200k is fine for 144p; 2800k gives you a sharp 720p
• -hls_time 6 — each .ts segment will be approximately 6 seconds long
• -hls_playlist_type vod — marks this as a Video On Demand playlist (as opposed to a live stream), so the .m3u8 includes #EXT-X-ENDLIST at the end
• -f hls — tells FFmpeg to output in HLS format, producing both the .m3u8 playlist and the .ts segment files

The reason we produce three bitrates isn’t just about quality options — it’s about respecting the user’s network. A viewer on a 2G connection in a low-bandwidth area has a completely different reality than someone on fibre. Generating 144p, 360p, and 720p means the HLS player can pick the right one and switch dynamically as conditions change.

The output structure on disk ends up looking like this:

videos/
  <video_id>/
    144p.m3u8
    144p_segment000.ts
    144p_segment001.ts
    ...
    360p.m3u8
    360p_segment000.ts
    ...
    720p.m3u8
    720p_segment000.ts
    ...

Why Async? The Case for a Message Queue

Let’s say you decide to transcode the video directly inside your Django view. The user hits upload, Django receives the file, and you call FFmpeg inline. What happens?

The web server blocks. For the entire duration of the transcoding — which could be anywhere from 30 seconds to several minutes depending on the video length and server hardware. Eventually, the request times out.They try again. Now you’re running two transcoding jobs simultaneously on a server that can barely handle one.

This is the fundamental problem: a web server’s job is to respond fast. HTTP is a request-response protocol — the contract is that you receive a request and return a response quickly. Transcoding violates that contract completely. It’s a long-running, CPU-intensive job that has no business living inside a request cycle.

The solution is to decouple the upload from the processing. When the user uploads a video, Django does two things: saves the file, and drops a message into a queue saying “this video needs to be transcoded.” Then it immediately returns a response to the user. The actual transcoding happens separately, on a worker process that picks up the message from the queue and does the heavy lifting independently.

This is the core idea behind a message queue. A producer (Django) puts tasks into the queue. One or more consumers (Celery workers) pick tasks up and execute them. The producer and consumer never talk directly — the queue sits between them as a buffer.

RabbitMQ is the message broker here — it’s the system that holds the queue and manages message delivery. The reason I chose RabbitMQ over simpler alternatives is its support for multiple concurrent workers consuming from the same queue. You can spin up as many Celery workers as your infrastructure allows, and RabbitMQ will distribute tasks across them automatically. Upload traffic spike? Add more workers(using celery worker command). Each one independently pulls from the queue and processes videos in parallel.

Celery is the task framework that sits on top of RabbitMQ. It handles the worker lifecycle, task serialization, and retry logic. From Django’s perspective, dispatching a task is a single line: transcode_video.delay(video_id). That call returns immediately — Django doesn't wait, doesn't block, doesn't care when the task finishes. Fire and forget.

What this buys you is clean separation of concerns. The Django API layer stays fast and responsive. The transcoding layer scales independently. A backlog of uploads doesn’t affect API response times. And if a worker crashes mid-transcode, the task can be retried without the user ever knowing something went wrong.

5. The Architecture

Before diving into code, it helps to see the full picture. Here’s how every component fits together.

The stack:

• Django + DRF — the API layer, handles upload and streaming requests
• PostgreSQL — stores video metadata and the file paths to HLS output
• RabbitMQ — the message broker, holds the transcoding task queue
• Celery — the worker framework, consumes tasks from RabbitMQ and runs FFmpeg
• FFmpeg — does the actual transcoding, produces .m3u8 playlists and .ts segments
• Local storage — stores both raw uploads and HLS output (S3 is the planned replacement)

The upload flow:

Client
  → POST /upload (video file)
  → Django saves raw video to local storage
  → Django creates a Video record in PostgreSQL (status: pending)
  → Django calls transcode_video.delay(video_id)  ← fire and forget
  → Django returns 202 Accepted to client

RabbitMQ receives the task message
  → Celery worker picks it up
  → Worker fetches video path from PostgreSQL
  → Worker runs FFmpeg: raw video → 144p, 360p, 720p HLS output
  → Worker saves .m3u8 paths back to PostgreSQL (status: ready)

The streaming flow:

Client
  → GET /video/<id>/stream
  → Django fetches .m3u8 path from PostgreSQL
  → Returns the playlist file
  → Client HLS player reads the playlist
  → Player fetches .ts segments sequentially
  → Video plays

PostgreSQL is the connective tissue between the two flows. Django writes the initial record on upload, the Celery worker updates it once transcoding is done, and the streaming endpoint reads from it. The video ID is the only thing that travels through the queue — not the file itself.

Local storage is split into two areas: a uploads/ directory for the original upload, and a hls/ directory where FFmpeg writes its output. Each video gets its own subdirectory under hls/<video_id>/, keeping the segment files organized.

The entire infrastructure is wired together with Docker Compose. RabbitMQ runs as a container on port 5672, and Celery connects to it as a broker. The Django app and the Celery worker are separate processes — but they share the same codebase and the same database. The worker is started independently with celery -A vsite worker.

It’s worth being honest about what this architecture is missing for production. There’s no S3 — all files live on local disk, which means the setup doesn’t scale horizontally (two servers can’t share a local filesystem). There’s no CDN, so segment delivery isn’t optimized for geography or load. And there’s no authentication, so any client can access any video. These are the next milestones .

Implementation Walkthrough

Let’s walk through the actual code, from the moment a user uploads a video to the moment the Celery worker finishes transcoding it.

The Django Upload View

The view’s job is simple: receive the file, save it, create a database record, and fire off the task.

class VideoUploadView(APIView):
    def post(self, request):
        file = request.FILES['video']
        video = Video.objects.create(
            title=request.data.get('title'),
            original_file=file,
            status='pending'
        )
        transcode_video.delay(video.id)
        return Response({'id': video.id, 'status': 'processing'}, status=202)

Notice what we’re passing to .delay() — the video's ID, not the model instance itself. This is intentional and important. Celery serializes task arguments to send them through RabbitMQ. A Django model instance doesn't serialize cleanly, and more critically, by the time the worker picks up the task, the instance you passed could be stale. Passing the ID and re-fetching inside the worker is the correct pattern.

Connecting Celery to RabbitMQ

In settings.py, you point Celery at RabbitMQ as the broker:

CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672//'

RabbitMQ runs on port 5672 (as defined in docker-compose.yaml). When Django calls .delay(), Celery serializes the task and publishes it to RabbitMQ. The worker, running as a separate process, has an open connection to RabbitMQ and picks up the message as soon as it arrives.

The Celery Task

@shared_task
def transcode_video(video_id):
    video = Video.objects.get(id=video_id)
    input_path = video.original_file.path
    output_dir = f'media/hls/{video_id}/'
    os.makedirs(output_dir, exist_ok=True)

    resolutions = [
        ('144p', '256:144', '200k'),
        ('360p', '640:360', '800k'),
        ('720p', '1280:720', '2800k'),
    ]    for label, scale, bitrate in resolutions:
        output_path = f'{output_dir}{label}.m3u8'
        command = [
            'ffmpeg', '-i', input_path,
            '-vf', f'scale=w={scale.split(":")[0]}:h={scale.split(":")[1]}',
            '-b:v', bitrate,
            '-hls_time', '6',
            '-hls_playlist_type', 'vod',
            '-f', 'hls',
            output_path
        ]
        subprocess.run(command, check=True)    video.hls_path = f'media/hls/{video_id}/'
    video.status = 'ready'
    video.save()

The task fetches a fresh instance from the database using the ID, builds the output directory, then loops through the three resolutions and runs FFmpeg for each. After all three complete, it updates the model with the HLS output path and marks the video as ready.

The Null Instance Bug

Early on, I was passing the model instance directly to the task instead of the ID:

# Wrong
transcode_video.delay(video)

Inside the task, I was then calling methods on that instance and saving it. The result was null values being written to the database — the instance had been serialized and deserialized through RabbitMQ, losing its state in the process. Some fields came back as None, and saving that instance overwrote the valid data already in the database.

The fix was straightforward: pass only the video.id, and do Video.objects.get(id=video_id) as the very first line of the task. Always fetch a fresh instance inside the worker — never trust an object that crossed a process boundary.

The Windows Celery Gotcha

If you’re developing on Windows and you start a Celery worker the standard way, it will crash with a mysterious error related to process forking. Windows doesn’t support the fork multiprocessing model that Celery uses by default.

The fix is to force Celery to use the solo pool, which runs tasks in a single thread without forking:

celery -A vsite worker --pool=solo --loglevel=info

This isn’t suitable for production — solo means no concurrency, one task at a time. But for local development on Windows, it's the pragmatic solution. On Linux or macOS, the default pool works fine.

7. What’s Next

This project works, but it’s not production-ready yet. The biggest gaps: local disk needs to move to S3 (you can’t share a filesystem across multiple workers), S3 unlocks CDN integration so .ts segments are served from edge locations instead of the origin, and the streaming GET endpoint is still in progress. Beyond that — a master .m3u8 for true adaptive bitrate, signed URLs for access control, Redis caching to reduce DB hits, and observability so silent Celery failures don't go unnoticed.

Building this system made something click that reading about distributed systems never quite did — the reason you decouple things isn’t just about performance, it’s about failure isolation. When the transcoding worker crashes, the upload API keeps running. When the queue backs up, no user gets a timeout. Every boundary in the architecture is also a fault boundary. That mental model is worth more than any individual line of code in this repo.