Six Minutes or Eight Seconds: Packaging a Model So It Runs Anywhere (AI Engineering in 30 Days, Day 3)
Why a model file alone breaks on the server, how image layers decide your rebuild time, and how one image goes from 6.1 GB to 220 MB.
You change one line of your model server’s code and rebuild the container. It takes six minutes. A colleague moves one line in the Dockerfile, makes the same change, and the rebuild takes about eight seconds. Same code, same libraries, same model. The only difference is the order of the lines.
This is Day 3 of AI Engineering in 30 Days: packaging a model so it runs the same way on every machine. On Day 1 we saw that most models never reach real use because of everything around the model. On Day 2 we read a model’s name to work out how much GPU memory it needs. Today is the first step of moving a model off a laptop: putting it in a box that does not fall apart on the way.
First, Day 2’s practice: what gets rebuilt?
Day 2 ended with this question. A model’s container image is 4.6 GB, and 4.2 GB of it is Python libraries. You change one line of code and rebuild. What gets rebuilt, and why does the order of the Dockerfile decide it?
A container image is not one big file. It is a stack of layers, and each build step in the Dockerfile makes one layer. In the lesson’s example image, the base operating system is about 30 MB, system libraries 40 MB, Python itself 45 MB, the Python libraries (torch, numpy, transformers) 4.2 GB, the model weights 248 MB, and your serving code about 1 MB.
The rule that decides rebuild time is short: Docker reuses a layer from the last build only if that layer and every layer under it did not change. The moment one layer changes, that layer and everything above it are built again.
So if the Dockerfile copies your code before it installs the libraries, a one-line code change breaks the cache for the install step too, and all 4.2 GB go in again. The lesson’s example puts that at about six minutes. Install the libraries first, from a fully pinned file, then copy the model and the code last. Now a code change rebuilds only the thin top layer, in about eight seconds.
“Fully pinned” matters too. torch==2.3.1 always means the same version. torch>=2.3 can mean a different version next month, so two builds of the same file give you two different environments.
Why a model file on its own is not enough
The usual handoff in machine learning is one file: model.pkl or model.pt. The data scientist sends it, the platform engineer loads it on a server, and it fails with an error nobody understands, often about a numpy version.
The reason is that a trained model is a frozen snapshot made by an exact stack: this Python, this PyTorch, this numpy, this driver. A pickle file stores references to the exact classes and library versions that existed when it was saved. If anything is different on the server, the model may refuse to load, or worse, load and quietly give different answers.
A requirements.txt helps, but it only pins Python packages. It does not pin the operating system, the Python version or native libraries. The fix is to stop shipping a file and start shipping the whole environment: the operating system, Python, the pinned libraries, the weights and the serving code, sealed into one container image.
Get Roni Das’s stories in your inbox
Join Medium for free to get updates from this writer.
There is also a security reason. Loading a pickle runs code, by design. Loading a .pkl file you did not create can run anything on your server.
From 6.1 GB to 220 MB
Size is not cosmetic. A big image is slow to push, slow to pull onto a new server, and slow to scale up, and that hurts most when traffic jumps and new copies of the service need to start right now. The lesson shows four moves that shrink one image, and they stack:
- A slim base image drops build tools and documentation: 6.1 GB to 5.1 GB.
- CPU-only PyTorch drops the whole CUDA stack: down to 900 MB. Only do this if you serve the model without a GPU.
- A multi-stage build installs and compiles everything in one stage, then copies only the finished files into a clean second stage, so compilers and caches never ship: 750 MB.
- An ONNX export. ONNX is a file format that saves the model as a plain graph of operations. A small library, onnxruntime, runs it, so PyTorch does not ship at all: 220 MB.
None of these moves are meant to change what the model answers. They remove weight the serving path never needed. After an ONNX export, it is still good practice to check a sample of predictions against the original model.
Ship by digest, not by tag
Once the image is built, it goes to a registry, and servers pull it from there. You can name an image two ways. A tag, like fraud-model:latest, is a label that someone can point at different bytes tomorrow. A digest, like sha256:a1b9f2c7e4d0, is the hash of the image itself and can only ever mean one image.
Deploy by digest. The server hashes what it received and refuses to start anything that does not match, so the image that passed your tests is exactly the image in production. A bonus of layers: the big library layer is pulled once per server and shared by every image that uses it.
Two common mistakes
- Deploying by a tag like
:latest. Two servers can end up running two different images under the same name. - Loading a pickle you did not create. It can run any code on your server.
Practice for Day 4
In a notebook, a model answers in 8 milliseconds. The team wraps it in a quick Flask app, and at about 200 requests a second the answers take 4 seconds, while the GPU is only 12% busy. What is wrong? We work through it at the start of Day 4, which is about serving: turning a model file into an API.
Where this comes from
Every number in this post is from the free packaging lesson linked as the source. AI Engineering in 30 Days runs next to System Design Interview in 30 Days, which today covers read replicas and replication lag. Both courses, AI Engineering and System Design, are on systemdesign.academy.
