Why Your Kubernetes GPUs Sit at 5% Utilization (and How MIG/Time-Slicing Fixes It)

The most expensive thing in your cluster is doing nothing

Vasanthan K

20 min read

·

Oct 14, 2026

--

Listen

Share

You finally got GPUs into your Kubernetes cluster. The data science team is happy, the inference service is live, and the cloud bill shows a shiny new line item. Then someone opens a Grafana dashboard and asks a very awkward question: why is this GPU at 5% utilization?

This is not a rare edge case. It is the default outcome for most teams running GPUs on Kubernetes. A single data-center GPU can cost anywhere from a few hundred to several thousand dollars per month in the cloud, and in many clusters most of that capacity sits idle while new pods wait in Pending because "there are no GPUs available."

The GPUs are not broken. The workloads are not broken either. The problem is a mismatch between how Kubernetes hands out GPUs (one whole card per request) and how most real workloads use them (in short bursts, with a fraction of the memory).

In this article we will walk through the whole story, step by step, assuming you are new to GPUs on Kubernetes:

1. How Kubernetes actually sees and allocates a GPU.
2. Why that model leads to 5% utilization.
3. How to measure the problem properly, so you are not guessing.
4. Time-slicing: letting several pods take turns on one GPU.
5. MIG (Multi-Instance GPU): physically carving one GPU into isolated smaller GPUs.
6. How to choose between them, and what to watch out for in production.

Every code example comes with an explanation of what it does and why it works, so you can adapt it instead of copy-pasting blindly. All diagrams in this post were created specifically for it.

Part 1: How Kubernetes actually sees a GPU

Kubernetes was built around two resources it understands natively: CPU and memory. Both are divisible. You can ask for 250m of CPU (a quarter of a core) or 512Mi of memory, and the scheduler happily packs many small pods onto one node.

A GPU is different. Kubernetes has no idea what a GPU is. It only learns that GPUs exist through a component called a device plugin.

What a device plugin does

A device plugin is a small program (usually a DaemonSet, so one copy runs on every GPU node) that talks to the kubelet over a gRPC API. For NVIDIA hardware this is the NVIDIA k8s-device-plugin, which is also installed for you by the NVIDIA GPU Operator. It does three jobs:

1. Discover the GPUs on the node (using the NVIDIA driver).
2. Advertise them to the kubelet as an extended resource named nvidia.com/gpu, with a count. A node with 4 GPUs reports nvidia.com/gpu: 4.
3. Allocate a specific GPU when a pod is scheduled, by telling the container runtime which device (GPU UUID) to mount into the container.

You can see the advertised count on any GPU node:

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia\.com/gpu

What this command does: kubectl get nodes lists nodes, and -o custom-columns prints only the fields we care about. The path .status.allocatable.nvidia\.com/gpu reads the number of GPUs the node can hand out. The backslash escapes the dot inside nvidia.com, because otherwise kubectl would treat nvidia and com/gpu as two nested keys.

How a pod asks for a GPU

Here is the simplest possible GPU pod:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

Line by line:

• image: nvcr.io/nvidia/cuda:... is an official CUDA base image. It contains the CUDA user-space libraries but not the driver; the driver lives on the host and is injected by the NVIDIA Container Toolkit.
• command: ["nvidia-smi"] just prints the GPU the container can see, then exits. Perfect for a smoke test.
• resources.limits.nvidia.com/gpu: 1 is the important line. It tells the scheduler: "this pod needs one unit of the nvidia.com/gpu resource."

For extended resources like GPUs, you only set limits. Kubernetes automatically treats the request as equal to the limit, and you cannot overcommit them.

The catch: GPUs are integers

With CPU you could write cpu: 0.25. With GPUs you cannot write nvidia.com/gpu: 0.25. Extended resources must be whole numbers. So the smallest thing a pod can ask for is one entire GPU, and once it gets it, that GPU is exclusively bound to that pod until the pod dies. No other pod can be scheduled onto it, even if the first pod only ever uses 2% of its compute and 1 GB of its 80 GB of memory.

Press enter or click to view image in full size

The default flow. The pod requests nvidia.com/gpu: 1, the scheduler finds a node with a free GPU, and the device plugin hands that whole card to the pod. With four lightly used pods holding four GPUs, a fifth pod stays Pending even though the node is almost idle.

That single design decision, whole-GPU allocation, is the root cause of almost every “why is my GPU at 5%” conversation.

Part 2: Why your GPU sits at 5%

Modern data-center GPUs are enormous. An NVIDIA A100 has 108 streaming multiprocessors (SMs) and 40 or 80 GB of memory; an H100 has even more. They were designed for workloads like training a large model, where thousands of matrix multiplications run back to back for hours.

Most workloads in a typical company cluster look nothing like that. Here are the usual suspects.

1. Online inference services

An inference API (say, a small text classifier or an embedding model behind a REST endpoint) only uses the GPU while a request is being processed. A forward pass might take 15 ms. If the service gets 5 requests per second, the GPU is busy for about 75 ms out of every 1,000 ms. That is roughly 7.5% utilization at best, and real traffic is far spikier than that: busy at 10 a.m., silent at 3 a.m.

Press enter or click to view image in full size

An illustrative utilization trace of one inference pod that owns a whole GPU. Short bursts reach 60 to 90%, but the average over ten minutes is around 5%.

2. Jupyter notebooks and dev pods

A data scientist launches a notebook with a GPU attached, runs one cell that trains for 4 minutes, then spends 40 minutes reading results, writing code, and attending a meeting. The GPU is reserved the entire time. Multiply that by a team of 10, and you have 10 GPUs that are idle 90% of the day.

3. Small models on big cards

A 300 MB BERT-style model needs maybe 2 GB of GPU memory to serve. Placing it on an 80 GB H100 wastes 78 GB, and its small kernels cannot fill 132 SMs either. The model is not slow; the card is simply far bigger than the job.

4. CPU-bound pipelines

Many “GPU” jobs spend most of their time not on the GPU: decoding images, tokenizing text, reading from object storage, waiting on a database. The GPU waits for data, finishes its part in milliseconds, and waits again.

5. Batch jobs and CI tests

Nightly evaluation jobs and CI pipelines that “just check the model loads on a GPU” each need a GPU for a few minutes. Because allocation is exclusive, they queue behind each other or force you to buy more cards.

The real cost

Put all of this together and you get the classic pattern: allocation is 100% (every GPU is claimed by some pod, new pods are Pending), but utilization is 5 to 20%. The fix is not more GPUs. The fix is letting several workloads share one GPU safely. That is exactly what time-slicing and MIG do, in two very different ways.

Part 3: Measure before you fix

Before changing anything, prove the problem with numbers. You need two views: a quick spot check and a long-running metric you can graph.

Quick check with nvidia-smi

You can run nvidia-smi inside any pod that has a GPU:

kubectl exec -it <your-gpu-pod> -- nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv -l 2

What each part means:

• kubectl exec -it <pod> -- runs a command inside the running container.
• --query-gpu=... asks for specific fields instead of the big human-readable table.
• utilization.gpu is the percentage of time in the last sample period during which at least one kernel was running on the GPU.
• memory.used / memory.total shows how much GPU memory is actually allocated.
• --format=csv -l 2 prints CSV and repeats every 2 seconds, so you can watch it for a while.

An important subtlety for beginners: utilization.gpu measures time, not space. If one tiny kernel uses only 1 of 108 SMs but runs continuously, nvidia-smi will still report close to 100%. So a low number is a reliable sign of waste, but a high number does not prove the GPU is fully used. We will use better metrics below.

Long-term monitoring with DCGM Exporter

For real decisions you want weeks of history in Prometheus. NVIDIA’s DCGM Exporter runs as a DaemonSet, reads metrics from the Data Center GPU Manager (DCGM), and exposes them on a /metrics endpoint that Prometheus scrapes. If you install the GPU Operator, it is enabled by default.

The metrics that matter most:

Press enter or click to view image in full size

PromQL queries you can paste into Grafana

Average utilization per GPU over the last 7 days:

avg by (Hostname, gpu) (
  avg_over_time(DCGM_FI_DEV_GPU_UTIL[7d])
)

How it works: avg_over_time(...[7d]) takes every sample of the metric in the last 7 days and averages it, per time series. Each series is one GPU, so the result is "average utilization per GPU." The outer avg by (Hostname, gpu) collapses any extra labels (like pod name) so you get exactly one number per physical GPU.

Find GPUs that are allocated to a pod but nearly idle:

avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[24h]) < 10

How it works: DCGM Exporter attaches namespace, pod and container labels when the GPU is in use by a pod. pod!="" keeps only GPUs that are assigned. The < 10 filter returns only those that averaged below 10% over the last day. This list is your sharing candidates.

Memory headroom per GPU, as a percentage:

100 * DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

Why this matters: memory decides how many workloads can share a GPU. If a pod uses 4 GB on a 40 GB card, you can fit several neighbors. If it uses 35 GB, sharing will not help no matter how idle the compute is.

A good rule of thumb: if a GPU averages under 30% utilization and under 50% memory, it is a strong candidate for sharing.

Part 4: Fix #1 — Time-slicing

The idea in plain English

Time-slicing is the GPU equivalent of how your laptop’s CPU runs a browser, an editor and Spotify at the same time. There is one physical processor, and the system rapidly switches between programs, giving each a turn.

With time-slicing enabled, the NVIDIA device plugin lies to Kubernetes on purpose. Instead of advertising nvidia.com/gpu: 1 for one physical GPU, it advertises, say, nvidia.com/gpu: 4. Kubernetes now believes the node has 4 GPUs and schedules 4 pods there. All 4 pods get the same physical GPU, and the GPU's hardware scheduler interleaves their work in time slices.

Press enter or click to view image in full size

Compute is shared by taking turns. Memory is not divided at all: all pods allocate from the same pool, and nothing stops one pod from taking more than its share.

Why it fixes the 5% problem

Remember the inference pod from Figure 2: busy for a few milliseconds, then idle. When four such pods share one card, the idle gaps of one pod get filled by the bursts of another. If each used about 5% on its own, together they use roughly 20%, on one GPU instead of four. You have just freed three GPUs.

Step 1: Install the GPU Operator

The easiest way to manage everything (driver, container toolkit, device plugin, DCGM exporter) is the NVIDIA GPU Operator, installed with Helm:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --wait

What happens here: helm repo add registers NVIDIA's chart repository. helm install deploys the operator into a new gpu-operator namespace. The operator then watches for GPU nodes and installs the full NVIDIA software stack on each of them automatically. --wait makes Helm block until all resources are ready, so you know when it is safe to continue. (On managed services like EKS, GKE or AKS, the driver may already be installed; check your provider's docs and set --set driver.enabled=false if so.)

Step 2: Describe the sharing policy in a ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Explaining every field:

• data.any is the name of a configuration profile. A single ConfigMap can hold several profiles (for example t4-x4 and a100-x2), and you pick which one each node uses. Here we create one profile called any.
• version: v1 is the schema version of the device plugin config file.
• flags.migStrategy: none tells the plugin that MIG is not in use on these GPUs.
• sharing.timeSlicing.resources is the list of resources to oversubscribe. We take nvidia.com/gpu and set replicas: 4, meaning each physical GPU is advertised as 4 schedulable units. A node with 2 GPUs will now report nvidia.com/gpu: 8.
• renameByDefault: false keeps the resource name as nvidia.com/gpu, so existing pod specs work unchanged. If you set it to true, the shared resource is advertised as nvidia.com/gpu.shared instead. That is useful when you want pods to explicitly opt in to sharing, so nobody lands on a shared GPU by accident.
• failRequestsGreaterThanOne: true rejects pods that ask for more than one replica. This prevents a common misunderstanding: requesting nvidia.com/gpu: 2 on a time-sliced GPU does not give you twice the compute, it just gives you two turns on the same card. Failing loudly is better than silently giving someone less than they think.

Apply it:

kubectl apply -f time-slicing-config.yaml

Step 3: Point the device plugin at the config

The GPU Operator is configured through a custom resource called ClusterPolicy. We patch it so the device plugin loads our ConfigMap and uses the any profile by default:

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  -n gpu-operator --type merge \
  -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config", "default": "any"}}}}'

What this does: --type merge merges the JSON into the existing object instead of replacing it. config.name is the ConfigMap to read, and config.default is the profile applied to every node that does not say otherwise. The operator restarts the device plugin pods, which re-register with the kubelet using the new replica count.

If you want different behavior on different nodes (for example, share the T4 dev nodes but not the H100 training nodes), label individual nodes with the profile name instead of relying on the default:

kubectl label node gpu-dev-node-1 nvidia.com/device-plugin.config=any --overwrite

Step 4: Verify

kubectl get node gpu-dev-node-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
# Output: 4   (for a node with 1 physical GPU)

kubectl get node gpu-dev-node-1 --show-labels | tr ',' '\n' | grep nvidia.com/gpu
# nvidia.com/gpu.product=Tesla-T4-SHARED
# nvidia.com/gpu.replicas=4

The node now reports 4 allocatable GPUs. The operator also updates node labels: gpu.replicas=4, and the product name gets a -SHARED suffix so you can target (or avoid) shared nodes with a nodeSelector.

Step 5: Run four pods on one GPU

apiVersion: apps/v1
kind: Deployment
metadata:
  name: embedder
spec:
  replicas: 4
  selector:
    matchLabels:
      app: embedder
  template:
    metadata:
      labels:
        app: embedder
    spec:
      nodeSelector:
        nvidia.com/gpu.replicas: "4"
      containers:
        - name: model
          image: my-registry/embedder:1.0
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: PYTORCH_CUDA_ALLOC_CONF
              value: "expandable_segments:True"

Why it is written this way:

• replicas: 4 with nvidia.com/gpu: 1 each consumes exactly the 4 advertised slices. On a single-GPU node, all four pods land on the same physical card.
• nodeSelector on nvidia.com/gpu.replicas: "4" makes sure this Deployment only goes to nodes where sharing is configured, never onto an exclusive training GPU.
• The PYTORCH_CUDA_ALLOC_CONF setting reduces memory fragmentation in PyTorch's caching allocator. On a shared GPU, wasted memory hurts every neighbor, so small memory hygiene settings matter more than usual.

If you kubectl exec into each pod and run nvidia-smi, all four will show the same GPU UUID. That is time-slicing working.

Time-slicing caveats (read these before production)

1. No memory isolation. All pods share one memory pool. If one pod loads a bigger model or leaks memory, the others hit CUDA out of memory. Each application should cap its own usage, for example torch.cuda.set_per_process_memory_fraction(0.25) in PyTorch, or the memory-fraction options most inference servers provide.
2. No fault isolation. A serious GPU error (an Xid fault) caused by one pod can affect every process on that GPU.
3. No performance guarantee. If all four pods get traffic at the same moment, each gets roughly a quarter of the GPU, plus some context-switching overhead. Latency becomes unpredictable under load.
4. The replica count is not a quota. Kubernetes thinks each pod has “one GPU.” Nothing enforces that a pod uses only a quarter. It is a scheduling hint, not a resource limit.

Time-slicing is perfect for dev notebooks, CI jobs, demos, and low-traffic internal inference, where everyone on the GPU belongs to the same team. When you need real walls between workloads, you need MIG.

Part 5: Fix #2 — MIG (Multi-Instance GPU)

The idea in plain English

If time-slicing is roommates sharing one kitchen in shifts, MIG is converting a house into separate apartments, each with its own kitchen, walls and front door.

MIG is a hardware feature introduced with NVIDIA’s Ampere generation. It physically partitions one GPU into up to 7 independent GPU instances. Each instance gets:

• its own dedicated streaming multiprocessors (compute),
• its own dedicated slice of GPU memory,
• its own share of L2 cache and memory bandwidth,
• its own fault domain: an error in one instance does not crash the others.

To the software running inside, a MIG instance looks and behaves like a small, standalone GPU. To Kubernetes, each instance is a separate schedulable resource.

Which GPUs support MIG?

MIG is available on data-center GPUs from Ampere onward: A100 and A30 (A30 supports up to 4 instances), H100, H200, and the Blackwell data-center line. Popular inference cards like the T4, L4 and A10 do not support MIG. For those, time-slicing (or MPS, covered later) is your option. Always confirm on NVIDIA’s MIG user guide for your exact model.

Understanding MIG profiles

You cannot cut a GPU into arbitrary sizes. NVIDIA defines fixed profiles. On an A100 40GB they look like this:

The naming is easy once you see the pattern: <compute slices>g.<memory>gb. On an H100 80GB the same pattern applies with doubled memory, for example 1g.10gb, 3g.40gb and 7g.80gb.

Press enter or click to view image in full size

Three ways to carve the same A100. Layout A gives seven small, equal instances (great for many small inference models). Layout B mixes sizes for teams with different needs. Layout C gives two medium instances, and one slice is left unused because the hardware placement rules do not allow a third 3g instance.

Step 1: Choose a MIG strategy in the GPU Operator

The device plugin can expose MIG instances to Kubernetes in two ways:

• single: every GPU on the node is split into the same profile, and instances are advertised with the familiar name nvidia.com/gpu. Existing pod specs keep working, but pods cannot choose a size.
• mixed: each profile gets its own resource name, like nvidia.com/mig-1g.5gb and nvidia.com/mig-3g.20gb. Pods request exactly the size they need. This is what most teams want.

helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set mig.strategy=mixed \
  --wait

Step 2: Tell the node how to partition itself

The GPU Operator includes a MIG Manager that watches a node label and reconfigures the GPUs to match. You never have to run low-level commands by hand:

kubectl label node gpu-a100-node-1 nvidia.com/mig.config=all-1g.5gb --overwrite

What happens behind the scenes:

1. MIG Manager sees the label change.
2. It evicts GPU pods from the node (the GPU must be idle to repartition).
3. It enables MIG mode on each GPU and creates seven 1g.5gb instances.
4. It restarts the device plugin, which now advertises nvidia.com/mig-1g.5gb: 7 per GPU.
5. It sets the label nvidia.com/mig.config.state=success when done.

Because step 2 evicts workloads, treat a MIG reconfiguration like a node maintenance event: cordon and drain first in production.

Other built-in layouts include all-2g.10gb, all-3g.20gb, and all-balanced (a mix of sizes). You can also define your own layouts in a custom ConfigMap for MIG Manager, for example two 1g.5gb plus one 2g.10gb plus one 3g.20gb.

Watch the reconfiguration complete and check the new resources:

kubectl get node gpu-a100-node-1 -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}'
# success

kubectl get node gpu-a100-node-1 -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com")))'
# {
#   "nvidia.com/mig-1g.5gb": "7"
# }

What the second command does: it fetches the node as JSON, and jq filters the allocatable map down to only keys starting with nvidia.com. Instead of nvidia.com/gpu: 1, you now see seven schedulable MIG instances.

What MIG Manager does under the hood (for the curious)

If you SSH into a GPU node, you can do the same thing manually with nvidia-smi. Understanding these commands helps when you are debugging:

# 1. Enable MIG mode on GPU 0 (requires the GPU to be idle; may need a reset)
sudo nvidia-smi -i 0 -mig 1

# 2. List the profiles this GPU supports, with their numeric IDs
sudo nvidia-smi mig -lgip

# 3. Create three 1g.5gb GPU instances (profile ID 19 on A100 40GB)
#    and a default compute instance inside each (-C)
sudo nvidia-smi mig -cgi 19,19,19 -C

# 4. List the GPU instances you created
sudo nvidia-smi mig -lgi

Key concepts in these commands:

• A GPU instance (GI) is the hardware partition: memory slice, cache and SM group.
• A compute instance (CI) lives inside a GI and is what processes actually run on. The -C flag creates one default CI per GI, which is what you want 99% of the time.
• Profile IDs (like 19) differ between GPU models, which is why -lgip exists. Always list first, then create.

In Kubernetes, prefer the label-based MIG Manager over manual commands. Manual changes are not tracked declaratively and can be overwritten.

Step 3: Request a MIG slice from a pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-scorer
spec:
  replicas: 7
  selector:
    matchLabels:
      app: fraud-scorer
  template:
    metadata:
      labels:
        app: fraud-scorer
    spec:
      containers:
        - name: model
          image: my-registry/fraud-scorer:2.3
          resources:
            limits:
              nvidia.com/mig-1g.5gb: 1

How to read this: the only difference from a normal GPU pod is the resource name. nvidia.com/mig-1g.5gb: 1 means "give me one 1g.5gb instance." With replicas: 7, one A100 now serves seven fully isolated copies of the model, each with a guaranteed 5 GB of memory and 1/7 of the compute. Inside each container, nvidia-smi -L shows a device with a MIG- prefixed UUID.

Why MIG isolation matters

Press enter or click to view image in full size

With time-slicing, one misbehaving pod can exhaust memory for everyone. With MIG, each pod is boxed into its own memory and compute, so a failure stays contained.

MIG caveats

1. Fixed sizes. You only get the predefined profiles. If your model needs 7 GB, a 1g.5gb slice is too small and you jump to 2g.10gb, wasting 3 GB.
2. Repartitioning is disruptive. Changing layouts evicts pods from the GPU. Plan layouts per node pool rather than changing them often.
3. Compute does not “burst.” A 1g instance can never borrow idle SMs from a neighbor. For very spiky workloads, that can be less efficient than time-slicing.
4. Hardware cost. Only higher-end data-center GPUs support it.

Combining MIG with time-slicing

You can stack the two techniques: partition an A100 with MIG, then time-slice each MIG instance. For example, 7 1g.5gb instances with replicas: 2 gives you 14 schedulable units. Teams use this for dev environments: hard isolation between teams (MIG), soft sharing within a team (time-slicing). In the time-slicing ConfigMap, you list the MIG resource name (such as nvidia.com/mig-1g.5gb) instead of nvidia.com/gpu, and set migStrategy: mixed.

Part 6: Time-slicing vs MIG — how to choose

Neither technique is “better.” They solve the same problem with different trade-offs.

Press enter or click to view image in full size

Press enter or click to view image in full size

A practical decision path. Start with “is the GPU actually busy?” because the cheapest optimization is the one you do not need.

A third option worth knowing: MPS

NVIDIA’s Multi-Process Service (MPS) sits between the two. Like time-slicing, it works in software and on most GPUs. Unlike time-slicing, it lets kernels from different processes run at the same time (concurrently) rather than taking turns, and it can cap each client’s memory and share of SMs. Recent versions of the NVIDIA device plugin support MPS as a sharing mode, configured almost exactly like the time-slicing ConfigMap (sharing.mps instead of sharing.timeSlicing). It still lacks full fault isolation, so treat it as "better time-slicing for trusted workloads" rather than a MIG replacement.

And on the horizon: Dynamic Resource Allocation (DRA)

The integer nvidia.com/gpu model is a decade-old Kubernetes limitation. The newer Dynamic Resource Allocation (DRA) API lets drivers describe devices richly (memory size, MIG profile, sharing mode) and lets pods ask for them with structured claims instead of a single number. NVIDIA maintains a DRA driver for GPUs. If you are designing a new platform, it is worth tracking, but time-slicing and MIG through the device plugin remain the most widely deployed approach today.

Part 7: Production checklist and common pitfalls

A rollout plan that works

1. Measure for at least a week with DCGM Exporter. Identify GPUs under 30% utilization and under 50% memory.
2. Split node pools by purpose. For example: gpu-train (exclusive, no sharing), gpu-infer (MIG), gpu-dev (time-slicing). Use node labels and taints so workloads land in the right pool.
3. Start with time-slicing on dev nodes. It is low-risk, reversible by editing a ConfigMap, and gives the fastest visible win.
4. Move production inference to MIG where your hardware supports it and you need predictable latency.
5. Re-measure. Compare utilization, pod Pending time, and GPU count before and after.

Taint shared nodes so only opted-in pods land there

kubectl taint nodes gpu-dev-node-1 gpu-sharing=time-sliced:NoSchedule

Pods that are happy to share add a matching toleration:

tolerations:
  - key: "gpu-sharing"
    operator: "Equal"
    value: "time-sliced"
    effect: "NoSchedule"

Why this matters: a taint is a “keep out” sign on a node. Only pods with a matching toleration can be scheduled there. This prevents a latency-sensitive production service from accidentally being placed on a crowded, time-sliced GPU.

Cap GPU memory inside your application (time-slicing)

Since time-slicing does not enforce memory limits, each workload should limit itself. In PyTorch:

import os
import torch

# Fraction of the GPU this process may use, e.g. 0.25 for 4 replicas.
fraction = float(os.getenv("GPU_MEMORY_FRACTION", "0.25"))
torch.cuda.set_per_process_memory_fraction(fraction, device=0)
model = torch.load("model.pt", map_location="cuda:0")

What this does: set_per_process_memory_fraction tells PyTorch's memory allocator never to grab more than that fraction of total GPU memory. If the code tries, it raises an out-of-memory error in this pod only, instead of silently starving its neighbors. Reading the value from an environment variable means you can tune it per Deployment without rebuilding the image. TensorFlow, vLLM and Triton Inference Server have similar settings.

Common pitfalls

• Expecting nvidia.com/gpu: 2 to mean "more GPU" on a time-sliced node. It does not. Use failRequestsGreaterThanOne: true to catch this early.
• Changing MIG layouts on a busy node. MIG Manager evicts GPU pods. Drain first.
• Mixing training and inference on shared GPUs. One training job can saturate a GPU and wreck inference latency for everyone else. Keep training on exclusive GPUs.
• Trusting GPU-Util alone. It measures time, not how much of the chip is used. Look at DCGM_FI_PROF_SM_ACTIVE and memory too.
• Forgetting about quotas. With 4x time-slicing, a namespace quota of requests.nvidia.com/gpu: 4 now means one physical GPU, not four. Revisit your ResourceQuotas after enabling sharing.

Conclusion

Your GPUs sit at 5% utilization not because something is broken, but because Kubernetes, by default, gives every pod an entire GPU, and most workloads need only a sliver of one, for a sliver of the time.

• Time-slicing fixes this in minutes on almost any NVIDIA GPU by letting pods take turns. It is ideal for trusted, bursty workloads, but offers no memory or fault isolation.
• MIG fixes it with real hardware partitions on A100, H100 and newer, giving each pod guaranteed memory and compute. It is ideal for multi-tenant production inference, at the cost of fixed sizes and disruptive reconfiguration.
• You can combine both, and you should measure before and after with DCGM Exporter.

Start small: pick your idlest dev GPU node, enable time-slicing with 4 replicas, and watch four pods share a card that used to serve one. The results in your Grafana dashboard, and in your cloud bill, tend to make the case for the rest of the rollout on their own.

Sources

• NVIDIA GPU Operator documentation
• Time-Slicing GPUs in Kubernetes (NVIDIA GPU Operator docs)
• NVIDIA MIG User Guide
• NVIDIA k8s-device-plugin
• NVIDIA DCGM Exporter