Autoscaling vLLM on GKE TPUs: A Hands-On Guide

Dineshkumar Anandan

8 min read

·

Jun 9, 2026

--

Listen

Share

Deploy a large language model on Google Kubernetes Engine with TPU v6e (Trillium), then watch the cluster scale TPU hardware up and down on demand — with every command you need to follow along.

Why TPU autoscaling matters

Serving LLMs is expensive, and TPUs are some of the most expensive hardware you’ll provision. The problem is that inference traffic is rarely flat. You get busy hours and quiet hours, spikes and lulls. If you keep enough TPU capacity online to handle your busiest moment, you’re paying for idle accelerators most of the day.

Autoscaling fixes this. Instead of running a fixed fleet, you let Kubernetes provision TPU nodes when demand rises and release them when demand falls. You pay for what you use, and your service still holds up under load.

This guide walks through the whole thing end to end on GKE: create a TPU cluster, deploy vLLM, validate that autoscaling actually provisions new TPU hardware, benchmark it, and tear it all down. Every command is here so you can follow along.

We’ll serve Qwen/Qwen2.5-3B-Instruct on TPU v6e, but the pattern applies to any model vLLM supports.

How TPU autoscaling works

Before the commands, it helps to understand the loop you’re about to build. The Cluster Autoscaler does the heavy lifting:

1. Incoming requests increase.
2. A new pod is created but can’t be scheduled — there’s no TPU node free, so it sits in Pending.
3. The Cluster Autoscaler notices the unschedulable pod and the unmet TPU resource request.
4. It provisions a new TPU VM and adds it to the node pool.
5. The pending pod gets scheduled onto the new node and becomes Running.
6. Traffic spreads across the now-larger set of replicas.

When traffic drops, the same machinery runs in reverse: idle TPU nodes are drained and removed automatically. The key insight is that a Pending pod is the trigger — the autoscaler reacts to demand it can’t currently satisfy.

The architecture itself is straightforward:

• A GKE Standard cluster for the control plane and a small default node pool.
• A TPU v6e node pool with autoscaling enabled.
• A vLLM Deployment plus a LoadBalancer Service to expose it.
• The Cluster Autoscaler wiring it all together.

Prerequisites

Install the command-line tools you’ll need:

sudo apt update
sudo apt install -y kubectl google-cloud-cli google-cloud-sdk-gke-gcloud-auth-plugin jq git python3-pip

Install Helm:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Authenticate with Google Cloud:

gcloud auth login

Set your active project and quota project:

gcloud config set project YOUR_PROJECT_ID
gcloud config set billing/quota_project YOUR_PROJECT_ID

Step 1 — Set your environment variables

Keeping config in one place makes the rest of the commands clean and reusable. Create a file named env.sh:

#!/bin/bash
export PROJECT_ID="YOUR_PROJECT_ID"
export PROJECT_NUMBER="YOUR_PROJECT_NUMBER"
export REGION="us-east5"
export ZONE="us-east5-b"
export CLUSTER_NAME="tpu-vllm-cluster"
export NODEPOOL_NAME="tpunodepool"
export NAMESPACE="vllm"
export MACHINE_TYPE="ct6e-standard-1t"
export MODEL_NAME="Qwen/Qwen2.5-3B-Instruct"

Load it into your current shell:

source env.sh

Step 2 — Enable the required APIs

gcloud services enable \
  container.googleapis.com \
  compute.googleapis.com \
  monitoring.googleapis.com \
  logging.googleapis.com \
  iam.googleapis.com \
  storage.googleapis.com

Step 3 — Create the GKE cluster

This creates a Standard cluster with Workload Identity, the GCS Fuse CSI driver, and managed Prometheus enabled. The default node pool is small and CPU-only — the TPUs come in the next step.

gcloud container clusters create ${CLUSTER_NAME} \
  --location=${REGION} \
  --release-channel=rapid \
  --workload-pool=${PROJECT_ID}.svc.id.goog \
  --addons=GcsFuseCsiDriver \
  --enable-managed-prometheus \
  --machine-type=e2-standard-4 \
  --num-nodes=1

Step 4 — Create the TPU node pool (with autoscaling)

This is the heart of the setup. The --enable-autoscaling flag, together with the min and max node bounds, is what lets the cluster grow and shrink TPU capacity automatically.

gcloud container node-pools create ${NODEPOOL_NAME} \
  --cluster=${CLUSTER_NAME} \
  --location=${REGION} \
  --node-locations=${ZONE} \
  --machine-type=${MACHINE_TYPE} \
  --num-nodes=1 \
  --enable-autoscaling \
  --total-min-nodes=1 \
  --total-max-nodes=2

The two numbers worth understanding here:

• --total-min-nodes is your floor — the autoscaler never goes below this, even when idle. Set it to whatever baseline capacity you always want available.
• --total-max-nodes is your ceiling — the most TPU nodes the autoscaler will ever provision. This is your cost guardrail. Raise it as your real serving needs grow.

Step 5 — Connect kubectl

Fetch credentials:

gcloud container clusters get-credentials ${CLUSTER_NAME} --location=${REGION}

Confirm your nodes are up:

kubectl get nodes -o wide

Step 6 — Create the namespace and secrets

You’ll need a namespace, a service account, and your Hugging Face token (vLLM uses it to pull the model).

kubectl create namespace vllm
kubectl create serviceaccount vllm-sa -n vllm
kubectl create secret generic hf-secret --from-literal=hf_api_token=<HF_TOKEN> -n vllm

Step 7 — Write the vLLM deployment manifest

Save the following as qwen-vllm.yaml. It defines both the Deployment and the LoadBalancer Service.

The two lines that make this a TPU workload are the nodeSelector labels and the google.com/tpu: 1 resource limit. That resource request is what makes a pod unschedulable until a TPU node exists — which is exactly what triggers the autoscaler.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen-vllm
  namespace: vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: qwen-vllm
  template:
    metadata:
      labels:
        app: qwen-vllm
    spec:
      serviceAccountName: vllm-sa
      nodeSelector:
        cloud.google.com/gke-tpu-accelerator: tpu-v6e-slice
        cloud.google.com/gke-tpu-topology: 1x1
      containers:
        - name: vllm
          image: vllm/vllm-tpu:latest
          command:
            - python3
            - -m
            - vllm.entrypoints.openai.api_server
          args:
            - --host=0.0.0.0
            - --port=8000
            - --model=Qwen/Qwen2.5-3B-Instruct
            - --tensor-parallel-size=1
            - --max-model-len=2048
            - --max-num-seqs=16
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: hf_api_token
            - name: VLLM_USE_V1
              value: "1"
          ports:
            - containerPort: 8000
          resources:
            limits:
              google.com/tpu: 1
          readinessProbe:
            tcpSocket:
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 15
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
---
apiVersion: v1
kind: Service
metadata:
  name: qwen-vllm-service
  namespace: vllm
spec:
  type: LoadBalancer
  selector:
    app: qwen-vllm
  ports:
    - port: 8000
      targetPort: 8000

Step 8 — Deploy and watch it come up

Apply the manifest:

kubectl apply -f qwen-vllm.yaml

Watch the pod move through its lifecycle:

kubectl get pods -n vllm -w

You should see it pass through Pending → ContainerCreating → Running.

Step 9 — Verify the logs

Find the pod and stream its logs:

kubectl get pods -n vllm
kubectl logs -f pod/<POD_NAME> -n vllm

Look for these milestones, which confirm vLLM is starting correctly on TPU:

• Downloading model
• Initializing TPU runtime
• Compiling XLA graphs
• Uvicorn running on http://0.0.0.0:8000

That XLA compilation step is worth noting — it can take a few minutes, and it happens every time a fresh pod starts. Keep it in mind; it matters for how quickly a freshly-scaled node can actually serve traffic.

Step 10 — Get the service IP

List the services:

kubectl get svc -n vllm

Export the external LoadBalancer IP into a variable:

export VLLM_IP=$(kubectl get svc qwen-vllm-service -n vllm -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Step 11 — Test inference

Check the health endpoint first:

curl http://${VLLM_IP}:8000/health

Then send a real chat-completion request:

curl http://${VLLM_IP}:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-3B-Instruct",
    "messages": [
      { "role": "user", "content": "Explain TPU autoscaling" }
    ],
    "max_tokens": 100
  }'

If you get a coherent reply, your model is live.

Step 12 — Validate autoscaling

This is the moment of truth. Scale the deployment up so it needs more TPU capacity than currently exists:

kubectl scale deployment qwen-vllm --replicas=2 -n vllm

Now watch the pods and the nodes in two separate terminals:

kubectl get pods -n vllm -w

kubectl get nodes -w

What you should see unfold:

• The second pod enters Pending — there’s no free TPU node for it.
• The Cluster Autoscaler detects the unschedulable pod.
• A new TPU VM is provisioned and joins the cluster.
• The second pod schedules onto it and becomes Running.

You just watched Kubernetes provision real TPU hardware in response to demand. That’s the whole point.

Step 13 — Benchmark it

A working endpoint isn’t enough; you want numbers. Install the OpenAI SDK:

pip install openai

Then run a simple concurrent benchmark to measure throughput and latency:

from openai import OpenAI
import concurrent.futures
import time

client = OpenAI(
    base_url="http://YOUR_IP:8000/v1",
    api_key="dummy"
)def run():
    start = time.time()
    r = client.chat.completions.create(
        model="Qwen/Qwen2.5-3B-Instruct",
        messages=[
            { "role": "user", "content": "Explain TPU inference" }
        ],
        max_tokens=128
    )
    print(time.time() - start)with concurrent.futures.ThreadPoolExecutor(max_workers=20) as ex:
    for _ in range(100):
        ex.submit(run)

This fires 100 requests with 20 in flight at a time and prints each request’s latency. Watch how the numbers behave as the second replica comes online — that’s your autoscaling paying off under real concurrency.

Step 14 — Validate scale-down

Scaling up is only half the value. Scale back down:

kubectl scale deployment qwen-vllm --replicas=1 -n vllm

Watch the nodes:

kubectl get nodes -w

After a short cooldown, the now-idle TPU node is drained and removed automatically. The Cluster Autoscaler shrinks the infrastructure back toward your minimum — and your bill shrinks with it. This automatic scale-down is what turns “expensive TPU fleet” into “pay for what you actually use.”

Troubleshooting

A few issues you’re likely to hit, and how to fix them:

TPU topology mismatch. The pod won’t schedule because its topology label doesn’t match the node pool. Make sure your nodeSelector uses the correct topology:

cloud.google.com/gke-tpu-topology: 1x1

Missing service account. If the pod complains about a missing service account, create it in the namespace:

kubectl create serviceaccount vllm-sa -n vllm

GCS Fuse mount errors. If you hit mount problems early on, remove GCS Fuse from the deployment until the core serving path is validated, then add it back.

TPU scheduling failures. Confirm the TPU node pool exists and that its labels actually match your deployment’s nodeSelector. A typo here keeps pods Pending forever.

RBAC / permission errors. If you’re wiring up an HPA or hitting permission walls, you likely need container.admin or cluster-admin IAM permissions.

Cleanup

TPUs cost money even when idle, so tear everything down when you’re done.

Scale the deployment to zero:

kubectl scale deployment qwen-vllm --replicas=0 -n vllm

Delete the deployment and service:

kubectl delete deployment qwen-vllm -n vllm
kubectl delete svc qwen-vllm-service -n vllm

Delete the namespace:

kubectl delete namespace vllm

Delete the TPU node pool:

gcloud container node-pools delete tpunodepool --cluster=tpu-vllm-cluster --region=us-east5 --quiet

Delete the cluster:

gcloud container clusters delete tpu-vllm-cluster --region=us-east5 --quiet

How to know it worked

You’ve successfully built TPU autoscaling when all of these are true:

• The TPU pod reaches Running.
• vLLM responds to inference requests.
• The LoadBalancer is reachable.
• The autoscaler provisions a new TPU node when you scale up.
• Multiple pods run across the larger fleet.
• Scale-down removes the extra TPU node automatically.
• You’ve collected throughput and latency numbers.

A few production notes

This guide gets you a working, autoscaling TPU service. Before you put it in front of real users, a handful of hardening steps are worth taking:

• Pin your image instead of vllm/vllm-tpu:latest, so deploys are reproducible.
• Secure the endpoint — the LoadBalancer here is plain HTTP. Put it behind a Gateway or Ingress with TLS and authentication.
• Cache the model in GCS or bake it into your image so new pods don’t re-download weights on every cold start.
• Use an HTTP readiness probe against /health rather than a bare TCP check, so traffic only routes to pods that are genuinely ready.
• Remember XLA compile time when tuning your autoscaler. A new TPU node isn’t instantly useful — provisioning plus model load plus graph compilation takes minutes. If your traffic pattern is predictable, scaling up a little ahead of demand beats reacting after it arrives.

Wrapping up

TPU autoscaling turns a fixed, expensive accelerator fleet into something that breathes with your traffic. The mechanism is simple once you see it: a pod that can’t be scheduled is a signal, and the Cluster Autoscaler answers that signal by provisioning real hardware — then takes it away again when the demand is gone.

Start small, watch the pods go Pending and then Running, confirm a node appears and disappears, and you’ll have the core loop that scales all the way up to serious production workloads.

If you build on this, the most valuable thing you can add is good observability — managed Prometheus is already enabled in this setup, so point Grafana at your vLLM metrics and watch queue depth, latency, and TPU utilization as the cluster scales. That’s where autoscaling stops being a demo and starts being a system you can trust.