Your Kubernetes App With HPA Was Running Fine… Until It Wasn’t

Paul Boby

9 min read

·

Aug 31, 2026

--

Listen

Share

“It was working yesterday.”

Famous last words in every on-call rotation.

You’ve deployed your application on Kubernetes. You’ve configured HPA (Horizontal Pod Autoscaler) so it automatically scales your app up when traffic spikes and scales it down when things calm down. It worked perfectly for weeks. Then suddenly — the app starts throwing errors, pods aren’t scaling, and your alerting is going crazy.

What do you do?

This article walks you through exactly how to think and exactly what commands to run — from the first moment of panic all the way to the final fix.

First — What Is HPA and How Does It Work?

Before we debug, let’s make sure we understand what HPA actually does.

HPA (Horizontal Pod Autoscaler) is Kubernetes’s built-in mechanism that watches your application’s resource usage and automatically adjusts the number of running pod replicas based on what it sees.

Think of it like a restaurant manager. On a normal Tuesday, you have 2 waiters. Friday night rush hits — the manager calls in 6 more. After closing time, they’re sent home. HPA is that manager, watching traffic and calling pods in or sending them home automatically.

A typical HPA configuration looks like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60   # Scale up when CPU crosses 60%

This tells Kubernetes: “Keep my app between 2 and 10 pods. If average CPU crosses 60%, add more pods. If it drops well below 60%, remove some.”

Simple enough. But when it breaks, the reason is rarely obvious. Let’s fix that.

The Troubleshooting Approach

Experienced engineers don’t panic and randomly restart things. They follow a structured path — like a doctor examining a patient. You go from general symptoms to specific causes.

Observe first. Conclude second. Act third. Never restart pods before you understand why they’re failing — you’ll hide the symptom but the problem will return.

Here is the exact order to follow.

Step 1: Check the HPA Status Immediately

This is your very first command. Run it as soon as you get alerted.

kubectl get hpa my-app-hpa -n my-namespace

# Output:
NAME          REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS
my-app-hpa   Deployment/my-app  <unknown>/60%    2         10        2

See that <unknown> in the TARGETS column? That is your red flag.

It means HPA cannot read the CPU metrics at all. It has absolutely no idea whether your app is busy or idle — so it does nothing. It just sits frozen at 2 pods even if your app is receiving 50,000 requests per second.

Get more details with describe:

kubectl describe hpa my-app-hpa -n my-namespace

Scroll to the bottom of the output and look at the Conditions and Events sections. This is where Kubernetes tells you exactly what is wrong in plain English.

# Common error you'll see:
Events:
  Warning  FailedGetScale  2m  horizontal-pod-autoscaler
  failed to get cpu utilization: missing request for cpu

“Missing request for cpu” is one of the most common root causes. It means your pod containers don’t have CPU resource requests defined. HPA needs a resource request to calculate the utilization percentage — without it, HPA is completely blind.

Step 2: Check If the Pods Are Actually Running

Before blaming HPA, verify that the pods themselves are in good health.

kubectl get pods -n my-namespace -l app=my-app

# Healthy:
NAME                     READY   STATUS    RESTARTS   AGE
my-app-7d9f8b-xkq2p     1/1     Running   0          5d# Unhealthy:
NAME                     READY   STATUS             RESTARTS   AGE
my-app-7d9f8b-xkq2p     0/1     CrashLoopBackOff   15         5d
my-app-7d9f8b-mn3pp     0/1     Pending            0          2m

CrashLoopBackOff means the application is crashing immediately after startup and Kubernetes keeps trying to restart it. Pending means Kubernetes cannot find a node with enough free resources to place the pod.

If pods are crashing, check their logs:

kubectl logs my-app-7d9f8b-xkq2p -n my-namespace --previous

# The --previous flag shows logs from the last crashed container
# Look for application-level errors:
# Error: ENOMEM: not enough memory
# java.lang.OutOfMemoryError: Java heap space
# Connection refused: postgres:5432

The logs will almost always tell you exactly why the application is crashing.

Step 3: Check the Metrics Server

HPA does not read CPU and memory directly. It relies on a separate component called the Metrics Server — a small service running inside your cluster that collects resource usage from all pods and exposes it through the Kubernetes API.

If the Metrics Server is down, HPA is completely blind. This is more common than you’d think.

kubectl top pods -n my-namespace

# If Metrics Server is working:
NAME                     CPU(cores)   MEMORY(bytes)
my-app-7d9f8b-xkq2p     245m         128Mi# If Metrics Server is down:
error: Metrics API not available

Check if the Metrics Server itself is running:

kubectl get pods -n kube-system | grep metrics-server

# Healthy:
metrics-server-6d94bc8694-t9kpg   1/1   Running          0   3d# Dead:
metrics-server-6d94bc8694-t9kpg   0/1   CrashLoopBackOff  8   2h

If it is crashing, check its logs:

kubectl logs -n kube-system deploy/metrics-server

A very common Metrics Server issue is a TLS certificate error with kubelets. The quick fix is to add --kubelet-insecure-tls to the Metrics Server deployment arguments. This is acceptable in most internal clusters.

Step 4: Verify Resource Requests Are Set on Your Pods

This is the single most common reason HPA shows <unknown> for its target metric.

HPA calculates CPU utilization as a percentage. To calculate a percentage, it needs to know “what is 100%?” That reference point is your container’s resource request. If you haven’t set one, HPA literally cannot do any math.

kubectl describe pod my-app-7d9f8b-xkq2p -n my-namespace | grep -A5 "Requests"

# Good — requests are defined:
Requests:
  cpu:     250m
  memory:  256Mi# Bad — no requests defined:
Requests:
  (none)

The fix is to add resource requests to your Deployment spec:

resources:
  requests:
    cpu: "250m"      # 250 millicores = 0.25 of one CPU core
    memory: "256Mi"
  limits:
    cpu: "1000m"     # Max 1 full CPU core
    memory: "512Mi"

After adding this and redeploying, HPA will immediately be able to read the CPU utilization correctly.

Step 5: Check If Your Nodes Have Capacity

HPA might be trying to scale up — but if every node in your cluster is already packed full, new pods will stay in Pending state forever. HPA keeps trying, your app keeps struggling, and nobody can figure out why nothing is changing.

kubectl describe nodes | grep -A8 "Allocated resources"

# Output:
Allocated resources:
  Resource    Requests      Limits
  --------    --------      ------
  cpu         3850m (96%)   4200m (105%)
  memory      6Gi   (94%)   8Gi   (118%)

If your nodes are at 96% CPU allocation, there is almost no room for new pods. HPA issues the scale-up command, Kubernetes tries to schedule new pods, finds no available space, and the pods stay Pending. From the user’s perspective, the app is just broken.

Solutions in this case: add more nodes to the cluster, resize existing nodes to be larger, or review whether existing pods are over-requesting resources they never actually use.

Step 6: Check If HPA Has Hit Its Ceiling

This one is subtle but very common during sudden traffic spikes.

kubectl get hpa my-app-hpa -n my-namespace

NAME          TARGETS   MINPODS   MAXPODS   REPLICAS
my-app-hpa   92%/60%    2         10        10

CPU is at 92%, which is way above the 60% threshold. But HPA has already scaled to 10 pods — and 10 is the maxReplicas limit. HPA physically cannot go higher. It is doing exactly what you told it to do, but you told it to stop at 10.

The temporary fix during an incident is to increase the limit:

kubectl patch hpa my-app-hpa -n my-namespace \
  -p '{"spec":{"maxReplicas":15}}'

The permanent fix is to review your maxReplicas value and make sure it has realistic headroom above your expected peak load.

The Sneaky One: The Bottleneck Is Not CPU

Here is a scenario that trips up even experienced engineers.

Your HPA is configured to scale on CPU. CPU is sitting at 30%. Pods all show Running. HPA says everything is fine. But users are still complaining the app is slow or timing out.

The problem? The bottleneck has nothing to do with CPU. HPA is correct — CPU is fine. But something else is overwhelmed:

• The database is the real bottleneck. More pods = more database connections = the database chokes. Adding more pods actually makes things worse in this case.
• An external API is rate-limiting your app. More pods send more requests to that API, which throttles them, causing timeouts.
• Thread pool exhaustion. Your app has a fixed thread pool. All threads are blocked waiting for slow I/O responses. CPU looks idle because threads are just waiting, not computing.
• Network or disk I/O saturation. CPU is fine but the data pipeline is clogged.

In these cases, CPU-based HPA scaling does nothing useful — or makes things worse. The real solution is to identify the actual bottleneck using APM tools, database slow query logs, or distributed tracing.

For these scenarios, consider configuring HPA on custom metrics like request queue depth or active HTTP connections rather than raw CPU utilization. This gives HPA a much more accurate picture of your application’s real load.

Quick Reference: Common Causes and Fixes

What You SeeWhat It MeansHow to Fix<unknown>/60% in TARGETSHPA cannot read metricsCheck Metrics Server + add CPU requests to podsPods stuck at PendingNo node capacity availableAdd nodes or reduce resource requestsCrashLoopBackOffApp crashes on startupCheck pod logs for application errorsHPA not scaling up despite high CPUAlready at maxReplicas limitIncrease maxReplicas in HPA specHPA not scaling down after load dropsCooldown period (default 5 minutes)Wait, or tune stabilizationWindowSecondsMetrics API not availableMetrics Server is downRestart/fix Metrics Server in kube-system namespaceReplicas at minimum but app is slowCPU is not the real bottleneckCheck database, network, thread pool, external APIs

A Real-World Example: The Friday Night Outage

Let me walk you through a real scenario. It’s 11 PM on a Friday (always Friday). Alerts fire: Response time over 5 seconds for my-app in production.

Step 1 — Check HPA:

kubectl get hpa my-app-hpa -n production

NAME          TARGETS   MINPODS   MAXPODS   REPLICAS
my-app-hpa   92%/60%    2         10        10

CPU is at 92%, which is way above the 60% threshold. HPA has scaled to 10 replicas. But 10 is the maximum. HPA has hit its ceiling and cannot go further.

Step 2 — Check nodes:

kubectl get nodes

NAME       STATUS    ROLES    AGE
node-01    Ready     worker   30d
node-02    Ready     worker   30d
node-03    NotReady  worker   30d   <-- This one just died

One of three nodes went down. The cluster lost 33% of its capacity. Kubernetes automatically rescheduled pods from the dead node onto the two remaining nodes — which caused CPU to spike across the cluster. HPA tried to scale up but was already at maxReplicas.

Fix applied:

# Temporary: increase maxReplicas while node is being restored
kubectl patch hpa my-app-hpa -n production \
  -p '{"spec":{"maxReplicas":15}}'

# Drain and restore the dead node
kubectl drain node-03 --ignore-daemonsets --delete-emptydir-data
# ... fix the node OS, reboot, or replace it ...
kubectl uncordon node-03

HPA immediately scaled to 13 pods. CPU dropped to 48%. Response times recovered within 3 minutes. The dead node was restored in 20 minutes. Total incident duration: 25 minutes.

The Complete Debugging Checklist

Save this. Use it on every HPA incident.

# 1. Check HPA status
kubectl get hpa -n <namespace>
kubectl describe hpa <hpa-name> -n <namespace>

# 2. Check pod health
kubectl get pods -n <namespace> -l app=<your-app>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous# 3. Check Metrics Server
kubectl top pods -n <namespace>
kubectl get pods -n kube-system | grep metrics-server
kubectl logs -n kube-system deploy/metrics-server# 4. Check node capacity
kubectl get nodes
kubectl describe nodes | grep -A8 "Allocated resources"# 5. Check events (gold mine for errors)
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20# 6. Check resource requests on pods
kubectl get pod <pod-name> -n <namespace> \
  -o jsonpath='{.spec.containers[*].resources}'

How to Prevent This From Happening Again

The best incident is the one that never becomes one.

• Always set resource requests AND limits on every container. Without requests, HPA cannot function. This is non-negotiable.
• Alert on HPA TARGETS showing <unknown>. Don’t wait for users to notice. Set up a monitoring alert that fires the moment HPA loses its metrics.
• Set maxReplicas with realistic headroom. Your maxReplicas should be significantly higher than your expected peak, not exactly at it.
• Monitor Metrics Server health separately. Treat the Metrics Server like any other critical component — alert if it goes down.
• Load test regularly. Simulate traffic spikes in a staging environment before they hit production. Discover your HPA ceiling before your users do.
• Use PodDisruptionBudgets. These ensure a minimum number of pods stay running during node drains or cluster upgrades, preventing cascading failures.

Summary

When an application with HPA configured suddenly stops working, follow this mental model:

1. Check HPA status — is TARGETS showing <unknown>?
2. Are pods Running, Pending, or CrashLoopBackOff?
3. Is the Metrics Server alive?
4. Do pods have CPU resource requests defined?
5. Are nodes out of capacity?
6. Has HPA hit the maxReplicas ceiling?
7. Is the real bottleneck actually CPU — or is it something else entirely?