Investigating High Memory Utilization in a WSO2 API Manager Deployment
Imagine you’re monitoring a production WSO2 API Manager deployment and suddenly notice one of your gateway pods sitting at 90% memory utilization.
Your first thought is probably the same as mine would be:
“Is this application about to run out of memory? 😰”
After all, high memory utilization is usually treated as a warning sign. In Kubernetes environments, it’s often associated with memory leaks, aggressive garbage collection, or even a potential OutOfMemory (OOM) kill.
💭 But here’s the interesting part.
In many JVM-based applications, a high memory reading at the pod level doesn’t necessarily mean the application is under memory pressure.
Let’s look at a real-world example.
Consider a WSO2 API Manager gateway pod was observed consuming approximately 3.7 GB of memory out of its 4 GB pod limit.
At first glance, the numbers looked concerning:
Pod Memory Usage: 3.7 GB
Pod Limit: 4.0 GB
Utilization: ~90%Most engineers would immediately start looking for memory leaks.
However, when we examined the JVM itself, we found something unexpected… i.e. The application was only using around 900 MB of actual heap space.
So where was the remaining memory coming from ? 🤔
To answer that question, we need to understand how the JVM manages memory and why the numbers reported by Kubernetes don’t always tell the full story.
First, What Is a JVM?
Before we walk through the investigation, let’s establish ground zero — because none of this makes sense without it.
WSO2 API Manager (like most enterprise Java applications) doesn’t run natively on the operating system. It runs inside a program called the JVM — the Java Virtual Machine.
Think of the JVM as a managed environment that sits between your application and the OS:
The JVM manages all the memory that APIM needs.
The OS and Kubernetes only see “the JVM process is consuming X GB” — they can’t see what’s happening inside it.
That gap between what Kubernetes sees and what’s actually happening inside the JVM is exactly what this case is about 💡.
What Is the JVM Heap?
The heap is the main chunk of memory the JVM uses to store your application’s live data — things like:
- Active user sessions
- API request and response objects
- Cached OAuth tokens
- Configuration objects loaded at startup
Imagine the heap as a warehouse:
The heap is controlled by two JVM startup flags per application (Java process):
Let’s say in APIM configuration (api-manager.sh file located at <APIM_HOME>/bin), we have:
-Xms 2g -Xmx 2gBoth set to 2 GB. That single configuration choice is the key to this entire investigation 📌.
The Xms = Xmx Trap
When you set Xms = Xmx, the JVM reserves the full heap amount per application the moment it starts — before a single user request arrives, before any data is loaded.
It walks up to the operating system and says: “Give me 2 GB. I’ll manage what I put in it.”
The OS hands over 2 GB and marks it as allocated. From that point on, the OS — and therefore Kubernetes and Grafana — reports it as used memory, even if the actual data inside is a fraction of that.
Here’s what the heap actually looked like in this use case:
The JVM was holding a 2 GB reservation it was mostly not using. And Kubernetes was dutifully reporting that 2 GB as consumed memory.
This is not a bug. It’s a deliberate JVM design choice :D
Reserving the full heap upfront prevents the JVM from having to resize it dynamically during production traffic, which would cause pauses and unpredictable latency. But it makes pod-level metrics deeply misleading if you don’t know what you’re looking at.
Real Heap vs Reserved Heap
This distinction is the single most important concept in JVM memory analysis:
The real heap is the number that tells you whether the application is healthy.
Get Dilky Liyanage’s stories in your inbox
Join Medium for free to get updates from this writer.
If real heap is steadily growing toward Xmx over hours or days → suspect a memory leak 😐. If real heap is stable and well below Xmx → the application is healthy 😃, regardless of what the pod-level gauge says.
What Lives Outside the Heap — Off-Heap Memory
Here’s where things get a little more complex. The heap is only one region of what a JVM process actually consumes. There is a whole world of off-heap memory — memory the JVM uses that sits completely outside the heap boundaries:
Metaspace is where Java class definitions live. Every time APIM loads a class — an API handler, a filter, a connector — its definition is stored here, not in the heap.
Direct buffers are memory regions outside the heap used for network I/O. APIM handles enormous volumes of HTTP traffic. Moving data directly without going through the heap is faster, so the JVM allocates these buffers off-heap. They don’t count against Xmx, but they absolutely count against pod-level memory.
Thread stacks are small per-thread memory areas that track the current call chain for each running thread.
None of this is controlled by Xmx. It all adds up on top of the heap reservation, and Kubernetes counts every byte of it.
The Investigation: Reading Memory at Four Layers
With this foundation, here’s how we can approach the incident.
We can collect data at four distinct layers, starting from the outermost Kubernetes view and drilling inward to the JVM internals:
Each layer answers a different question. The mistake most engineers make is stopping at Layer 1 and treating a Kubernetes dashboard as a diagnosis. It isn’t — it’s just the first clue.
Layer 3: What Is jcmd?
jcmd is a command-line tool that ships with the JDK. You run it directly inside the pod where APIM is running, and it connects to the live JVM and queries it in real time.
jcmd <PID> GC.heap_infoGC stands for Garbage Collector — the JVM subsystem that automatically frees memory occupied by objects that are no longer needed. GC.heap_info asks the garbage collector: “Right now, how much heap are you managing, and how much of it is actually used ? 🤓”
A simplified version of what the output looks like:
Heap usage:
garbage-first heap total 2097152K, used 921600KTranslation:
total heap = 2 GB, actually used = ~900 MB. The other 1.1 GB is empty, sitting in reserve.
This single output was the turning point in the investigation. The pod looked like it was at 90% from the outside. The JVM heap was less than half full on the inside.
Layer 4: What Is a Heap Dump?
A heap dump is a snapshot of the application’s (process ID) heap usage— a file that captures every object in the JVM heap at a specific point in time. Think of it as a photograph of the warehouse, with every box labeled and counted.
Heap dumps are analyzed with tools like Eclipse MAT or VisualVM. In this case, the dump confirmed ~900 MB of live objects — perfectly consistent with what jcmd had already shown.
That cross-check matters. If jcmd and the heap dump disagreed, it would point to either a measurement issue or memory consumption in non-heap regions that needed separate investigation.
Putting It All Together: The Full Memory Breakdown
Here’s where every byte of those 3.7 GB actually came from:
There was no memory leak. No GC pressure. No objects accumulating. The application was running at ~900 MB of real heap usage against a 2 GB ceiling — 45% of available heap, completely comfortable.
The alarm was real. The danger wasn’t.
The Diagnosis: Three Signs This Is NOT a Memory Leak
1. jcmd showed ~900 MB used — far below the 2 GB Xmx. A real leak would push the used heap toward Xmx steadily over time.
2. Heap dump confirmed ~900 MB live objects — consistent with jcmd. A leak typically shows large collections of stale objects accumulating across multiple snapshots taken hours apart.
3. No OOM errors in logs — an OutOfMemoryError appears when the JVM genuinely runs out of heap.
A real memory leak looks like this: heap usage climbing steadily, GC running more and more frequently trying to free space, GC pause times increasing, eventually ending in an OOM crash.
The Fix
Recommendation 1 — Increase the pod memory limit
The math is fixed: with Xms = Xmx = 2 GB plus off-heap overhead, this pod’s floor is roughly 3.7 GB. The only practical way to make the 90% alarm go away is to raise the ceiling, not change the JVM:
Before: [████████████████████░░░] 3.7 / 4.0 GB = 90% 🚨
After : [████████████░░░░░░░░░░░░░░░░░░░░░░░░] 3.7 / 8.0 GB = 46% ✅The JVM settings stay unchanged. The heap is healthy — raising Xmx would just cause the JVM to reserve even more memory it doesn’t need and make the situation worse.
Recommendation 2 — Disable the unused diagnostic tool
That second JVM process (PID 305, 0.7 GB) was APIM’s built-in diagnostic tool running as a completely separate JVM. If you’re not actively using APIM’s diagnostic dashboard, a single config change disables it and reclaims that memory:
[diagnostic_tool.server_configuration]
diagnostic_tool_enabled = "false"Note: this configuration key requires APIM 4.3.0 onwards.
The dashboard is your smoke detector. jcmd and heap dumps are the fire investigation. One tells you something might be wrong. The other tells you what’s actually happening 😆
Happy Debugging! 👋🏻
