Member-only story
We Deleted 14 Dependencies. Our Java Container Now Uses 47% Less Memory
The heap was fine. The other 500 MB was hiding in code we had invited in.
Our payments service needed 1140 MB of memory to accept a JSON payload, validate it, and write 1 row to Postgres.
Kubernetes kept killing it at 3 in the morning. Nobody had touched the code in 6 weeks.
The pod limit was 1.25 GB. Any burst of traffic pushed it over the edge, the pod got OOMKilled, and the on-call engineer got a page that explained nothing.
The Heap Was Not the Problem
My first instinct was to blame the heap. It is always the heap, right?
Except our heap was fine. GC logs showed 410 MB used out of 640 MB committed, with short and boring pauses. So where was the other 500 MB going?
Java will not tell you unless you ask, and you have to ask in a very specific way.
java -XX:NativeMemoryTracking=summary -jar app.jar
jcmd 1 VM.native_memory summaryInside a container, your app is usually PID 1, hence the 1. NMT costs a little CPU and memory of its own, so switch it on to investigate, not forever.
