OutOfMemoryError: Java Heap Space — Diagnosing With Heap Dumps

The first alert came in at 17:12 on a Thursday, and the stack trace pointed at a line that formats a currency amount. Fourteen characters into a StringBuilder. That is the allocation that failed, and I spent forty minutes reading the formatting code before I accepted that it had nothing to do with anything.

The heap dump, once I finally had one, said the process was holding 2.1 GB in a single ConcurrentHashMap that a colleague had added eleven months earlier as a "temporary" idempotency cache. It had never evicted anything. It had been fine for eleven months because the traffic had been fine for eleven months, and that Thursday a partner started replaying a backlog.

That gap — between where the error is thrown and where the memory actually went — is the whole difficulty with OutOfMemoryError: Java heap space. Every other Java failure gives you a trace that names the problem. This one names a bystander. Whoever allocated last pays for everyone else, and the code in the trace is usually the most innocent code in the process, guilty of nothing except asking for a few bytes at the wrong microsecond.

So the diagnostic method is different. You do not read the trace. You read the heap, and this post is mostly about how to do that: what to turn on before the incident, how to get a dump out of a live process, and what to actually look at once you have a two-gigabyte file open in front of you.

Table of Contents

• What the Error Actually Means
• Why It Happens: Reachability, Not Usefulness
• Cause 1: A Genuine Leak
• Cause 2: No Leak, Just Too Much Live Data
• Cause 3: Retention You Did Not Expect
• Diagnosing With Heap Dumps
• How to Fix It, and What Each Fix Costs
• How to Prevent It
• Frequently Asked Questions

What the Error Actually Means

Same family as the last one:

java.lang.Throwable
 └── java.lang.Error
      └── java.lang.VirtualMachineError
           ├── java.lang.StackOverflowError               // no stack left
           ├── java.lang.OutOfMemoryError                 // no heap left
           └── java.lang.InternalError

OutOfMemoryError is a VirtualMachineError, which means the JVM is telling you it can no longer do the job you gave it. It is not an exception describing a condition your code was supposed to anticipate. There is no correct handler.

The next thing to do, before anything else, is read the message after the colon. OutOfMemoryError is one class carrying at least six distinct failures, and the message decides where you look. Getting this wrong costs hours, because you will happily spend an afternoon in a heap dump for a problem that has nothing to do with the heap.

java.lang.OutOfMemoryError: Java heap space
    → the collector could not make room in the heap under -Xmx. This post.

java.lang.OutOfMemoryError: GC overhead limit exceeded
    → same underlying problem, caught earlier: >98% of time in GC,
      <2% of heap recovered. A heap-space failure that announced itself.

java.lang.OutOfMemoryError: Metaspace
    → class metadata, not objects. Classloader leak, or a framework
      generating proxies in a loop. -Xmx is irrelevant here.

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    → someone asked for an array near Integer.MAX_VALUE elements.
      Almost always a corrupt length field read off a stream.

java.lang.OutOfMemoryError: unable to create native thread
    → the OS refused a thread. Ulimits, cgroup pids limit, or an
      unbounded executor. Heap is fine; you have too many threads.

java.lang.OutOfMemoryError: Direct buffer memory
    → off-heap ByteBuffers under MaxDirectMemorySize. NIO, Netty,
      a driver. Will not appear in an ordinary heap dump.

Only the first two send you to a heap dump. The others are different problems wearing the same class name, and the most common wasted afternoon in this whole area is someone analysing object retention for a Metaspace failure.

For the rest of this post, the message is Java heap space. What that specifically means is: a thread asked the allocator for room, the allocator could not find it, the collector ran — usually a full, stop-the-world collection, and usually not the first one — and afterwards there was still not enough contiguous space to satisfy the request. Only then does the JVM give up and throw.

Two consequences follow from that, and both matter.

First, the throw site is not the leak. The failing allocation is whichever one happened to arrive after the heap was already full. It is a lottery, and small frequent allocations win it disproportionately — string formatting, boxing, an ArrayList growing by one. If you find yourself optimising the code in the trace, stop. It is a witness, not a suspect.

Second, the process was already unwell before it threw. A JVM does not go from healthy to OutOfMemoryError in one step. It spends minutes, sometimes hours, doing progressively more full GCs that recover progressively less. Latency degrades, throughput drops, and every one of those symptoms is visible in GC logs before the error appears. By the time you see the error, you have missed several chances to see it coming.

And unlike StackOverflowError, which kills one thread and leaves the other two hundred perfectly fine, this one is global. The heap is shared. Every thread in the process is now allocating into the same exhausted space, so the failure spreads: one thread throws, then another, then the health-check endpoint, and a process that is technically still running answers nothing. Half-completed work is scattered everywhere, because each of those threads died at an arbitrary point.

Why It Happens: Reachability, Not Usefulness

The heap is where every object you allocate with new lives, along with every array, every string, every lambda capture. It is sized between -Xms and -Xmx, and -Xmx is a hard ceiling: the JVM will not exceed it no matter how much free memory the machine has.

Collectors divide it up in various ways — generations in G1 and Parallel, regions, survivor spaces — and those details change how efficiently memory is reclaimed, not what can be reclaimed. The rule underneath every collector is one sentence, and it is the sentence people quietly disbelieve:

An object is collectable if and only if it is unreachable from a GC root.

Not "if you are done with it". Not "if it has fallen out of scope". Not "if it is old". Reachability, and nothing else. GC roots are the anchors: static fields of loaded classes, local variables and parameters on every live thread stack, JNI references, active threads themselves, and a few internal structures. If there is any chain of references from any root to your object, it stays, and the collector will never take a view about whether it deserves to.

This is exactly why memory leaks in Java feel unfair. You cannot leak by forgetting to free something. You leak by remembering — by holding a reference in a structure that outlives the thing it points at. A static Map is a GC root with a hole in it: everything you put in is permanently reachable, and the collector will faithfully preserve two gigabytes of dead session data for as long as the process runs, because you told it to.

So there are only two ways to run out. Either the live set is genuinely bigger than -Xmx — you legitimately need more memory than you have — or something is reachable that should not be. Every real incident is one of those, and the whole job of a heap dump is deciding which.

One more thing worth internalising, because it saves you from a bad reflex during an incident: the JVM has already tried everything by the time it throws. It ran full collections. It compacted. On most collectors it cleared every SoftReference it had, which is the last-ditch measure, and if a soft-reference cache was holding memory it is already gone by the time you see the error. There is no additional work you can ask for. This is the reason System.gc() in a catch block does nothing except add a pause: the collector you are politely asking to run has just run, exhaustively, and failed.

Cause 1: A Genuine Leak

Something accumulates, and nothing ever removes it. The live set climbs across the lifetime of the process, the sawtooth in your memory graph drifts upward instead of returning to the same floor, and the time-to-failure is a function of uptime and traffic rather than of any single request.

My idempotency cache was the plainest version of this:

package com.acme.payments;

public final class IdempotencyGuard {

    // "temporary — replace with Redis before launch"  (added 2025-09)
    private static final Map<String, PaymentResult> SEEN =
            new ConcurrentHashMap<>();

    public PaymentResult once(String key, Supplier<PaymentResult> work) {
        return SEEN.computeIfAbsent(key, k -> work.get());
    }
}

Static field, so it is a GC root. Nothing removes. Every distinct payment key the service has ever seen is still in there, along with the full PaymentResult each one produced, and each of those holds a line-item list and a copy of the request payload. Eleven months and roughly 3.4 million keys later, a partner replayed a backlog and pushed it over.

The three shapes I have actually seen in production, in order of how often:

The unbounded cache. Every one of them started as a HashMap with a comment promising an eviction policy. If a map is a cache it needs a bound and an eviction rule from the first commit, because the version without one works perfectly right up until it does not, and there is no intermediate state where somebody notices.

The registry nobody deregisters. A listener list, a callback registry, an event-bus subscription, a metrics registry keyed by something with unbounded cardinality. Registration is easy to write and easy to review. Deregistration happens on a path somebody forgot — the error path, the timeout path, the path where the client disconnects mid-request. Each orphaned listener holds its enclosing object, which holds a request context, which holds a response buffer. One leaked lambda can retain a surprising amount.

ThreadLocal on a pooled thread. This one is a design trap rather than an oversight. ThreadLocal is fine when the thread dies afterwards. On a pool thread that lives for the life of the process, whatever you set stays set until someone removes it, and the next four thousand requests handled by that thread inherit a slot holding the first request's data.

private static final ThreadLocal<RequestContext> CTX = new ThreadLocal<>();

// in the filter
CTX.set(new RequestContext(request));   // 200 pool threads
try {
    chain.doFilter(request, response);
} finally {
    CTX.remove();                       // ← the line that is always missing
}

Without the finally, you retain 200 request contexts forever — and worse, they are the contexts of 200 arbitrary historical requests, which makes the dump confusing to read because the retained objects have no relationship to current traffic. The half-fix people reach for is a WeakReference in the value, which does not help: the ThreadLocalMap entry's key is already weak, and it is the strongly-held value that pins your object.

In a dump this shape is unmistakable. The dominator tree has one entry near the top holding a large fraction of the heap, a single collection with an implausible element count, and a path to GC roots that terminates in a static field. It is the easiest of the three to diagnose and usually the easiest to fix.

Cause 2: No Leak, Just Too Much Live Data

Nothing is retained wrongly. Every object in the heap at the moment of failure is genuinely in use by work that is genuinely in progress. There is simply more of it than -Xmx allows.

This is the one people resist, because "we have a memory leak" is a more satisfying story than "we asked for too much at once". It is also, in my experience, at least as common.

// Ran nightly for two years. 10k rows, ~40 MB, never a problem.
List<Transaction> all = repository.findAllByAccount(accountId);
byte[] pdf = reportRenderer.render(all);

Nothing about that is wrong. It is correct, readable, well-tested code that encodes one assumption nowhere written down: that the result set is small. Then a corporate account onboards with 4.1 million transactions, and the same line asks for a list roughly four hundred times bigger than the one anybody tested. The heap does not leak. It is asked, in a single call, for more than it has.

The variants are all the same mistake with different nouns. Files.readAllBytes on a file whose size is a property of somebody else's upload. new String(bytes) on an HTTP response with no content-length cap. A JPA query without pagination, which is the same unbounded query wearing a framework. A batch size tuned when a batch meant ten thousand rows. A Kafka consumer that buffers a whole partition's worth of records because max.poll.records was never set and the messages turned out to be a hundred times larger than they used to be.

The concurrency multiplier is what usually turns it into an incident. A request that materialises 80 MB is fine alone and fatal at forty concurrent. Your capacity is not the size of one request; it is one request times peak concurrency, plus everything else the process needs, and almost nobody tests the second number.

The tell in a dump is the absence of a smoking gun. No single dominator holds more than a few percent. Instead there are two hundred instances of the same request-scoped object, each holding a few megabytes, all of them legitimately alive, all of them referenced from a thread stack rather than a static. When the paths-to-GC-roots all end at live threads doing real work, you are not looking at a leak. You are looking at capacity, and the fix is to make each unit of work smaller rather than to hunt for a defect.

Cause 3: Retention You Did Not Expect

The heap holds far more than the code suggests, because a small object is keeping a large one alive. Everything is reachable, so the collector is behaving correctly, and reading the source tells you nothing because the retention is in a reference you did not think about.

The canonical form is a slice that pins its backing store:

// Reads a 12 MB file, keeps 40 characters, retains all 12 MB.
final class Header {
    private final byte[] raw;      // the whole file
    private final int off, len;    // the bit we care about

    String value() { return new String(raw, off, len, UTF_8); }
}

Cache a few thousand Header objects and you have quietly cached a few thousand whole files. The shallow size of a Header is about 32 bytes. Its retained size is twelve megabytes. That gap between shallow and retained is the single most useful idea in heap analysis and I will come back to it.

Java's own String.substring used to do exactly this — before Java 7u6, a substring shared the parent's char[], so keeping one word of a large document kept the document. It was fixed by copying, and the pattern survives everywhere people write wrappers: a ByteBuffer slice, a wrapper around a byte[], a parsed record holding a reference to the raw payload it came from, a domain object that kept the JSON it was deserialised from "for debugging".

The one that surprised me most involved an exception. Someone was collecting failures for a summary report:

class BatchFailure {
    final String recordId;
    final Exception cause;    // ← this
}
List<BatchFailure> failures = new ArrayList<>();

Holding an exception holds its stack trace, and every frame in that trace holds nothing much — but the exception was constructed with the request object as a field so the message could be built lazily, and that request held a parsed document. Twelve thousand failures in a bad run, each retaining about 900 KB, and the "summary" list was three quarters of the heap. The code reads as if it stores an id and an error.

Then there are references people believe are magic. A WeakReference is only weak if there is no strong reference elsewhere, and putting a weak reference inside a strongly-held wrapper achieves precisely nothing. WeakHashMap is weak in its keys, so a value that points back at its own key keeps the entry alive forever, which is a genuinely nasty bug to find. SoftReference is not a cache policy; the specification lets the collector clear soft references whenever it likes and in practice HotSpot clears them all just before throwing this error, so a soft-reference cache is either useless under memory pressure or invisible in the dump you took afterwards.

And listeners on long-lived objects. An anonymous inner class or a non-capturing-looking lambda that references an instance field captures this, and this may be a controller holding a whole object graph. Registering it on a singleton event bus and never removing it retains that graph for the life of the process. The source line is one lambda. The retained size is a subsystem.

Diagnosing With Heap Dumps

A heap dump is a snapshot of every object in the heap: classes, fields, references, array contents, and enough structure to reconstruct the entire object graph. It is the only artefact that answers the question you actually have, which is not "what failed" but "what was in there".

Turn the flags on before you need them

These belong in your base image, not in a runbook step you perform after the first outage:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/app/dumps/
-Xlog:gc*:file=/var/log/app/gc.log:time,uptime,level:filecount=5,filesize=20M

The first is off by default and costs nothing while the process is healthy. The second decides where the file goes: a directory gets you java_pid<pid>.hprof, a filename gets you exactly that name, and omitting it drops the file in the working directory — which in a container is usually an ephemeral layer that vanishes with the pod. Point it at a mounted volume. The third gives you the GC log, which is how you would have seen this coming.

Three warnings that people learn the expensive way. The file is roughly the size of the live heap, so an 8 GB heap writes an 8 GB file, and if the volume has 2 GB free you get a truncated dump and a second error in the log. Writing it takes time — tens of seconds for a large heap — during which the process is unresponsive, so your orchestrator may kill it mid-write unless the termination grace period allows for it. And by default only one dump is written per process lifetime, which is the right default: you do not want a crash-looping pod filling a volume.

The dump also contains everything in memory, which means every password, token, PII field and cached response body sitting in the heap at that moment. Treat the file as production data with production access controls. I have seen an .hprof attached to a public ticket more than once.

Getting a dump from a live process

Often better, because you can take one before the process dies and another a few minutes later. jcmd is the current tool:

$ jcmd -l
48812 com.acme.payments.PaymentServiceApplication

$ jcmd 48812 GC.heap_dump -all=false /var/log/app/dumps/t1.hprof
48812:
Heap dump file created [1841266432 bytes in 9.412 secs]

# five minutes later, same process
$ jcmd 48812 GC.heap_dump -all=false /var/log/app/dumps/t2.hprof

# quick shape of the heap without writing a file at all
$ jcmd 48812 GC.class_histogram | head -12

Be honest with yourself about the cost. GC.heap_dump stops the world for the duration of the write. Nine seconds on a 1.8 GB heap here; on a 16 GB heap it can be a minute, and every request in flight times out. This is a deliberate act with a real availability cost, not a diagnostic you sprinkle around. Take it from one instance out of rotation if you can.

-all=false is the default and the one you want: it runs a full GC first so the dump contains only reachable objects. -all=true includes unreachable ones too, which makes the file bigger and the analysis harder for the sake of a question you are rarely asking. jmap -dump:live,format=b,file=... does the same thing and still works; jcmd is where the tooling is going.

Reading it: histogram first

Open the dump in Eclipse MAT if it is large — it handles multi-gigabyte files better than anything else free — or VisualVM if it is small and you want something already installed. Start with the histogram, which is just object counts and shallow sizes grouped by class:

Class Name                                    Objects    Shallow Heap   Retained Heap
--------------------------------------------------------------------------------------
byte[]                                      1,204,882     1,412,338,K      1,412,338,K
java.lang.String                            1,198,441        28,762,K      1,441,004,K
java.util.concurrent.ConcurrentHashMap$Node 3,401,772       108,856,K      2,147,483,K
com.acme.payments.PaymentResult             3,398,104       190,293,K      2,038,911,K
com.acme.payments.LineItem                  9,442,006       302,144,K        302,144,K
java.util.HashMap$Node                        482,119        15,428,K         92,006,K
--------------------------------------------------------------------------------------

Read that top down and the histogram tells you almost nothing useful. byte[] and String are at the top of every histogram ever taken; that is what Java programs are made of. The line that matters is the fourth: 3.4 million PaymentResult objects. That is a domain class, and a domain class with a seven-figure instance count is a fact about your application, not about Java.

So the histogram question is not "what is biggest" — it is "which of these counts is absurd?" Three million results in a service handling a few hundred payments a minute is absurd. Nine million line items is the consequence of the first number, not an independent problem. That one observation took me from "somewhere in the heap" to "something is keeping every payment result" in about fifteen seconds.

Then the dominator tree

The histogram groups by class, which scatters one logical problem across six rows. The dominator tree groups by who is responsible. Object A dominates B if every path from any GC root to B goes through A — so if A were collected, B would be too, and A's retained size is the memory that would actually come back.

Class Name                                             Shallow    Retained    Pct
---------------------------------------------------------------------------------
com.acme.payments.IdempotencyGuard$SEEN
  └─ java.util.concurrent.ConcurrentHashMap                 64   2,147,483,K   87.4%
     └─ ConcurrentHashMap$Node[4194304]                16,777,K   2,147,419,K   87.4%
        ├─ ConcurrentHashMap$Node                           32           612    0.0%
        ├─ ConcurrentHashMap$Node                           32           598    0.0%
        └─ ... 3,401,770 more entries ...
org.apache.tomcat.util.threads.TaskThread @ 0x7f2a1c        88     104,882,K    4.2%
java.lang.ref.Finalizer                                     40      12,004,K    0.4%
---------------------------------------------------------------------------------

There it is, in one line. One map, 87.4% of the heap, 3.4 million entries, and the label above it names the static field holding it. Notice how each individual entry is trivially small — 612 bytes — which is exactly why nothing in the histogram screamed and why no code review would have caught it. Nothing here is big. There are just far too many things, and one map is responsible for all of them.

Compare that to the shape of cause 2: no single row above 3%, and the top entries are all threads. When the dominator tree's biggest holders are thread objects doing live work, the memory is in flight rather than stuck.

Shallow versus retained

The distinction the whole tool is built on. Shallow size is the object's own header and fields — a few dozen bytes, and for anything with references it tells you nothing. Retained size is everything that would be freed if that object became unreachable: itself plus everything it exclusively dominates.

The Header class from cause 3 has a shallow size of about 32 bytes and a retained size of 12 MB. Sort a dump by shallow size and you will never see it. Sort by retained and it is the first thing you notice. If you take one operational habit from this post, make it that: sort by retained size, always, and treat any object whose retained size is wildly disproportionate to its shallow size as a suspect.

Paths to GC roots — the question that actually answers "why"

The dominator tree tells you what is holding the heap. It does not tell you why it is still alive. For that, pick the object and ask MAT for Path to GC Roots, excluding weak and soft references so you only see the references that genuinely prevent collection:

com.acme.payments.PaymentResult @ 0x6c40a8b90
 ↑ value of java.util.concurrent.ConcurrentHashMap$Node @ 0x6c40a8b40
 ↑ [1874318] of java.util.concurrent.ConcurrentHashMap$Node[4194304] @ 0x6c1000000
 ↑ table of java.util.concurrent.ConcurrentHashMap @ 0x6c0fffe20
 ↑ SEEN of class com.acme.payments.IdempotencyGuard @ 0x6c0a41200
 ↑ <system class loader>  ← GC root: static field

Read it bottom-up and it is a sentence: a static field named SEEN on IdempotencyGuard holds a map, whose table holds a node, whose value is this result. That is the leak, stated completely, in five lines. No code reading required.

The last line is the one to read first, because the root type classifies the problem. Static field means a class-level structure — a cache, a registry, a singleton. Thread means a live stack frame is holding it, so the work is in progress and you are probably in cause 2. Busy monitor or JNI global points at native code or a driver. And a path that runs through ThreadLocalMap names the ThreadLocal problem outright.

Two dumps beat one

A single dump tells you what the heap contains. It cannot tell you whether that is a leak or a high-water mark, and the difference decides your entire fix. So take two, several minutes apart, under comparable load, and compare them — MAT does this directly with a histogram delta:

$ jcmd 48812 GC.class_histogram | head -6      # t1, 17:04
 num     #instances         #bytes  class name
   1:       2,904,118      179,142,K  com.acme.payments.PaymentResult
   2:       2,901,004       92,832,K  java.util.concurrent.ConcurrentHashMap$Node

$ jcmd 48812 GC.class_histogram | head -6      # t2, 17:09
 num     #instances         #bytes  class name
   1:       3,398,104      190,293,K  com.acme.payments.PaymentResult      (+494k)
   2:       3,401,772      108,856,K  java.util.concurrent.ConcurrentHashMap$Node

Half a million instances in five minutes, and none of the older ones went away. That is monotonic growth, which is a leak. If both dumps had shown 2.9 million and the count had been flat, the same 2.9 million would be a working set that is merely large — annoying, possibly worth reducing, but not a defect and not fixable by finding a missing remove() call.

Note that GC.class_histogram also forces a full GC, so what you are comparing is live objects both times. That is what you want, and it is also why you should not run it every thirty seconds on a busy service.

When you cannot take a dump

Sometimes the honest answer is that the dump is not available or not useful. The heap is 60 GB and no machine you have can open it. The process is crash-looping and dies before you can attach. The dump was written to a container layer that no longer exists. Or you take it and everything looks completely reasonable, because the leak is slow and you caught it at 30% of the way there.

JFR's old-object-sample event is the answer for the last two. It samples a small number of allocations and keeps them under observation, recording the allocation stack trace and the path that kept them alive — so instead of a snapshot of what is alive now, you get a low-overhead record of what got old and why, and it will run continuously in production at a cost you will struggle to measure:

$ java -XX:StartFlightRecording=settings=profile,filename=app.jfr,maxsize=200M -jar app.jar

# or attach to something already running
$ jcmd 48812 JFR.start name=leak settings=profile
$ jcmd 48812 JFR.dump name=leak filename=/var/log/app/leak.jfr

Open that in JDK Mission Control and look at the TLAB and old-object-sample views. It is less complete than a heap dump and much cheaper, and for a leak that takes three days to develop it is often the only practical option.

How to Fix It, and What Each Fix Costs

1. Bound the collection

If the dump named a cache, give it a size limit and an eviction policy. Do not write your own; Caffeine has spent years on the eviction algorithm you would get wrong:

private static final Cache<String, PaymentResult> SEEN = Caffeine.newBuilder()
        .maximumSize(100_000)
        .expireAfterWrite(Duration.ofHours(24))
        .recordStats()
        .build();

Cost: correctness, and you have to think about it rather than wave at it. For an idempotency guard, eviction means a replayed request older than the window executes twice, so the bound has to be at least as long as the retry window your partners actually use, and the memory ceiling has to accommodate that. That is a real trade-off with a real conversation attached. It is still the fix, and maximumSize plus expireAfterWrite is the version I reach for by default because it bounds both memory and staleness.

For registries, the fix is symmetry: every register has a deregister in a finally, and every ThreadLocal.set has a ThreadLocal.remove in a finally. Cost: nothing but discipline.

2. Stream instead of materialising

For cause 2, stop holding the whole thing:

// Before: the entire result set in memory.
List<Transaction> all = repository.findAllByAccount(accountId);
byte[] pdf = renderer.render(all);

// After: one row at a time, straight to the output.
try (Stream<Transaction> rows = repository.streamByAccount(accountId)) {
    renderer.renderTo(outputStream, rows);
}

Same for files — Files.lines or a buffered read loop rather than readAllBytes, and a streaming JSON parser rather than binding a whole document to a tree. Peak memory stops being a function of input size and becomes a function of buffer size.

Cost: real, and worth stating plainly. The streaming version holds a database cursor and a transaction open for the length of the render, so a slow consumer now holds a connection; you need setFetchSize configured or the driver will helpfully buffer everything anyway and you will have changed nothing; and you can no longer make two passes over the data or ask it for its size up front. Some report formats need a total before they can write the header, and for those you need a separate count query. It is more code and more care. It also converts a failure mode into a non-issue permanently.

3. Page the query, cap the batch

Where streaming does not fit, put a ceiling on the unit of work. Paginate the query. Set max.poll.records. Chunk the batch into slices of a size you have actually tested, and enforce a request-level cap that returns a 413 rather than trying and dying. Cost: pagination is fiddly if the underlying data mutates between pages, and you need a stable sort key.

4. Fix the retention

For cause 3, copy the small thing out and let the big thing go:

// Before: retains the whole 12 MB backing array.
this.value = new Slice(raw, off, len);

// After: retains 40 bytes.
this.value = new String(raw, off, len, UTF_8);

Null out the reference to the payload once you have parsed it. Do not store the source document on the domain object "for debugging". Do not put a request object inside an exception you intend to keep. Cost: an allocation and a copy at the point of extraction, which is nothing next to retaining the source.

5. Container limits and MaxRAMPercentage

Before touching -Xmx, get the relationship between heap and container right, because these two failures look nothing alike and get confused constantly:

# A Java heap OOME: the JVM threw it, logged it, and dumped.
java.lang.OutOfMemoryError: Java heap space
  Dumping heap to /var/log/app/dumps/java_pid1.hprof ...

# A container OOM kill: no exception, no dump, log stops mid-line.
$ kubectl get pod payments-7d9f -o jsonpath='{.status.containerStatuses[0].lastState}'
{"terminated":{"exitCode":137,"reason":"OOMKilled"}}

Exit code 137 is SIGKILL from the kernel. The JVM never got a say, which is why there is no trace and no dump — and why "we set HeapDumpOnOutOfMemoryError and got nothing" is nearly always this. The cause is total process memory, not heap: heap plus metaspace plus code cache plus thread stacks plus direct buffers plus native allocator overhead all count against the cgroup limit. Setting -Xmx to the container limit guarantees this eventually.

# Fragile: a fixed heap that ignores what the container actually got.
-Xmx2g                          # in a 2 GiB container. This will be killed.

# Better: heap as a fraction of the detected limit, leaving native headroom.
-XX:MaxRAMPercentage=70.0
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/app/dumps/

Modern JVMs are container-aware and read the cgroup limit rather than the host's memory, so MaxRAMPercentage does the right thing automatically. Somewhere between 50% and 75% is the usual landing zone; the correct figure depends on how many threads you run and how much direct memory your stack uses, so measure the process RSS under load rather than trusting mine. Confirm what you actually got with java -XX:+PrintFlagsFinal -version | grep MaxHeapSize instead of assuming.

6. Raise -Xmx — last, not first

$ java -Xmx4g -jar payment-service.jar

Last, and only once you know which shape you have. If the live set after a full GC is flat and simply close to the ceiling, this is the right fix and there is nothing clever to do instead — you need more memory, so buy more memory.

Against a leak it is not a fix at all, and it actively makes things worse in three ways. It buys time and nothing else: doubling the heap doubles the interval between incidents, so the failure now arrives at a less convenient hour and further from the deploy that caused it. It makes the eventual failure worse, because a bigger heap means longer full GCs on the way down and a longer period of degraded latency before anything actually breaks. And it makes diagnosis harder, because your heap dump is now 8 GB instead of 2 GB, takes a minute of stopped-world time to write, needs a volume with room for it, and requires a machine with more memory than your laptop to open.

Raising -Xmx against a leak means the next person to debug it has strictly less to work with than you did.

What is not a fix

Catching OutOfMemoryError to keep the service up.

The same argument I made about NoClassDefFoundError and about StackOverflowError applies, and here it has an extra edge. Your handler needs to allocate — a log message, a formatted string, a metric — and allocation is exactly what just failed. Worse, catching it does not restore anything: the memory is still gone, the collector has already tried everything including clearing every soft reference, and the next allocation on any of your other threads fails too. What you get is a process that stays alive, fails every request, keeps passing a health check that does not allocate, and stays in the load-balancer pool doing nothing but returning 500s. A process that dies is removed and replaced. A process that limps is not.

The narrow defensible case is the same as before: an outermost worker boundary that catches Throwable, abandons the task, and lets the thread return clean — and even that only if you also alert on it, because a leak that quietly retries forever is worse than one that crashes.

System.gc() in the handler does nothing. The collector has just run, fully, and failed. What you have added is a stop-the-world pause to an already-dying process.

Switching collectors to make the symptom move is the third one. Going from G1 to ZGC or Parallel changes pause behaviour and allocation throughput, and it can genuinely shift the point at which a marginal workload falls over — which is what makes it tempting and what makes it a trap. No collector can free a reachable object. If your live set exceeds the heap, every collector fails, and the only thing you have bought is a different date and an unfamiliar set of GC logs to learn during the next incident. Choose a collector for latency and throughput goals. Never as a response to this error.

How to Prevent It

Put the heap-dump flags in the base image. Not in a runbook, not in a "we'll add it if it happens again" ticket — in the image, so that every process everywhere already has them on. They cost nothing when nothing is wrong, and the first incident without them costs you an entire repeat outage to gather what you should have had the first time. Same for GC logging with rotation. This is the highest-value item on the list and it takes twenty minutes.

Alert on the live set after a full GC, not on used heap. Used heap is a sawtooth that touches the ceiling constantly during normal operation, which makes it a terrible alerting signal — you either page constantly or set the threshold so high it fires after the failure. What you want is the floor of the sawtooth: how much is still alive once the collector has done its best. In JMX that is the G1 Old Gen pool's CollectionUsage rather than Usage, and Micrometer exposes it as jvm.memory.used tagged with the pool. Alert when that floor rises for several consecutive collections, and you find leaks days before they become incidents.

Load test with production-shaped data, which mostly means production-shaped sizes. A test fixture with 50 rows exercises none of the behaviour that matters. Take your largest real account, or synthesise one that matches it, and run the report for it — at peak concurrency, not one at a time, because one request that materialises 80 MB is fine and forty of them are not. Almost every cause-2 incident I have seen would have been caught by one test that used the ninety-ninth percentile input instead of the median.

Bound every cache at the moment you create it. Not later. A HashMap field with a comment about eviction is a scheduled outage, and the interval between writing that comment and the outage is however long it takes your traffic to grow. If it is a cache, it has a maximum size and an expiry from the first commit.

And build the review habits that catch accumulation, because these are all visible on the diff if you know what to look for. A static collection with no removal path. A findAll with no pagination. A ThreadLocal.set whose remove is not in a finally. A listener registration on a long-lived object with no matching deregistration. A domain object holding the raw payload it was parsed from. None of these are subtle once named — the same way the fail-fast behaviour in ConcurrentModificationException is obvious once you know the modCount check exists. The trick is that they all look completely reasonable if you are reading the diff for logic rather than for lifetime.

Frequently Asked Questions

Where do I find the heap dump after an OutOfMemoryError?

Only where you told the JVM to put it, and only if you asked for one before the incident. -XX:+HeapDumpOnOutOfMemoryError is off by default, so a process that dies without it leaves nothing behind but the log line. With the flag on, -XX:HeapDumpPath decides the destination: give it a directory and the JVM writes java_pid<pid>.hprof inside it, give it a filename and it writes exactly that, and give it nothing and the file lands in the process working directory, which in a container is usually a layer that disappears when the container is replaced. Point it at a mounted volume that survives a restart, make sure the filesystem has room for a file the size of your live heap, and check the JVM's own stdout — it prints the path it wrote to, or the reason it could not.

Does OutOfMemoryError always mean a memory leak?

No, and assuming it does sends you looking for a defect that is not there. There are three distinct shapes behind the same message. A genuine leak, where objects stay reachable forever and the live set climbs steadily across restarts. A working set that is simply larger than the heap, where nothing is retained wrongly but a single request or batch legitimately needs more memory than you gave the process. And a retention pattern, where a small object transitively holds a large one alive and the total is much bigger than the code implies. Two heap dumps taken several minutes apart tell them apart faster than any amount of code reading: if the live set after a full GC has grown between the dumps and keeps growing, it is a leak; if it is flat and merely high, it is capacity.

Should I just increase -Xmx?

Only after you know which of the three shapes you have, and never as the first move. Against a genuine leak, more heap buys time and nothing else: the same failure arrives later, with a bigger live set, a longer GC pause on the way down and a heap dump that is now too large to open comfortably on a laptop. Against a working set that is honestly too big, more heap is the correct fix and you should stop apologising for it. The check that decides it is the live set after a full GC over time. Flat and close to the ceiling means you need more room. Rising monotonically means you have a defect, and giving it more room only changes the date on the incident.

Why did my container get OOM-killed with no Java heap dump?

Because nothing in the JVM failed. A container OOM kill is the kernel deciding the cgroup exceeded its memory limit and sending SIGKILL to the process; the JVM gets no chance to throw anything, run a handler or write a dump, which is why the application log simply stops mid-line and the platform reports exit code 137. A Java heap OutOfMemoryError is the opposite: the collector tried, failed to free enough room under -Xmx, and the JVM threw an error it can log and dump from. The usual cause of the kill is total process memory rather than heap — heap plus metaspace plus code cache plus thread stacks plus direct byte buffers plus the native allocator all count against the cgroup limit, and an -Xmx set close to the container limit leaves nothing for the rest. Set the heap as a fraction of the limit with -XX:MaxRAMPercentage and leave real headroom for the non-heap footprint.