A JVM search service uses more and more memory but shows no obvious leak. How do you investigate?

"Memory keeps growing" is three different problems that look the same on a dashboard. A good answer separates them before touching any flag.

Signal one: the live set after GC. Look at heap occupancy right after old-generation or full collections over hours or days, not the sawtooth in between. A heap that climbs between collections but returns to the same floor afterwards is normal; the collector is simply lazy until it needs to work. A floor that keeps rising means objects are being retained. Oracle's troubleshooting guide makes the same point: an OutOfMemoryError can mean a leak, but it can also mean the heap, Metaspace or compressed class space is sized smaller than the application needs, so check that "before assuming a memory leak".

Signal two: the allocation rate. A high allocation rate with a flat live set is a garbage problem, not a leak. It costs CPU and GC frequency rather than memory. JDK Flight Recorder's allocation events show which code paths allocate most; typical culprits in a search service are building large intermediate collections per request, boxing in hot loops, or re-parsing the same JSON.

Signal three: process or container memory. A container is killed when the whole process exceeds its memory limit, and that limit counts far more than the heap. If the heap floor is flat but the container's resident memory grows, the growth is outside the heap: thread stacks, Metaspace, the code cache, direct ByteBuffers, memory-mapped files or a native library. Native Memory Tracking breaks JVM-managed native memory into categories. Enable it with -XX:NativeMemoryTracking=summary, take a jcmd <pid> VM.native_memory baseline, then compare with summary.diff later. Oracle documents a 5–10% performance overhead for NMT, so plan it as a diagnostic window, not a permanent setting. Memory-mapped index files are a special case for an Elasticsearch node: the OS page cache is meant to use spare memory, which is why the heap should not take all of it.

When the live set really grows. Use Flight Recorder's Old Object Sample events, which record allocation time, the allocation stack trace and the path back to a GC root. Oracle describes JFR's overhead as very low and designed to be safe for production, which is why it is the first tool to reach for there. Then take two or three heap dumps spaced apart (jcmd <pid> GC.heap_dump) and compare the dominator trees; the class whose retained size keeps rising is the lead. Common causes are unbounded caches, listeners never removed, ThreadLocal values left in pooled threads, and static collections keyed by request data.

What not to do. Do not raise -Xmx first; it hides the trend and makes the eventual dump larger and slower. Do not use desktop tools such as VisualVM against a production pod when JFR and jcmd give the same evidence with less risk.

Alternate interview scenarios