java-heap-dump-triage

Java Heap Dump Troubleshooting

Goal

Turn heap dump evidence into actionable engineering diagnosis. Prioritize giving narrow root causes supported by evidence; do not generically list JVM tuning suggestions.

Troubleshooting process

Locate analysis materials.
- Search .hprof、*Leak_Suspects*.zip、*System_Overview*.zip、*Top_Components*.zip、.threads、.index, MAT-exported HTML reports.
- Record the dump size and report generation time.
- If MAT's System Properties are available, record the application name, startup class, profile, JDK version, service port, and module information in the classpath.
- If there is only the raw .hprof, do not stop; first generate a report with the Eclipse MAT command line.

If there is no MAT report, generate a report from .hprof .
- Find the MAT command-line tool, for example ParseHeapDump.sh or MemoryAnalyzer。
- Do not delete the original .hprof; if you want to simulate a no-report state, only move or clean up derived files such as .index、.threads、*_Leak_Suspects.zip、*_System_Overview.zip、*_Top_Components.zip .
- Use ParseHeapDump.sh <dump.hprof> org.eclipse.mat.api:suspects org.eclipse.mat.api:overview org.eclipse.mat.api:top_components to generate three types of reports.
- The expected output is usually in the same directory as the dump with the same prefix:
  - <name>_Leak_Suspects.zip
  - <name>_System_Overview.zip
  - <name>_Top_Components.zip
  - <name>.threads
  - <name>*.index
- When parsing a large dump, there is a lot of output, and the logs may be truncated; rely on the final exit code and generated artifacts.
- Key stages include: Parsing heap dump, Writing threads, Processing object reachability, Calculating Dominator Tree, Leak Suspects, System Overview, Top Components.
Prefer reading the MAT report rather than starting by struggling through the raw dump.
- Use unzip -l to view the contents of the report zip.
- Extract to a unique temporary directory, and do not delete user files.
- Focus on parsing index.html、pages/Class_Histogram*.html、pages/Top_Consumers*.html、pages/Thread_Overview*.html、pages/System_Properties*.html, and the Problem Suspect pages.
- After extracting text from the HTML, summarize top retained objects, class histogram, shortest paths to GC roots, thread stacks, and JVM/application properties.
Find the real holders of retained memory.
- Prioritize retained heap and the dominator tree; do not be misled by shallow heap.
- Identify one to a few main accumulation points, such as cache/map, scheduler/job store, queue, ThreadLocal, classloader, ORM/session, HTTP buffer, byte array, String, reactive queue.
- Record specific numbers: retained bytes, proportion, object count, top classes, shortest path fields.
Correlate the class names in the dump back to the code.
- Use rg to search class names, package names, framework types, key fields, job/cache/listener/scheduler APIs, and configuration items.
- Read the code paths that create, refresh, delete, and retain suspicious objects.
- Check whether lifecycles are symmetric: add/remove, subscribe/unsubscribe, schedule/delete, cache put/evict, ThreadLocal set/remove, close/shutdown.
- Check whether identity identifiers are symmetric: key name, group/namespace, task id, project id, hash/md5, tenant, channel, shard.
- Check whether startup or reload logic repeatedly registers without cleanup.
Form a falsifiable root cause.
- Bind the dump evidence to specific code behavior.
- Explain why the object grows and why GC cannot reclaim it.
- Distinguish the direct cause from long-term risk. For example:RAMJobStore holding the trigger is direct evidence; large-scale use of RAMJobStore is a scalability risk;deleteJob using the wrong group is the more specific code defect.
Provide a fix.
- If a clear bug is found, the first fix should be as small and precise as possible.
- Write clearly the file, function, behavior before the change, and behavior after the change.
- Add logs or metrics only when they help verification.
- Treat structural risks as second-phase optimizations; do not mix them into the immediate fix.
Verification.
- Run the narrowest possible compilation or tests for the changed module.
- When tests can be added, prioritize unit/integration tests for lifecycle symmetry.
- Online verification should include restart, re-dump, metrics, and log checks.
Output conclusions.
- Give the root cause and fix first.
- Attach a table of key MAT numerical evidence.
- Attach the code path and problem points.
- If the object is kept in process memory, clearly state whether a restart is needed after release.
- When evidence is insufficient, mark it as an "assumption" or "needs further verification"; do not overstate.

Common commands

Find heap-related files:

find ~/Desktop -maxdepth 3 \( -iname '*.hprof' -o -iname '*dump*' -o -iname '*heap*' -o -iname '*Leak_Suspects*.zip' -o -iname '*System_Overview*.zip' -o -iname '*Top_Components*.zip' \) -print

Find the local Eclipse MAT command line:

command -v ParseHeapDump.sh || true
command -v MemoryAnalyzer || true
find /Applications "$HOME" -maxdepth 5 \( -iname 'ParseHeapDump.sh' -o -iname 'MemoryAnalyzer' \) -print 2>/dev/null

Generate a MAT report from the raw hprof:

/Applications/MemoryAnalyzer.app/Contents/Eclipse/ParseHeapDump.sh \
  /path/to/heap_dump.hprof \
  org.eclipse.mat.api:suspects \
  org.eclipse.mat.api:overview \
  org.eclipse.mat.api:top_components

After generation, confirm the artifacts:

find /path/to/dump-dir -maxdepth 1 \( \
  -name 'heap_dump*.index' \
  -o -name 'heap_dump.threads' \
  -o -name 'heap_dump_*Suspects.zip' \
  -o -name 'heap_dump_*Overview.zip' \
  -o -name 'heap_dump_*Components.zip' \
\) -print | sort

View the MAT report zip:

unzip -l /path/to/heap_dump_Leak_Suspects.zip | sed -n '1,160p'

Safely extract MAT HTML text:

TMP=/tmp/heap_report_$$
mkdir -p "$TMP/leak" "$TMP/overview" "$TMP/top"
unzip -q /path/to/heap_dump_Leak_Suspects.zip -d "$TMP/leak"
unzip -q /path/to/heap_dump_System_Overview.zip -d "$TMP/overview"
unzip -q /path/to/heap_dump_Top_Components.zip -d "$TMP/top"
python3 - "$TMP" <<'PY'
from pathlib import Path
import html, re, sys
base = Path(sys.argv[1])
for p in base.rglob("*.html"):
    if p.name.startswith(("Class_Histogram", "Top_Consumers", "Thread_Overview", "System_Properties")) or p.name == "index.html":
        s = p.read_text(errors="ignore")
        s = re.sub(r"<script.*?</script>|<style.*?</style>", "", s, flags=re.S|re.I)
        s = re.sub(r"<[^>]+>", "\n", s)
        lines = [html.unescape(x.strip()) for x in s.splitlines() if x.strip()]
        print(f"\n===== {p.relative_to(base)} =====")
        print("\n".join(lines[:180]))
PY

Look up code by suspicious classes and APIs:

rg -n "Quartz|RAMJobStore|CronTrigger|scheduleJob|deleteJob|ThreadLocal|Cache|put\\(|evict|shutdown|close\\(" src -g '*.kt' -g '*.java' -g '*.yml' -g '*.yaml' -g '*.properties'

Record of the actual generation process

This skill was distilled from an actual process of regenerating a report from a raw dump:

Keep only the raw input /Users/greysonfang/Desktop/heap_dump.hprof, size about 2.6G。
First move away existing MAT derivatives, including heap_dump*.index、heap_dump.threads、heap_dump_Leak_Suspects.zip、heap_dump_System_Overview.zip、heap_dump_Top_Components.zip。
Find the local MAT command:/Applications/MemoryAnalyzer.app/Contents/Eclipse/ParseHeapDump.sh。
Execute:

/Applications/MemoryAnalyzer.app/Contents/Eclipse/ParseHeapDump.sh \
  /Users/greysonfang/Desktop/heap_dump.hprof \
  org.eclipse.mat.api:suspects \
  org.eclipse.mat.api:overview \
  org.eclipse.mat.api:top_components

MAT first parses the hprof, generates .threads and indexes, then computes the Dominator Tree, and finally outputs three report zips.
The size of the report generated this time is approximately:
- heap_dump_Leak_Suspects.zip: 168K
- heap_dump_System_Overview.zip: 140K
- heap_dump_Top_Components.zip: 772K
Top Components There may be an extremely large number of stage logs; truncated terminal output is normal; as long as the process exit code is 0 and the report zip exists, analysis can continue.

Diagnostic experience

If an object occupies an extremely high retained heap, the occupancy path should start from its dominator path.
If the Class Histogram has a large number of collection entries, first find the upper-level collection holding these entries; do not optimize the entry type itself.
If the framework's internal store accounts for most of it, focus on how the business code uses this framework's lifecycle API.
If the code has deletion logic but objects still accumulate, first compare whether the key/group/namespace used at creation, check, and deletion are exactly the same.
If the dump shows many threads but the retained heap is mainly elsewhere, do not treat the thread count as the root cause unless ThreadLocal or thread stacks directly retain large objects.
If System Properties show the application name, profile, and classpath, use them to locate the specific service module.

Output template

Unless the user requests another format, output using the following structure:

# Java Heap Dump 排查结论

## 现象
- Dump / MAT 报告来源：
- 主要 retained heap：
- 关键对象数量：

## 根因
一句话结论。

## 证据链
| 证据 | 数值/位置 | 说明 |
| --- | --- | --- |

## 代码定位
- 文件：
- 调用链：
- 问题点：

## 修复方案
1. 立即修复：
2. 验证：
3. 上线/重启：

## 长期优化
- 监控：
- 架构/存储：
- 测试：

Case reference

If Quartz appears in the MAT RAMJobStore、QuartzSchedulerResources、CronTriggerImpl、CronExpression、TriggerWrapper、TreeMap$Entry or JobKey occupies the majority of the retained heap, read references/quartz-ramjobstore.md。