doris-debug-node

Node Health

Causes

ID	Cause	Evidence	Source anchor
A	BE OOM / killed	`dmesg` shows `Killed`; `be.WARNING` MemTrackerLimiter	`mem_tracker_limiter.cpp`
B	FE JVM OOM	GC overhead / heap dump; `fe.out` shows `OutOfMemoryError`	`fe.conf` JVM opts
C	BE crash (SIGSEGV/SIGABRT)	Core dump; `be.out` stacktrace	`doris_main.cpp` signal handler
D	False Alive	`SHOW BACKENDS` Alive=true but queries fail brpc	`heartbeat` vs brpc health
E	FD exhaustion	`Too many open files` in be.WARNING	`ulimit -n`

10 min triage

# Memory pressure signals
./scripts/doris-debug log-grep be/log --pack memory
dmesg -T | grep -i "killed process" | tail -5

FE JVM

jstack $(cat /path/to/fe.pid) > /tmp/fe.jstack jmap -heap $(cat /path/to/fe.pid) 2>/dev/null | head -30

FD count

ls /proc/$(cat /path/to/be.pid)/fd | wc -l cat /proc/$(cat /path/to/be.pid)/limits | grep "open files"

Resource snapshot

top -b -n 1 -p $(cat /path/to/be.pid) | head -5 free -h


## Cause A — BE OOM

Doris tracks memory via MemTracker, but process RSS exceeds tracked memory due to:
- jemalloc arena fragmentation (10-20% overhead)
- brpc buffer pools (outside MemTracker)
- Thread stacks (~8MB each × thread count)

```bash
# RSS vs tracked
ps -o rss= -p $BE_PID | awk '{printf "RSS: %.1f GB\n", $1/1024/1024}'
grep "MemTrackerLimiter.*total" be/log/be.WARNING | tail -1

# be.conf — defensive settings
mem_limit = 70%              # Leave 30% for overhead
max_segment_cache_size = 0   # Or cap explicitly
enable_je_purge = true       # jemalloc dirty page cleanup

If ProcessMemoryReachedCancel didn't fire but OOM killer did: the gap between mem_limit and system RAM wasn't enough for jemalloc + brpc overhead.

Cause B — FE JVM

# FE JVM opts (from fe.conf or start script)
ps aux | grep "PaloFe" | grep -o "\-Xmx[^ ]*"

Common: FE heap grows with catalog metadata (many tables/partitions). Raise -Xmx or reduce hive_meta_cache_ttl_sec / catalog refresh frequency.

Cause D — False Alive

Alive=true means the heartbeat RPC succeeds. This does NOT mean the brpc port (8060) is healthy for data transfer. If queries fail with E1008 while Alive=true:

→ Route to doris-debug-query (Cause D — Exchange / brpc) → Check priority_networks, enable_brpc_connection_check, reset_rpc_channel → Do not trust SHOW BACKENDS Alive alone

Source

be/src/runtime/memory/mem_tracker_limiter.cpp
be/src/service/doris_main.cpp — jemalloc init, signal handler
fe/.../master/HeartbeatHandler.java
be/src/common/config.cpp — mem_limit, enable_je_purge, max_segment_cache_size