re-crash-triage

Crash analysis and vulnerability localization (crash triage)

When to use / when not to use

Use: fuzzing produces a crashing input; the program segfaults/aborts; you need to determine whether the crash is a real vulnerability (reachability) and produce a minimal PoC; multi-threaded/non-deterministic crashes need to be reproduced
Don't use: it hasn't crashed yet, and you want to actively find bugs (→ [[re-fuzzing]]); the crash has already been localized, and you need to dig deep into function logic (→ [[re-binary-core]] foundation)
Don't use: you only need to look at the symbol stack (going directly to [[re-gdb]] is also fine, but this skill contains the complete "reproduce→minimize→localize→PoC" workflow)

Tool preparation

Verify every tool before using it. Reproducing a crash is dynamic execution, which by default runs inside a sandbox ([[re-analyze/platform-tips]] highest principle, see [[re-sandbox]]).

ASAN / UBSAN — memory and undefined behavior detection (built into clang/gcc)

clang: sudo apt install clang / sudo dnf install clang / sudo pacman -S clang / bundled with macOS Xcode
gcc: Debian/Ubuntu sudo apt install gcc(ASAN is built into gcc, no separate package); Fedora sudo dnf install gcc；Arch sudo pacman -S gcc
Compile: gcc -fsanitize=address,undefined -g -fno-omit-frame-pointer -o target_asan target.c(clang parameters are the same)
Verify: write a small program with an array out-of-bounds and compile and run it, confirming the output contains AddressSanitizer text

gdb — backtrace and core analysis

Debian/Ubuntu: sudo apt install gdb
Fedora/RHEL: sudo dnf install gdb
Arch: sudo pacman -S gdb
macOS: brew install gdb(requires codesign authorization); Xcode's bundled lldb also works (see [[re-lldb]])
Windows: Linux version inside WSL; on native Windows use WinDbg ([[re-windbg]])
Verify: gdb --version

rr — record/replay (non-deterministic crashes)

Debian/Ubuntu: sudo apt install rr(universe repository)
Fedora/RHEL: sudo dnf install rr
Arch: paru -S rr(AUR package; no rr in the official repositories)
macOS: not supported (rr only supports Linux; the official wording is "Apple hardware is fine, Apple software is not"); Apple Silicon users should use Asahi Linux or a Linux virtual machine (requires PMU passthrough)
Windows: Linux version inside WSL2

Build from source:

git clone https://github.com/rr-debugger/rr && mkdir rr/obj && cd rr/obj
cmake -DCMAKE_BUILD_TYPE=Release ../rr && make -j$(nproc) && sudo make install

Prerequisite: Linux kernel 3.11+ (4.7+ recommended);perf_event_paranoid > 1 is extremely slow, so relax it first:
```
sudo sysctl kernel.perf_event_paranoid=1
```
Inside a virtual machine PMU passthrough is required (VMware/KVM works, VirtualBox does not)
Self-check first in containers/restricted environments: rr check checks PMU/perf_event availability and known limitations; use it only after it passes; rr cannot record in nested virtualization or in containers without PMU passthrough
Fallback when PMU is unavailable(don't get stuck on rr): instead use "rerun the same input multiple times + logs/conditional breakpoints" to narrow the race window; for data-race-type issues use valgrind --tool=helgrind or -fsanitize=thread builds to localize; for pure ASLR jitter you can first set disable-randomization on to rule it out
Verify: rr --version

afl-tmin / afl-cmin — input minimization and deduplication (installed with AFL++)

For installation see the AFL++ section of [[re-fuzzing]] (afl-tmin / afl-cmin installed with the package)
Prerequisite (afl-cmin especially important): afl-cmin requires the target to be observable by AFL—usually requiring it to be built with AFL instrumentation (afl-clang-fast / afl-gcc), otherwise it will report "no instrumentation / not instrumented" and fail.afl-tmin has much weaker constraints (it can work on a pure crash criterion), and usually can run even on uninstrumented targets
Common pitfalls: The target is a standalone ASAN build (gcc -fsanitize=address -g) anddoes notgo through afl-clang-fast when,afl-cmin it is not guaranteed to run. First confirm the build method; when it is not produced by the AFL compiler, switch to afl-tmin to minimize one by one, or go back to [[re-fuzzing]] to rebuild the instrumented version
Verification: afl-tmin -h / afl-cmin -h

Operating steps

Execute in order; first preserve the crash input and PoC with sha256 as evidence ([[re-triage]] method).

Deterministic reproduction：
```
sha256sum crash_input > crash_input.sha256
./target < crash_input          # stdin 输入
./target crash_input            # 文件输入
./target -f crash_input         # 按目标实际参数形态
```
- Crashing 3 times in a row = deterministic crash, continue to step 2; crashing sometimes but not others = non-deterministic (multithreading/uninitialized/ASLR), skip to step 4 and use rr
- core dump enabled (leave a core after the crash for analysis): ulimit -c unlimited；cat /proc/sys/kernel/core_pattern If it points to systemd-coredump/apport, the core will be collected by the system (see pitfall 4)
ASAN report interpretation(the target must be an -fsanitize=address -g -fno-omit-frame-pointer build, see tool preparation):
- heap-buffer-overflow / stack-buffer-overflow / global-buffer-overflow: out-of-bounds read/write — the report includes the out-of-bounds direction (READ of size N / WRITE of size N) and the out-of-bounds offset
- heap-use-after-free: UAF — look at freed by and allocated by the two stack traces, find the free point and the reuse point
- attempting double-free: double free
- SEGV: null pointer/wild pointer access, details require gdb to supplement (step 4)
- Key:#0 frame is the crash point,don't just look at the line number—combine with the data flow to determine which input byte caused it
Input minimization：
```
afl-tmin -i crash_input -o crash.min -- ./target @@
# 多个崩溃输入先按路径去重（需 AFL 插桩，见工具准备）：
afl-cmin -i crashes_dir -o minimized -- ./target @@
```
- @@ only applies to "file argument" targets(AFL replaces @@ with the input file path). The target is stdin when reading input, do not use @@, add the "file → stdin" step into the command line, for example:
```
afl-tmin -i crash_input -o crash.min -- sh -c './target < "$1"' _ @@
```
  or when the target has a built-in -f <file> file argument like that, use it instead of @@; first choose the syntax according to the real input form confirmed in step 1
- afl-tmin produces the shortest input that still crashes; you can further manually delete head/tail bytes to verify (binary search for the minimum boundary)
- The minimized input is the PoC draft; keeping the crash type the same = the basis for judging it is the same bug

gdb backtrace localization：

gdb -q ./target
(gdb) set disable-randomization on     # 关 ASLR 便于地址对照
(gdb) run crash.min
(gdb) bt                                # 完整回溯
(gdb) info registers                    # 崩溃点寄存器
(gdb) x/16i $pc                         # 崩溃点反汇编

core file analysis:gdb -q ./target /path/to/core
bt are all ??(symbol-less): record the module base address + offset, go into [[re-ghidra]] / [[re-ida]] decompilation to supplement frames (see pitfall 5)

Non-deterministic crash → rr record and replay:

rr record ./target crash_input
rr replay                            # 打开 gdb 会话
(rr) continue                         # 跑回崩溃点
(rr) bt
(rr) reverse-continue                 # 崩溃前反向执行找根因

PoC / Report：
- PoC：crash.min + sha256 + reproduction command (environment: ASAN build parameters, run parameters, input channel)
- Report points: crash type (ASAN category), crash point function/line/offset, data flow (which input byte → out-of-bounds index), root cause (missing length check / unvalidated index / use after free),reachability(see pitfall 3: the crash path is truly reachable and the input is controllable), impact assessment (out-of-bounds read → information disclosure, out-of-bounds write → RCE, pure crash → DoS), remediation suggestions
- Output: PoC hash and crash characteristics can go into [[re-ioc]]

Cross-domain combination

[[re-vuln]]: step 3 of this gateway workflow fixedly calls this skill (crash analysis is a core part of the vulnerability discovery chain)
[[re-fuzzing]]: its crash output (out/crashes/) is handed to this skill for analysis (this skill also uses its afl-tmin/cmin toolchain in reverse)
[[re-binary-core]]: after locating, adding symbols/reading function logic goes through the base ([[re-ghidra]] / [[re-ida]] / [[re-radare2]]); [[re-triage]] provides evidence preservation methods
[[re-memdump]]: when the process is already dead / a core file already exists and more context is needed, switch to dump analysis (crash core → memdump)
[[re-sandbox]] / [[re-analyze/platform-tips]]: reproduction is dynamic execution, run inside a sandbox by default

Common pitfalls and traps

Non-deterministic crashes (multithreading): symptom—the same input crashes 3 out of 10 times, and gdb reruns can't reproduce it; cause—thread races / uninitialized memory / ASLR address differences; countermeasure—rr recording (rr record ./target input) automatically stops at the crash point,rr replay + reverse-continue reverse-locates the root cause; first set disable-randomization on to rule out the ASLR factor
ASAN vs. optimization differences: symptom—ASAN (-O0/-O1) crashes but release (-O2) doesn't, or conversely only release crashes; cause—optimization changes memory layout/timing, masking or exposing undefined behavior; countermeasure—try both builds; add -fsanitize=undefined to catch UB; impact and reachability judgments should be based on real release behavior (what ASAN reports is a potential vulnerability and needs confirmation)
A crash ≠ a vulnerability (reachability): symptom—ASAN reports an out-of-bounds access but it's in dead code/an unreachable branch, or the crash is introduced by the harness; cause—the input triggered by the fuzzer doesn't necessarily correspond to a real attack surface; countermeasure—gdb backtrace + decompilation to confirm the crash path is reachable and the input is controllable; distinguish harness bugs from target bugs ([[re-vuln]] workflow step 4 for the final confirmation)
core is collected by the system: symptom—ulimit -c unlimited after that, the crash still has no core file; cause—core_pattern points to systemd-coredump (pipe method)/apport interception; countermeasure—first cat /proc/sys/kernel/core_pattern confirm the destination, then retrieve it according to the destination:
- core_pattern looks like |/usr/lib/systemd/systemd-coredump ...: the core has been taken over by systemd,don't rush to change global settings, just use it directly—coredumpctl list find the entry (by executable name/time),coredumpctl info <PID|可执行名> view metadata,coredumpctl dump <PID> -o core export, or coredumpctl gdb <PID> go straight into gdb for debugging. This is more complete than changing core_pattern , and it doesn't affect the core behavior of other programs on the system
- apport takeover (Ubuntu): the core lands in /var/crash/, use apport-unpack to unpack
- When you really need the raw core file, temporarily change it:sudo sysctl kernel.core_pattern=core(the core lands in the process working directory; note it only takes effect for processes started afterward)
- Fallback:gdb --args ./target crash.min run directly, without relying on a core file
Missing symbols: symptom—bt all ??, addresses don't match; cause—strip / static linking / the binary and core versions are inconsistent; countermeasure—the core and binary must be the same version (compare sha256); record the crash module base address and offset, locate the function by offset in [[re-ghidra]] / [[re-ida]], and convert back to the gdb address