re-deobfuscate

Deobfuscation (junk instructions / control flow flattening / string encryption)

When to use / when not to use

Use: when decompiled output shows junk instructions (disassembly fragments, identity jumps), control flow flattening (if/while all become switch dispatch), string encryption (statically only ciphertext arrays visible); need to recover the algorithm / licensing logic
Use: CTF deobfuscation challenges; samples that still have code obfuscation after unpacking (step 5 of the [[re-anti-analysis]] workflow)
Don't use: clean code (directly analyze with [[re-binary-core]])
Don't use: when you only need to bypass protections and don't care about the algorithm (dynamic patch / hook is more economical, see [[re-gdb]] / [[re-frida]])
Don't use: packer-layer obfuscation (that is the packer's decompression logic - unpack first [[re-unpack-simple]] / [[re-unpack-advanced]])
Note: deobfuscation is an iterative process - restore one layer, verify one layer; back up the original file first

Tool preparation

Decompiled output (the restoration workbench)

[[re-ghidra]]：apt install ghidra / brew install --cask ghidra(the official release zip also works), verify analyzeHeadless -help; or the decompiler view of [[re-ida]] / [[re-radare2]]. First produce decompiled code, then handle the obfuscation points

idapython (batch scripting)

Built into IDA (8.x/9.x includes Python 3); for the executable name and usage for running scripts headlessly from the command line, see [[re-ida]]
Verify: inside IDA, File > Script Command can execute Python

rizin script (batch patch / query)

For the installation matrix, see [[re-radare2]]; after installation use rizin -v to verify
For Python driver use the Rizin-side bindings rzpipe（pip install rzpipe，rizinorg/rz-pipe）

D-810 (IDA plugin, optional)

Original repo GitLab eshard/d810(GitHub mirror zhkl0228/d810, the original repo README has been noted in the fork) release, copy to the IDA plugins/ directory (requires IDA 7.5+ / Python 3.7+, source: official README)
Function: automatic control flow flattening restoration (Deobfuscate menu)
Verify: the D-810 item appears in the IDA menu

python3 (emulation / batch decryption)

apt install python3(included on most systems); as needed pip install pefile rzpipe
Verify: python3 --version

Operation steps

Execute in order, record results at each step (evidence path + sha256, see [[re-triage]]).Back up the original file before modification(see pitfall 1).

Junk instruction removal (patch NOP / jump repair)：
- Identification features: identity jumps (jz/jnz the next statement is the target),call+pop address-taking, inserted junk instructions (xor eax,eax meaningless afterward), fragments formed by the disassembler straying into junk bytes.
- Removal: patch junk instructions to NOP, repair the disrupted jump targets—-
  - rizin: wx 90 @ 0x401000(single-byte NOP); for batch use, scan and fill by script according to features
  - Ghidra: select the region, right-click Patch Instruction; IDA: Edit > Patch Program > Assemble
- Repair function boundaries: in Ghidra select the range and press C to force-mark as code / right-click Create Function; manually define segments the disassembler missed.
- After each patch step, re-disassemble to confirm there are no "undefined" instructions (see pitfall 1).
Control flow flattening identification and restoration (D-810 / manual)：
- Recognition features: a large number of basic blocks converge to a single dispatcher (switch dispatch loop), a state variable (the dispatcher's index) is rewritten at the end of each block, and the original if/while branch blocks are split into small blocks.
- Automatic: IDA + D-810 → Deobfuscate menu → Flattened code, select the flattened function and restore it with one click (falls back to manual on failure).
- Manual: first locate the dispatcher's switch variable → track its write points (at the end of each block) → connect each case target back to the original control flow according to the state variable; use idapython / rizin scripts to export the dispatcher's target table to assist with wiring.
- Finding the wrong state variable is the biggest risk(see Pitfall 2) — use D-810 automatically first; when doing it manually, first confirm that the variable really participates in dispatcher indexing.
Locating and simulating string decryption loops：
- Static location: find the ciphertext array in the data section (high entropy / no plaintext) → find functions that reference it via xref (or reference the memcpy/strcpy preceding pointer) → analyze the decryption loop (single-byte / multi-byte XOR key, table lookup, byte-by-byte transformation).
- Dynamic read (requires a sandbox, [[re-analyze/platform-tips]] highest principle): [[re-gdb]] / [[re-x64dbg]] break at the return of the decryption function,x/s read the result.
- Simulation: replicate small loops with python3 (according to the reversed algorithm and key):
```
# 例：单字节 XOR 解密
key, out = 0x7f, bytearray()
for b in open('data.bin','rb').read():
    out.append(b ^ key)
print(out[:64])
```
- Compare the dynamic and simulation results; once they match, proceed to batch processing.
Batch scripting (decrypt all strings)：
- idapython: iterate over xrefs that reference the decryption function, and after calling it write the results into comments / an output file:
```
import idaapi, idc
# 例：枚举引用 sym_decrypt 的调用点，取参数地址后 idc.get_bytes 解密并打印
```
- rizin / rzpipe：rizin -A sample -c 'axt @ sym.decrypt' list references, the script loops through decryption, export strings_decrypted.txt for [[re-ioc]] / manual analysis.
- Output: a full decrypted string table (address → plaintext), archived into the analysis record.
Before-and-after restoration comparison verification：
- sha256sum record before and after modification;objdump -d compare byte differences in the junk-code region.
- Re-disassemble to confirm there is no "undefined"/no undefined jump target (check immediately after each step; do not save it all for the end).
- Re-decompile the target function to confirm that the control flow and call relationships are reasonable; run it in a sandbox to verify consistent behavior ([[re-sandbox]], see [[re-analyze/platform-tips]] highest principle).
- Cross-confirm string decryption results with runtime verification (dynamic decrypted value == script simulation value).

Cross-domain combination

[[re-anti-analysis]]: step 5 of the workflow (deobfuscation) always invokes this skill (obfuscation still remains after unpacking)
The triage "CTF challenge" path of [[re-analyze]]: re-ctf → re-deobfuscate (deobfuscation and restoration)
[[re-ctf]]: CTF deobfuscation challenges (junk code / flattening / string decryption)
[[re-binary-core]]: deep static logic analysis (continue digging into the clean restored output with [[re-ghidra]] / [[re-ida]])
Companions: [[re-gdb]] / [[re-x64dbg]] (dynamic reading of decryption results), [[re-sandbox]] (dynamic verification sandbox), [[re-ioc]] (decrypted strings as a source of features), [[re-memdump]] (needed only when the ciphertext is in memory at runtime)

Common pitfalls and traps

Deleting the wrong instruction → corrupted control flow (back up first): Symptom — after patching, the function cannot be decompiled / jumps into garbage bytes / crashes at runtime; Cause — junk code is interleaved with real instructions, and valid instructions were mistakenly deleted; Countermeasure —Back up the original file before patching(sha256 archive), patch in small steps + re-disassemble after each step to confirm there are no undefined instructions, and start over from the backup if it gets corrupted
Wrong flattening state variable identified → chaos: Symptom—after restoration, all if/else branches are out of order and the logic is absurd; Cause—an ordinary data variable was mistaken for the dispatcher state variable; Countermeasure—first confirm that the variable really is used as the switch index (value range = number of branches, rewritten at the end of each block), prefer D-810 for automatic restoration, and when doing it manually track the variable's lifetime to verify
Encrypted strings require runtime (dynamic decryption): Symptom—static analysis cannot find the plaintext, or even the decryption function; Cause—the string is only decrypted at runtime (the decryption loop may also be in the obfuscated code); Countermeasure—set a debugger breakpoint on the decryption function to read it dynamically (inside a sandbox), or restore the decryption loop first and then emulate; for cold data (constants only decrypted at runtime), use [[re-memdump]] to retrieve it from memory
Obfuscation has more than one layer: Symptom—after restoring the flattening, you find the strings are still ciphertext, or there are junk instructions nested inside junk instructions; Cause—stacking multiple layers of obfuscation is a common configuration; Countermeasure—restore layer by layer and run step 5 verification after each layer; do not try to do it all in one go
Opaque predicate (always-true/always-false branches): Symptom—after restoration, a large number of branches appear that "look conditional but are actually constant"; the logic is bizarre after patching, and symbolic execution ([[re-angr]]) gets stuck / path explosion occurs there; Cause—obfuscators (VMProtect family / Tigress, etc.) insert always-true/always-false conditions (such as x*x-x>=0, parity identities) to expand paths and interfere with analysis; Countermeasure—after identifying them, directly determine the branch target (take the true branch for always-true, the false branch for always-false) and simplify them in bulk; eliminate opaque predicates first, then do flattening restoration and symbolic execution
Seven-step anti-CFF framework and tool selection (IDA microcode): Symptom—manually restoring CFF by trial and error based on experience; Cause—lack of a systematic process; Countermeasure—follow the seven steps: a. Find the dispatcher (BLT_2WAY block with the most predecessors) b. Find the effective blocks c. Find the state variable (the dispatcher tail is a jcond and is compared with a constant) d. Which block sets a value for the state variable e. What the state variable equals at the entry of which block f. Build the Block→Block mapping g. Change the control flow; Tool comparison: IDA's built-in value-range analysis (has shortcomings; cannot complete step e for non-standard CFF) < D810 (MicroCodeInterpreter emulates and executes the dispatcher logic, stronger, and its framework is worth learning from) < angr symbolic execution (no need to explicitly find the state variable; one-shot but heavyweight); IDA microcode API:ida_hexrays 's mblock.npred()/pred()/tail/type、mop_r/mop_S/mop_d recursively retrieve the variable
Distinguishing VM obfuscation vs flattening: Symptom—treating virtual machine obfuscation as flattening (looking for state variables/constant connections) and finding that nothing matches; Cause—the two obfuscation mechanisms are different: in flattening, the end of each basic block has a constant connecting to another block, while VM obfuscation uses bytecode + jump table dispatch (operands are encoded into instructions); Countermeasure—key identification points: the function header requestsan abnormally large stack space(the VM's context/memory, passing an uninitialized pointer directly into the VM function), the data segment has a bytecode area and a jump table, and the entry first takes 4 bytes and splits them into bit fields (opcode/register index); after identifying a VM, follow the VM restoration process
VM restoration methodology (bytecode + jump table + multi-level opcode): Symptom — VM function decompilation is incomprehensible (large numbers of parallel basic blocks + dispatch); Cause — handlers are dispatched by a bytecode-indexed jump table; Countermeasures — ① locate the bytecode region (data segment, count length/instruction width to derive instruction count, e.g. 0x2F4/4=189 instructions) ② tally opcode frequency (analyze high-frequency ones first) ③ jump table = base address + opcode×4, analyze handler by handler ④ notemulti-level opcode(when op1=47, look at op2 to decide the second-level jump table; count the opcode bit-field widths clearly, e.g. 6-bit op + 5-bit register = 4-byte instruction) ⑤ sort out the real register roles (virtual PC pointer, control variable, virtual register array base address) ⑥ control variable mechanism (jumps/calls/exits are implemented by setting the control variable; both the common block at the handler tail and the fetch head check it)
When the VM only calls external functions, no full recovery is needed: Symptom — grinding through all VM handlers to recover the encryption algorithm; Cause — the VM may merely call an external encryption function (AES/random numbers, etc.), with byte-level processing outside the VM; Countermeasures — first confirm whether the VM handler only does "fetch args/call external function/store result" — if so, recovering just the parameter passing and call sequence is enough to debug the algorithm's content, without byte-by-byte VM recovery; but note that the signature algorithm of the same algorithm family (tt family) may put the encryption inside the VM, in which case full recovery is still required
CFF misjudgment → DSVM (domain-specific virtual machine) identification: Symptom — the OLLVM detector reports a large number of CFF functions (e.g. 49), but flattening-based recovery never matches; Cause — it is actually a custom VM that borrows the VMP architecture (jump-table dispatch + bytecode interpretation) but whose instruction set is fully domain-specific (DSVM, e.g. UE4 traversal engine, hardened so); Countermeasures — track down anomalous signals: large numbers of syscalls within a function (OLLVM only alters control flow and introduces no system calls), multiple dispatchers in a single function (standard OLLVM has one dispatcher per function); four-dimensional comparison of VMP vs DSVM: processor semantics (general-purpose vs domain-specific 98%), loop structure (single-layer flat vs recursive-descent parsing), bytecode format (compact binary vs textual magic number + single-byte op), virtual stack (present vs absent); DSVM is semantic-level obfuscation — understanding "what it is doing" (object traversal/protocol parsing) matters more than "how it does it" (opcode dispatch)
PAC prologue causes missed function boundaries: Symptom — function count/boundary analysis results are too few, with a single giant function actually hiding multiple functions; Cause — ARM64 PAC (Pointer Authentication Code)-protected functions use non-traditional prologues, so detecting only by stp x29,x30 misses them; Countermeasures — extend detection to three prologue patterns, and the function count can increase greatly (238→291); PAC/BTI landing pads are instead precise markers of function boundaries and indirect jump targets — exploit them rather than bypass them
Jump table entry validation (to prevent misjudging string regions): Symptom — jump table parsing yields a bunch of "targets" pointing into data/string regions; Cause — rodata contains both the jump table (16-bit offset array) and the C++ demangler name table, and entries may point into the string region; Countermeasures — validate that each entry's target address lies within the .text segment, excluding entries that point into the string region;static jump table extraction + Unicorn dynamic validation of the dispatch flow(synthesize bytecode input such as "gs1a" to trace the execution path) combining static and dynamic is a necessary combination for VM recovery
Removal of indirect jumps/calls (BR/BLR X8): Symptom — pseudocode shows __asm { BR X8 } or v278 = v277(...)(no jump symbol), IDA control flow graph is broken (branch not recognized into the function body); cause — OLLVM indirect obfuscation: the target address is stored in a register after complex logical operations and then jumped to, so the target cannot be determined statically; countermeasure —dynamically execute to obtain the register value(set a breakpoint at BR/BLR and read X8), manually calculating the target address is too inefficient; once the target is obtained, use keypatch to uniformly patch it into a direct jump (BR X9 → B 0x153AF0）；CSEL conditional branch decomposition：X8 = (X0>0) ? X8 : X9 the ternary expression computes the two targets segment by segment, patch it into a conditional jump; multi-condition chains (EQ series) are decomposed one by one in the same way
the real instructions between CSEL and BR: symptom — after patching out CSEL, the control flow is disrupted / real instructions are missing; cause — there may be real instructions between the CSEL instruction and the BR instruction, and if you patch between CSEL and its next instruction, all instructions after the B instruction will not execute; countermeasure — the patch location must be chosen between the CSEL instruction and the BR instruction(preserving the real instructions in between), not the instruction after CSEL
New adversarial surface in the AI era: adversarial text induces AI to refuse analysis: symptom — code thrown at AI (ClaudeCode, etc.) is stripped clean in seconds, and the effectiveness of traditional obfuscation declines; cause — LLMs are too strong at reverse engineering; countermeasure (defensive perspective/research): combine adversarial text suffixes (GCG-type algorithms: an LLM is a discrete-vocabulary classifier, so continuous gradients cannot be used directly — treat gradients as heuristic signals, compute each position's replacement-token contribution to the loss function, Top-K (e.g., 256) candidates + forward verification with greedy selection; in black-box scenarios use approximate models / GA evolution / distilled proxies) with obfuscated code to induce AI to output a refusal to analyze; identify the signals that AI is analyzing and counter them (e.g., after detecting a reasoning chain, inject obfuscation guidance); mind the compliance boundary (legitimately protect intellectual property code, must not be used to hide malicious code)
Custom DSL VM (pure JS virtual machine): symptom — a large JS file is mistaken for WASM / conventional obfuscation; cause — a custom interpreter loop + custom opcodes; countermeasure — the five steps of a DSL VM: case extraction → opcode classification → constant table analysis → function tracing → export extraction; first confirm it is a pure JS interpreter before deciding on the toolchain
VM export function hiding: symptom — the export function names cannot be found in the VM file; cause — the export names are hidden by instruction encoding; countermeasure — extract the real exports from the module registration center (not exposed on the surface of the VM file)
Process ready ≠ session established: symptom — the interface returns success but the process does not start; cause — success is only session confirmation; countermeasure — judge whether the process is truly ready by a specific status code / stage flag, and do not mistake session confirmation for startup (source: reverse-skill field-journal, MIT)
Z3 solving state machine transitions hits call boundaries: symptom — when symbolic execution restores flattening, continuing to solve across calls yields wrong transitions; when the solver times out / is unknown, it still patches accordingly, disrupting the control flow; cause — a call invalidates register and memory bindings and is by default a hard barrier; "unreachable path" and "unsolved" are conflated; countermeasure — perform bounded symbolic execution on each case block, solve for the block-end state variable values, treat calls as barriers by default, and allow retention across calls only for private state explicitly proven not to escape; on failure, distinguish "proven unreachable" (can be discarded) from "timeout/unknown" (fail-closed, do not touch the edge); set a timeout upper bound for the solver, and reuse translation results and solution caches
Automated rewriting lacks a coverage threshold: Symptom — after automatic restoration and flattening, large parts of the function body become unreachable and decompilation is corrupted; Cause — the CFG is modified after too few transition edges have been solved, and the rewrite leaves no rollback trace; Countermeasure — perform an overall safety check before applying (source/target blocks valid, no self-loops, terminating instruction form matches successor count); if any condition is not met, abandon the whole thing; if the number of solved transition edges does not reach the lower bound of the number of case blocks (e.g., one third and no fewer than 3 edges), refuse the rewrite; byte-level modifications use reversible patches that preserve the original bytes (never modify the input file); complex rewrite plans are applied within transaction boundaries — snapshot first, apply step by step, and roll back the whole thing if any step is rejected
UNSAT treated as always-true/always-false and changed directly: Symptom — when the solver returns UNSAT, the predicate is concluded to be always true/always false and the edge is changed directly, or the conclusion is overturned in reverse by an opposite branch observed in simulation; Cause — the three-tier evidence semantics of universal assertions are not distinguished: a SAT model only shows that some path exists, UNSAT can only serve as corroboration, and the only thing that can refute a universal is an "observed opposite branch from concrete execution" (a counterexample); Countermeasure — a counterexample must be qualified to have veto power (covers all dependent code and read-only data, no trace truncation, no summary answering on its behalf, no permission violation, does not escape the image); when static proof of always-true is impossible, keep indirect operations and only record candidate targets; "undecidable" (depends on input/timeout/backend lacks capability) means do nothing at all; missing information is unknown, not negative evidence
Reconnaissance and transformation mixed in one channel: Symptom — transforming the whole database right away, rebuilding cached microcode for every function, treating a whole-database scan as an ordinary query, costs spiral out of control; Cause — the reconnaissance channel and the transformation channel are not separated, and costs/side effects are not graded before invocation; Countermeasure — reconnaissance uses only the read-only detection channel (no cached microcode construction, no transformation components installed) to scan the whole database for candidates, and only candidates are transformed; cost grading — pure queries have zero cost and zero side effects, detection-type operations build once per function without caching and do not write cache, only transformation-type operations modify bytes and the AST (reversible patches), and whole-database scans are both expensive and write to the database; explicit admission — only requested functions go through the transformation pipeline, and admission and transformation are two separate steps; exploratory operations are synchronously bounded, single-function, with no polling, no background threads, and no repeated whole-database scans
Detection flags treated as results report: Symptom — reporting "N obfuscations detected" while not a single byte was actually changed; repeatedly transforming the same function has no effect, or if the pseudocode does not change after transformation it is assumed to have failed; Cause — detection flags are only structural candidates, and without a matching guardable rewrite no change is produced at all; caching and tracking are not managed, so repeated transformations are idempotently blocked without anyone knowing; Countermeasure — report the actual processing count and the actually applied modifications, and do not use detection flags as results; iteration loop: detect → (optional) explore to obtain evidence → transform → re-decompile without cache → detect again/compare pseudocode to confirm convergence; to rerun, function-level tracking and evidence must first be cleared; handle failures by return value (numeric returns 0/-1, string returns empty string), not by catching exceptions; before transformation, confirm that automatic analysis has completed and the decompiler is available
Simulation evidence out of bounds treated as universal evidence：Phenomenon—when the simulation engine observes the opposite branch, it uses it as a counterexample; after external calls are answered by summaries on their behalf, they still support universal conclusions, and when the function has changed, it still speaks using old observations. Cause—simulation evidence has boundaries (a single function is bounded, and the number of runs is finite); the semantics of answering on behalf of a summary is partial compensation by the host model; evidence carries lineage but is not verified before consumption. Countermeasures—simulation should only perform bounded single-function work (mirror snapshots serve only as in-memory context); external calls may be answered by summaries on their behalf but must be marked as exploratory and not used for universal proof; unmodeled targets are reported as "environment boundaries" rather than decoding failures; each observation carries lineage (library, function byte hash, mirror hash, function generation, seed), and before consumption the function block and trace interval must be compared byte by byte against the current library, discarding stale ones; dynamic observations have a minimum run-count consumption threshold (e.g., 2 different runs); missing backend capabilities are reported as unknown rather than negative evidence