re-unpack-advanced

Unpacking: Strong packers (VMProtect/Themida)

When to use / when not to use

• Use: [[re-packer-id]] identifies it as a strong packer / virtualization-based packer (VMProtect, Themida, Enigma, etc.); the simple packer manual process fails (it may be wrapped in a strong packer); the sample has multiple layers of anti-debugging that block ordinary debugging.
• Use: A clean sample is needed for deep static analysis / as a prerequisite for cracking (the [[re-cracking]] path).
• Don't use: Simple compression packers ([[re-unpack-simple]] solves them in seconds; don't use the heavy process).
• Don't use: Targets where virtualized code can acceptably be black-box annotated (flagging and bypassing saves 10x the time compared with restoration; see pitfall 2).
• Note: This skill is high-cost and failure-prone—first assess the target's value and whether "annotate only, don't restore" is sufficient.

Tool preparation

Debugger (by OS)

• Debugging PE under Linux / Wine: [[re-gdb]] —— after apt install gdb / dnf install gdb / pacman -S gdb；Wine direct-read：wine sample.exe , gdb -p <pid> attach (see [[re-analyze/platform-tips]] Linux branch).
• Windows: [[re-x64dbg]] —— official release zip; x64dbg has better attach/breakpoint support for strong packers, so prefer it for strong packer scenarios.
• WSL: Cannot attach to Windows processes; use Windows-side tools (see [[re-analyze/platform-tips]] WSL branch).
• Verification: gdb --version; x64dbg can load the sample and single-step.

Scylla (IAT repair, including redirection handling)

• Built into new x64dbg versions; standalone version on GitHub NtQuery/Scylla release
• Wine environment: The standalone version of Scylla can attach to Wine processes (if it fails, complete it in a Windows environment).
• Verification: The Scylla entry appears in the x64dbg plugin menu.

scyllaHide (anti-anti-debugging, GitHub)

• GitHub x64dbg/ScyllaHide release zip; place it in the x64dbg plugin directory and enable it; the release includes a Wine loading solution.
• Purpose: Hide debugging traces in bulk (NtQueryInformationProcess, ThreadHideFromDebugger, timing differences, debug port, etc.).
• Verification: The ScyllaHide entry appears in the x64dbg plugin menu; after checking the options, it can hide the corresponding detections.

Script tools (unpacking assistance/annotation)

• idapython: Built into IDA ([[re-ida]] workflow); verification: File > Script Command inside IDA can execute Python.
• rizin script: For the installation matrix, see [[re-radare2]]; after installation, verify with rizin -v .

Operation steps

Execute in order, and record the result of each step (evidence path + sha256, see [[re-triage]]).

1. Anti-debugging countermeasures (scyllaHide approach)：

   • Core detection points:NtQueryInformationProcess（ProcessDebugPort / ProcessDebugFlags / ProcessDebugObjectHandle）、NtSetInformationThread(ThreadHideFromDebugger), timing differences (RDTSC / GetTickCount / QueryPerformanceCounter), debug port (NtQuerySystemInformation), window / process name enumeration.
   • x64dbg + ScyllaHide: Check all options in the plugin menu → start the target in hidden state. Under Wine, load using the wine solution included in the release and then attach.
   • Attack the outermost layer first: Detections are usually chained; first statically locate the outermost detection point (use [[re-ida]] / [[re-radare2]] to find the xref of NtQueryInformationProcess ), hide in bulk with scyllaHide + patch the remaining single points (see pitfall 1).

2. Find the OEP (stack backtracking / memory breakpoint combination)：

   • The entry point ≠ the original program logic entry point: The virtualization shell still has an "entry point" — after the EP is loaded, control flow enters the shell code (Themida etc. enter the VM dispatcher); this entry point is merely the shell's load/dispatch point, not the logical entry point of the original program. The claim that "virtualization shells have no concept of OEP" is inaccurate — the original entry point still exists, it is just hidden by the shell into / routed through the VM interpretation layer (see Pitfalls and Traps, "VM entry characteristics vary with build"), and locating it is the goal of this step.
   • Stack backtrace: the tail of a strong shell's stub uses ret to return to the OEP — single-step / break until stopped at ret, and the address at the top of the stack is the OEP clue (stubs are often multi-segment and multi-threaded; break several times to find the pattern).
   • Memory breakpoint combination:bp VirtualAlloc(x64dbg; for gdb/Wine use break *VirtualAlloc) → after each return, check whether the allocated region has been written with executable content → set a memory access breakpoint on that region (View > Memory Map > target section > Set breakpoint on access) → break after "the last decryption write", and the OEP is nearby.
   • OEP characteristics: function prologue (push ebp; mov ebp,esp) + dense references to normal APIs. Record the OEP address and the image base.

3. Dump (default dump preferred)：

   • Linux/Wine: after running to the OEP, gcore -o out <pid>(default dump preferred; for the full workflow see [[re-memdump]]; before dumping, filter vsyscall/vdso according to maps; for timing see [[re-analyze/platform-tips]] key experience).
   • Windows: x64dbg to the OEP → Scylla → Attach → fill in the OEP → Dump.
   • Save the sha256 before and after dumping, to facilitate comparison and verification.

4. IAT repair (Scylla, including redirection handling)：

   • Plugins > Scylla > IAT Autosearch → Get Imports → check "invalid" imports item by item.
   • Redirection handling: strong shells hijack API calls to shell stubs (jumping within the VM) — for invalid entries pointing to VM stubs: record the redirection relationship (original API → stub address), and use idapython / rizin scripts to batch-annotate the stubs back to the real APIs; mark entries that cannot be resolved and fill them in manually in subsequent analysis (first use Get Imports to confirm that most normal APIs have been resolved).
   • Import relationship recovery when IAT Autosearch is abnormal (three clues): if Autosearch finds nothing / there are many invalid entries, don't just rerun Autosearch — ① API call chain: locate the actual CALL [addr] call site, and work backward to the address table; ② Parsing functions: find the shell's API parsing loop (the GetProcAddress call site or its API hash parsing implementation), set a breakpoint at thememory write point, and record the addresses it writes into the IAT area; ③ strong shells may API hash / dynamic resolution / direct syscall(no IAT to repair) — switch to the "annotation" path (record the semantics of call sites; for the approach see Step 5).Tracing the "first API call" is not a universal method: a strong shell's resolution may occur in a multithreaded / delayed phase, or may go directly through syscalls (with no API call to trace).
   • Delayed imports / bound import / API resolver: in Scylla, check the Delayed option for delayed imports (to handle the delayed import table); for bound import, judge by the binding timestamp and replace with the real imports; for API resolver-type shells (resolved only at runtime), run to the point where the API is actually used before Get Imports, don't Fix Dump at an early stage.
   • Fix Dump Output the repaired exe; re-decompile to verify the import table ([[re-ghidra]] / [[re-ida]]).

5. Annotation of virtualized code regions (mark as bypass if not restored)：

   • Goal = reduce the complexity of the VM interpreter layer: The adversarial goal of virtualization protection is to reduce the analysis complexity of the virtual machine interpreter layer, not to "restore the virtualized function to its original code"—for VMProtect/Themida the latter has no feasible path. Choose one of two achievable goals:VM restoration(restore handler semantics and rewrite logic for a very small number of critical checks) orannotate as bypass(black-box observation of inputs and outputs). Effort is accepted based on "analyzability", not based on "degree of restoration".
   • Locate the VM entry: VMProtect's VM_Entry form push imm32; mov eax, imm; jmp vm_handler(Themida is similar, section name .vmp0/.vmp1/.themida）。
   • Annotate rather than restore: Mark the virtualized function as a black box (record the entry address, number of parameters, and list of call sites); for subsequent analysis, use dynamic observation of inputs/outputs ([[re-gdb]] / [[re-x64dbg]] break at the VM entry to record parameters and return values) instead of static restoration. Only invest in reverse-engineering the VM bytecode for the very few critical checks that truly need restoration (such as registration code verification).
   • Output:vm_blackbox.txt annotation list (address → description), archived with the analysis report.

Cross-domain combination

• [[re-anti-analysis]]: step 4 of the workflow (strong packer branch) always calls this skill; unknown packers that cannot be identified / fail the simple packer process also go through this process
• Called by the triage "sample is packed / unpack" path of [[re-analyze]]
• [[re-malware]]: strongly packed malware samples (commonly packed with VMProtect)
• [[re-cracking]]: prerequisite for cracking—for strongly packed samples, unpack first, then locate the authorization logic
• Companions: [[re-memdump]] (default dump), [[re-sandbox]] (re-run the unpacked product for verification), [[re-tracing]] (used together when anti-debugging detects the trace environment), [[re-ida]] / [[re-radare2]] (statically locate detection points and the VM entry)

Common pitfalls and traps

• Multi-layered anti-debugging chain → attack the outermost layer first: Symptom—still kicked out even with scyllaHide fully enabled, or after passing one check you are blocked by the next; Cause—the checks are chained (the inner layer is triggered only after the outer layer is passed); Countermeasure—attack layer by layer: first statically locate the outermost detection point (xref NtQueryInformationProcess / timing functions), use scyllaHide to batch-hide common items, then continue after patching the remaining single points
• High cost of restoring VM code → mark rather than restore: Symptom—stuck reverse-engineering VMProtect bytecode for days with no progress; Cause—the virtual machine instruction set is proprietary and the restoration workload is enormous;the goal is set wrong—the original code is not an attainable product of virtualization protection; Countermeasure—Goal = reduce the complexity of the VM interpreter layer(two paths: VM restoration / annotate as bypass, see step 5), mark the virtualized regions as black boxes, dynamically observe inputs and outputs (break at the VM entry to record parameters/return values), and restore only the truly critical small sections (such as registration code verification),accept based on "analyzability" rather than "degree of restoration"
• Interference from multi-threaded packers (monitor threads): Symptom—kicked out during debugging, process self-destructs, breakpoints are changed by a thread as soon as they are set; Cause—the packer starts monitor threads to detect the debugging state; Countermeasure—suspend the extra threads at the entry (x64dbg Threads panel Suspend; gdb info threads single-threaded continue), advance only within the target thread
• Packer detects debugger environment: Symptom — the process exits as soon as a debugger is opened / self-destructs immediately; Cause — detection of debug ports, parent process name, window names (FindWindowA), timing differences; Countermeasure — ScyllaHide hiding + renaming the debugger process + statically patching the detection points first before attaching; still operate within the sandbox snapshot ([[re-sandbox]])
• Hardware breakpoints detected by the packer: Symptom — the hardware access breakpoint for the ESP law does not trigger / the process exits as soon as it breaks; Cause — newer versions of Themida and the like detect hardware debug registers (DR0-3); Countermeasure — use ScyllaHide to hide via injection (with matching HookLibraryx86.dll + InjectorCLIx86.exe configuration), or switch to memory breakpoints / TitanHide, and if that still fails, statically patch the detection points first before attaching
• VM entry features mutate with each build → do not rely on signatures alone: Symptom — locating the VM entry / dispatcher by tutorial signatures fails, or the recovered output is garbled; Cause — VMProtect's handlers mutate on every build (same opcode, different code), the dispatcher contains opaque predicates, and multi-layer VMs (VM inside VM) stack up (see the opaque predicate handling in [[re-deobfuscate]]); Countermeasure — first try automated frameworks (x64Unpack hybrid emulation / DragonSlayer symbolic execution + taint tracking), and for manual work use dynamic tracing to record the handler execution sequence, locating via "bytecode pointer register + loop dispatch" rather than fixed signatures
• Samples bearing a strong packer name do not necessarily have virtualization enabled: Symptom — DIE reports VMProtect/Themida so you run the full strong-packer workflow, when standard breakpoint techniques could actually unpack it; Cause — virtualization/anti-debugging are configuration options of the packer, and a large number of real samples (RisePro/Amadey/PrivateLoader, etc.) do not enable them and have no anti-debugging; Countermeasure — try the conventional workflow first (VirtualAlloc breakpoint + memory breakpoints + single-stepping), and only invest in high-cost recovery once you confirm a VM dispatch-style entry actually exists
• Treating the packer entry as the original entry / assuming a virtualized packer has "no entry": Symptom — because you cannot find the original logic at the packer entry (the VM dispatcher dispatch point) you conclude "a strong packer has no concept of an OEP" and give up, or you dig endlessly for the original code inside the dispatcher; Cause — confusing "packer entry" (the loading/dispatch point entered after the EP) with "original program logic entry" (the original OEP hidden by virtualization); Countermeasure — the packer entry objectively exists (Themida enters the dispatcher upon loading), so use it to set breakpoints/logging and observe initialization behavior; locate the original logic entry per step 2, and handle the virtualized parts per step 5
• Obsessing over Autosearch for IAT repair: Symptom — IAT Autosearch finds nothing / all imports are invalid, and rerunning repeatedly still yields nothing; Cause — a strong packer rebuilds the import table at runtime (API hash / dynamic resolution / direct syscalls), so there is no static table to search; Countermeasure — switch to the three clues from step 4 (call chain / resolution function / memory write points) to recover import relationships, and turn items that cannot be recovered into annotations; "trace the first API call" is not a universal method with strong packers (with multithreading / delayed resolution / direct syscalls there is no call to trace), so do not waste time on the first call
• Treating "found the OEP" as successful unpacking: Symptom — you stopped at the OEP, dumped, and repaired the IAT, yet the decompilation result is still unusable (the code section is still packer data / relocations are a mess / the exception table points to packer code / it crashes on run), and repeated unpacking attempts yield nothing; Cause — narrowing the definition of unpacking to "finding the OEP"; Countermeasure —Unpacking = restoring an analyzable state: The code section is real instructions, the import table is parseable, and the relocations / exception table / load configuration match the real image (see [[re-format-pe]] for item-by-item verification). It is only considered complete once verification passes; fix whichever item is missing.
• Anti-debugging has already run before the entry breakpoint (TLS callback preempts it): Symptom — when setting a breakpoint / single-stepping at the EP, the anti-debugging check has already triggered first (exits on startup or the state has already been changed), and the entry breakpoint never takes effect; Cause — the TLS callback (TLS Callback) executes earlier than the Entry Point, and the packer hides anti-debugging inside the TLS callback; Countermeasure — first check the PE's TLS directory ([[re-format-pe]]) to list the callback addresses, set a breakpoint at the callback entry / statically analyze the logic inside the callback. When the entry breakpoint does not trigger anti-debugging, check TLS first (see [[re-anti-analysis]] anti-debugging methodology AD18)
• Anti-debugging does not exit but instead supplies fake logic: Symptom — after bypassing the detection point the process does not exit, but returns wrong data / the algorithm result is altered / key logic is skipped / execution is delayed, and the unpacked output behaves bizarrely and is hard to pinpoint; Cause — anti-debugging may deliberately create fake logic (return wrong data / modify algorithm results / delay execution / skip key logic), which is more common than directly exiting; Countermeasure — after patching the detection point, do not directly trust the output; compare the result differences between the two paths of "detection true / detection false", first confirm how the detection result affects the control flow (exit / reduced functionality / fake data / delay) before deciding how to patch (see [[re-anti-analysis]] anti-debugging methodology AD44)
• The packer detects the VM environment (VM-specific channels): Symptom — when debugging / running the unpacking process inside a VM, the sample's behavior does not match expectations (different initialization path, some functionality not triggered, incomplete unpacked output), but no signs of anti-debugging are visible; Cause — the sample calls VM-specific channels such as the VMware backdoor / Guest Additions / vmtools to detect the virtualized environment — a high-probability virtualization detection signal; Countermeasure — xref these call sites and the CPUID vendor leaf (0x40000000), confirm how the detection result affects the control flow, and if necessary patch the return value to non-VM before proceeding with the unpacking process (see [[re-anti-analysis]] anti-virtualization methodology AV-007)
• After detecting a VM, supplies fake logic / switches branches (not just exiting): Symptom — the sample does not exit in the VM environment, but the unpacked output behaves bizarrely (fake data / skipped key logic / different payload), and its behavior is inconsistent compared with the host machine; Cause — the anti-virtualization detection result affects the execution path, and the VM environment enters a different branch — anti-virtualization is not only used for exiting, it may change the sample's behavior; Countermeasure — analyze the differences between the VM / non-VM branches, and after patching the detection point compare the outputs of the two paths before drawing conclusions; do not directly trust the unpacked output (see [[re-anti-analysis]] anti-virtualization methodology AV-008)
• IAT camouflage (whitelist API noise padding): Symptom — after unpacking the import table is "repaired normally" and decompilation is smooth, but static analysis conclusions do not match runtime behavior (not a single API in the import table is actually called); Cause — malicious code / packers inject whitelist API noise into the IAT (IAT camouflage, confirmed by 2025 research to be used to mislead static detection), making the import table look harmless; Countermeasure — xref to verify CALL [IAT] actual call sites rather than trusting the import table, use [[re-tracing]] / [[re-frida]] to compare against real runtime calls, mark decorative imports separately and exclude them before drawing conclusions
• VMP 3.8+ obfuscation strategy upgrade (instruction text features become ineffective): Symptom — all analysis plugins based on instruction text feature matching for Handler failed before 3.8; Cause — VMProtect 3.8 (refactored after source code leak) introduced five types of obfuscation: ① full instruction mutation (almost all instructions except XOR decryption instructions have variants, including multiple forms of pseudocode decryption/instruction reading/Handler address calculation); ② memory address constant encryption (wherever an operand contains a memory address constant, the constant is encrypted); ③ Handler mutation (the Handler that calculates the virtual register address is replaced); ④ first-level Handler fusion (adjacent Handlers are merged, discarding the "calculate next Handler address" code block, jumping/directly connecting); ⑤ second-level Handler fusion (e.g., the three Handlers "read addend → read augend → execute ADD" are merged, virtual stack operations are discarded, bound registers directly add eax,ecx); Countermeasure — analysis plugins should switch to semantic/data-flow matching rather than instruction text; when identifying fused Handlers, note that virtual stack operations may have been replaced by direct register connections; 3.5.1 source code disclosure ≠ 3.8+ can be copied directly (3.8 has been refactored)
• VMP 3.96 dump timing and true/false OEP determination: Symptom — dumping VMP code section as soon as it is transferred yields the packer code, not the original program; Cause — .text contains VMP's own code mixed in, and before decryption/decompression is complete, any dump is garbage; Countermeasure — usememory attribute state machineto determine the decryption stage: ciphertext RX → (VMP needs to write plaintext, so write permission is granted first) → writable during decryption (WRITECOPY/RWX, key signal) → after decryption completes, returns to RX (about to jmp OEP, this is the dump timing); VMP packer loading flow: Loader bootstrap (calculate delta, self-decrypt, walk PEB to find LoadLibrary/GetProcAddress) → environment check (anti-debug/anti-VT) → unpack original image → fix relocations → rebuild IAT (polymorphic stub) → initialize VM engine → restore memory protection attributes (VMP uses syscall 50 to restore, kernel solutions can set a breakpoint here）→ jmp OEP
• VEH + Instrumentation Callback tug-of-war dump (three-ring solution): Symptom — VMP detects debugger attachment (syscall evades hooks, detects VM, detects test mode), kernel dump is complex; Cause — attaching a debugger itself will be detected; Countermeasure — DLL injection + exception takeover, completely forcibly evading attachment: set .text to non-executable to trigger a DEP execution violation → VEH takes over (only recognizes execution violations ExceptionInformation[0]==8 and the address is within .text, at this point RIP is the true OEP); counter VMP scanning: VMP will repair .text attributes and erase the VEH callback → a watchdog thread periodically reinstalls the callback (NtSetInformationProcess 0x28 ProcessInstrumentationCallback) + Instrumentation Callback strips .text back to RW on every syscall return (tug-of-war); use LATE_ARM for arming timing:has been seen writable (decryption occurred) and is currently executable but not writable(looking only at executable will misjudge the initial ciphertext state as the timing)
• Memory snapshot directly as a file (three elements): Symptom — the dumped memory cannot be loaded by a decompiler (alignment/entry/section offsets are all messed up); Cause — memory layout differs from file layout; Countermeasure — fix the PE after dumping: ① change EntryPoint to OEP; ② FileAlignment = SectionAlignment (file alignment = memory alignment); ③ for each section, PointerToRawData = VirtualAddress, SizeOfRawData = align(VirtualSize) (raw offset directly equals virtual offset, file byte X = memory RVA X); then export the full dependency module table to facilitate IAT repair