SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsre-unpack-simple

re-unpack-simple

General & other

View repository

Updated 10-15 21:35

About this skill

Unpacking: compressed packers (UPX/ASPack/FSG)

When to use / when not to use

  • Use: [[re-packer-id]] identifies it as a simple compressed packer (UPX/ASPack/FSG, etc.); the packer entry has pushad/pusha signature; a clean sample is needed to continue static/dynamic analysis
  • Use: first attempt in the manual workflow for an unknown packer (the ESP law works for most compressed packers)
  • Do not use: strong packers / virtualization packers (VMProtect/Themida, go to [[re-unpack-advanced]])
  • Do not use: the target is confirmed to be unpacked (directly go to [[re-binary-core]])
  • Do not use: when the packed behavior itself needs to be observed (behavior analysis sometimes deliberately keeps the packer; evaluate in conjunction with [[re-behavior]])
  • Note: verification of unpacking output must be done in a sandbox ([[re-sandbox]], highest principle of [[re-analyze/platform-tips]])

Tool preparation

upx (the main force for official unpacking)

  • Linux: apt install upx-ucl / dnf install upx / pacman -S upx
  • macOS: brew install upx
  • Windows: official GitHub release upx/upx download the win64 zip (or choco install upx）
  • WSL: Linux package
  • Verify: upx --version

Debugger (by OS)

  • Debugging PE under Linux / Wine: [[re-gdb]] —— apt install gdb / dnf install gdb / pacman -S gdb, verify gdb --version。Wine direct read：wine sample.exe after running gdb -p <pid> attach directly to the Wine process (see the Linux branch of [[re-analyze/platform-tips]])
  • Windows: [[re-x64dbg]] —— official release zip, verify: loading the sample allows single-stepping
  • macOS: for PE unpacking scenarios use Wine + [[re-gdb]] or a Windows VM, do not use the native debugger
  • WSL: cannot attach to Windows processes; for cross-boundary cases use Windows-side tools (WSL branch of [[re-analyze/platform-tips]])

Scylla (Windows IAT repair)

  • Built into the new official release of x64dbg (Scylla is under the plugins/ directory)
  • Standalone version: GitHub NtQuery/Scylla release
  • Wine environment: the standalone Scylla can attach to the Wine process for dump repair (if it fails, complete the IAT repair in a Windows environment)
  • Verify: Scylla appears in the x64dbg plugin menu, and can Attach to the process and list modules

Operation steps

Execute in order, recording the result of each step (evidence path + sha256, see [[re-triage]]);before starting, record the unpacking baseline five-piece set according to [[re-triage]](SHA256 / PE timestamp / EP RVA / section hash / import table), and after unpacking compare item by item before judging success or failure.

  1. Prefer official / automatic unpacking (upx -d）：

    upx -d sample.exe -o unpacked.exe
file unpacked.exe     # 应显示正常 PE 结构；仍报 UPX / 报错 → 失败
sha256sum unpacked.exe

    upx -d common reasons for failure: the file has been modified (self-check / secondary packing), UPX version incompatibility. If it fails, go to the manual workflow in step 2.

  2. Find the OEP using the ESP law(set a breakpoint after pushad):

    • Principle: the compressed packer entry pushad saves registers → after the packer finishes decrypting popad → jmp OEP。set a hardware access breakpoint on the address pointed to by ESP after pushad, and after running, look for the OEP near the first instruction after popad .
    • Essence (a technique, not an unpacking formula): the entry pushad push onto the stack is the packer's inevitable behavior of "saving the scene"; the ESP law uses it to locate the "restore the scene" point——popad Once restored, decryption is complete and control flow will jump to the real code. This holds only under the implementation of "push onto the stack at the entry to save the context, then restore at the tail and jump"; it is not effective against all packers: TLS callback packers (the callback executes before the EP, and there is no saved-stack context at the EP), multithreaded packers (multiple threads each push onto the stack, and the stack address hit by the breakpoint is misaligned), and virtualization packers such as VMProtect/Themida (the entry goes directly into the VM dispatcher, with no pushad/popad cycle) are all inapplicable.First confirm that the EP entry is indeed a pushad/popad cycle before using it, otherwise go directly to step 3 or switch to [[re-unpack-advanced]].
    • x64dbg (Windows): load → stop at the EP → single-step and see pushad → then single-step to execute pushad → record the after pushad executesESP value → right-click the ESP register → Hardware Breakpoint (on access) → run → break at the popad first instruction after → the target of the jmp in the next few lines is the OEP. Write down the OEP address.
    • gdb (Linux/Wine, see [[re-analyze/platform-tips]] Wine direct read): 
      (gdb) starti                    # 停在 EP
(gdb) si                        # 单步执行 pushad（x86 pushad 压 8 个寄存器）
(gdb) awatch *(int*)$esp        # 对 pushad 后的 ESP 所指地址下硬件观察点（访问即断）
(gdb) continue                  # 运行，popad 访问该地址时触发
    • OEP characteristics: a function prologue (push ebp; mov ebp,esp), followed immediately by a large number of normal API references.

  3. Memory breakpoint method (VirtualAlloc breakpoint)：

    • Applicable: when the ESP law fails (no pushad, multiple stubs, multi-stage decryption).
    • Approach (set execution monitoring on the decrypted region): the packer writes the fully decrypted real code into some region (the decompression target section / a VirtualAlloc-allocated region); set execution/access monitoring on that region, and when the packer's control flow first lands on the decrypted code, it breaks—that is, theentry point of the real code after decryption is complete, with no need to guess "which address the OEP is at".
    • "The first execution is the OEP" does not hold: some packers first jump to a loader (the packer's own runtime library), use multi-stage decryption (what is executed the first time is still an intermediate layer, with more decryption loops afterward), or set up a fake OEP (a disguised function prologue to lure an early dump)—after breaking you must verify that the target has real code characteristics (a function prologue push ebp; mov ebp,esp + a subsequent dense set of normal API references); for multi-stage packers, repeatedly set breakpoints until control flow stably leaves the packer code, then proceed to step 4.
    • x64dbg：bp VirtualAlloc → after each return, check whether the allocated region has been written with executable content (the packer's decompression target section) → set a memory access breakpoint on that region (View > Memory Map > target section > Set breakpoint on access) → run until the point where the packer finishes decryption. You can also directly set a memory breakpoint on the packer's second section (UPX1/.aspack).
    • gdb（Wine）：break *VirtualAlloc, after returning check the allocated region, then set a watch on the allocated address.

  4. Dump the OEP(timing: after decryption is complete, see [[re-analyze/platform-tips]] key experience; dumping is preferred by default):

    • Linux/Wine: after running to the OEP (decryption complete), dump by default with gcore -o out <pid>(for the full process see [[re-memdump]]; before dumping, filter out vsyscall/vdso according to the maps).
    • Windows: in x64dbg run to the OEP → Plugins > Scylla > Attach to process → fill in the OEP address (the value recorded in step 2) → Dump. First note the image base + OEP offset.
    • Do not dump before the packer finishes decryption—what you get is the packer's initial state (see pitfall 1).

  5. IAT repair (Scylla/ImpREC)：

    • x64dbg: Plugins > Scylla > Attach to process → fill the OEP with the address from step 2 → IAT Autosearch → Get Imports(Verify API parsing is correct) → Fix Dump Output unpacked_fixed.exe。
    • Alternative: ImpREC (Windows, old tool) same workflow: Open Process → IAT AutoSearch → Fix Dump.
    • Post-repair verification:file unpacked_fixed.exe → Import [[re-ghidra]] / [[re-ida]] decompile to confirm the import table parses → Re-run in sandbox ([[re-sandbox]]) to confirm behavior matches pre-unpacking and no self-check is triggered.

Cross-domain combination

  • [[re-anti-analysis]]: workflow step 3 (simple packer branch) always calls this skill
  • [[re-analyze]]'s triage "sample packed / unpack" path calls it (re-anti-analysis → re-unpack-simple)
  • [[re-malware]]: after unpacking packed malware samples (UPX etc.), re-run in sandbox before behavior analysis
  • [[re-mobile]] / [[re-apk]]: some Android hardening packers can be approached analogously to this workflow's idea (for specific hardening packers, use [[re-frida]] / [[re-memdump]] for in-memory DEX extraction)
  • [[re-crypto-decrypt]] / [[re-crypto-keys]]: when decryption / keys are inside the packer, unpack first (dump after OEP, see [[re-memdump]] for dump timing)
  • Supporting: [[re-memdump]] (default dump), [[re-gdb]] / [[re-x64dbg]] (debuggers), [[re-sandbox]] (re-run verification of unpacked output)

Common pitfalls and traps

  • Dumping too early = packer's initial state: Symptom—after dump, file still reports UPX, decompilation shows only compressed data; Cause—dumped before the packer ran to OEP and finished decrypting; Countermeasure—after decryption is completedump again ([[re-analyze/platform-tips]] key experience), default dump takes priority
  • IAT not repaired → messed-up import table: Symptom—the unpacked sample's decompilation is full of GetProcAddress dynamic calls, import table fails to parse; Cause—the import table is rebuilt by the packer at runtime, and the dump was not repaired; Countermeasure—step 5 Scylla/ImpREC repair, then re-decompile to verify
  • Self-check (CRC) → patch needed after unpacking: Symptom—the unpacked sample exits immediately in sandbox / shows a check-failure popup; Cause—the program compares its own bytes (when originally packed they were compressed data, after unpacking they don't match); Countermeasure—locate the check code (xref own image base / CRC calculation function), patch to skip it or hook it, and do the verification inside the sandbox
  • Wrong debugger used in Wine environment: Symptom—Windows debugger fails to attach under Wine / breakpoints don't take effect; Cause—Wine is a Linux process, needs the Linux debugging path; Countermeasure—per [[re-analyze/platform-tips]] Wine direct read:gdb -p attach to the Wine process to read memory, set breakpoints,gcore dump
  • Double-layer compressed packer → ESP law must be used twice: Symptom—after the ESP law breakpoint hits, what you enter is still the unpacking layer, and what's dumped is still packer code,upx -d after unpacking one layer file still reports packer characteristics; Cause—packer nesting (e.g., ASPack v2.12 has another layer still decrypting the IAT after the first layer is done); Countermeasure—after the first layer lands, keep observing: when another pushad/popad cycle appears, or VirtualFree releases the IAT decryption heap, that's the last step, follow through to the final jmp and verify the target is the real code prologue (push ebp; mov ebp,esp + dense normal API references) before dumping, don't jump too early before the transition jmp
  • Searching for the IAT by FF 25 is unreliable: Symptom—binary searching per the tutorial FF 25(Indirect jmp) Find the IAT start address; if it cannot be found or the wrong location is found; cause—not all programs call APIs via indirect jumps (direct CALL [addr] is very common); countermeasure—use a debugger Find > All intermodular calls to locate a real program call site, then follow CALL/JMP [addr] it up to the top of the jump table to calculate the IAT start address and block size, then fill in ImportREC/Scylla
  • After dump, PE header fields are not fixed → loading fails: symptom—after IAT repair the sample still won't open / "invalid Win32 application" / crashes on load; cause—the dump tool did not fix SizeOfImage/NumberOfSections/SizeOfHeaders, or the RVA/raw ranges of the sections in the section table do not match SectionAlignment/FileAlignment, or LordPE/ImportREC fails on Win7+ due to ASLR; countermeasure—if it is still abnormal after Fix Dump, use pefile to check and correct these fields item by item, and if necessary disable the random base address and re-dump.CheckSum does not belong to this category—ordinary user-mode EXE/DLLs are not checked at load time; only drivers, DLLs loaded during boot, and DLLs loaded into critical system processes need recalculation
  • Crashing after dumping ≠ unpacking failure: symptom—IAT repair is complete and decompilation is also normal, but running still crashes/flashes and exits; it is misjudged as an unpacking failure and abandoned; cause—besides the IAT, it also depends on other mechanisms: TLS callback (the packer resets parameters/decrypts data in the callback), relocation table (base relocations are not handled under ASLR), exception table (the SEH chain is rearranged by the packer), Load Config (security cookie / GS checks point to packer data); countermeasure—check the TLS callback table, relocation directory, exception directory, and Load Config directory item by item according to [[re-format-pe]], and fix whichever is missing; the criterion for successful unpacking = the code/imports/relocations/exception mechanisms are complete (see [[re-unpack-advanced]]), not "does not crash" or "IAT repaired"
  • Applying the ESP law as a rigid formula: symptom—applying the ESP law to TLS callback packers / multi-threaded packers / VMProtect, the hardware breakpoint does not trigger or stops at a misaligned stack address, and hours are spent trying repeatedly with no result; cause—treating "push at entry to find the restoration point" as a general formula, when it actually only applies to single-threaded compression packers with pushad at the entry; countermeasure—before starting, confirm the applicable premise (the EP entry is indeed a pushad/popad cycle), otherwise go directly to the step 3 memory breakpoint method / switch to [[re-unpack-advanced]], and do not waste time on the formula
  • Treating "the first execution of the decryption region" as the OEP and dumping directly: symptom—dumping as soon as the memory breakpoint triggers for the first time; what is obtained is the packer loader / fake OEP, and decompilation is still packer code; cause—the packer first jumps to the loader, decrypts in multiple stages, or sets up a fake OEP, so the first execution ≠ decryption complete; countermeasure—after breaking, first verify the real code characteristics (function prologue + dense API references), and for multi-stage packers set the breakpoint repeatedly until the control flow stabilizes
  • Failure to unpack under Wine is not necessarily a process error: symptom—under Wine, operating according to the complete process, breakpoints do not trigger / the packer runs abnormally / decryption results are garbled; cause—a Wine environment problem rather than a process problem: missing APIs that the packer depends on, inability to load packer driver components, anti-virtualization detection (Wine-characteristic APIs/registry/process names) or environment detection (system version/hardware characteristics) causing the packer to refuse normal decryption; countermeasure—first locate the failure point:WINEDEBUG=+loaddll check the load failure log, break on GetVersionEx/registry reads to see whether environment detection is hit; after confirming it is a Wine environment limitationdecisively switch to a Windows VM([[re-sandbox]]) to complete unpacking, and do not repeatedly struggle in Wine