re-fw-extract

Firmware extraction and unpacking

When to use / when not to use

• Use: you have a firmware file (.bin / .img / update package / IoT image) and need to unpack it to see the contents
• Use: you need to determine the firmware packaging method, architecture, and endianness
• Don't use: it is already a filesystem image / rootfs (go directly to [[re-fw-rootfs]])
• Don't use: you need to run the firmware to observe behavior (go to [[re-fw-emulate]])
• Don't use: you need hardware extraction from a physical board (go to [[re-hardware-io]])

Tool preparation

Verify all tools before use. Unpacking and magic scanning are pure static analysis, sandbox can be skipped ([[re-analyze/platform-tips]] highest principle); when extracted artifacts need to run, switch to [[re-fw-emulate]].

binwalk —— the mainstay for automatic unpacking

binwalk has two coexisting product lines; before installing, first distinguish them:v3 (Rust rewrite) is the current upstream mainline，v2 (the original Python implementation) belongs to legacy, and the two differ in parameter sets and module support.

• v3 (current mainline): install via the channels in the official README——Docker image / cargo install binwalk / build from source. Upstream no longer treats pip as the installation entry point
• v2 (legacy): the PyPI binwalk package stops at 2.1.0 (2015-01), and distro repository packages belong to the same 2.x line (Debian trixie 2.4.3, bookworm 2.3.4)
  • Linux: apt install binwalk（Debian/Ubuntu）/ dnf install binwalk / pacman -S binwalk
  • pip: pip install binwalk
  • macOS: brew install binwalk
  • Windows/WSL: there is no official package for native Windows; use the Linux version inside WSL
• Verify: binwalk --version——confirm the major before use. The command examples in this skill (-e / -M / -E) are retained in both v2 and v3; when you need other v3 parameters, check according to the actual version binwalk --help

unblob —— more accurate automatic unpacking (recommended mainstay)

• Cross-platform: pip install unblob(many dependencies, venv recommended: python3 -m venv venv && venv/bin/pip install unblob）
• distros basically have no official package, use pip
• Verify: unblob --version

dd —— split by offset

• Linux/macOS: included with coreutils (macOS ships /usr/bin/dd)
• Windows/WSL: Linux version inside WSL; on native Windows use the dd bundled with Git Bash or WSL
• Verify: dd --version

hexdump —— hexadecimal viewing and manual magic scanning

• Linux: apt install bsdmainutils（Debian/Ubuntu）/ dnf install util-linux(Fedora/RHEL, includes hexdump)
• macOS: ships /usr/bin/hexdump
• Windows/WSL: Linux version inside WSL
• Verify: hexdump -C /dev/null(usable if no error; the macOS version lacks --version）

sasquatch —— unpacking old/vendor-modified squashfs

• Linux: build from source (git clone https://github.com/devttys0/sasquatch, requires apt install zlib1g-dev liblzma-dev build-essential then make）
• macOS: build from source as above (requires Xcode Command Line Tools)
• Windows/WSL: build inside WSL
• Verify: sasquatch -h outputs usage (the build artifact is in a repository subdirectory and needs to be added to PATH)

Procedure

Execute in order, and record the result of each step.

1. Automatic unpacking (unblob first, binwalk as fallback)：

   unblob firmware.bin                      # 更准，自动识别 50+ 容器格式并递归解包
   # 或 binwalk：
   binwalk firmware.bin                     # 先列出签名与偏移
   binwalk -Me firmware.bin                 # -M 递归 -e 提取
   

   Artifacts:firmware.bin/(unblob) or _firmware.bin.extracted/(binwalk) directory. Extracted filesystems go to [[re-fw-rootfs]], ELF goes to [[re-binary-core]].

2. magic manual scanning (when automatic unpacking is incomplete)：

   hexdump -C firmware.bin | head -40
   # 或按 binwalk 报告的偏移核对：
   dd if=firmware.bin bs=1 skip=<偏移> count=16 | hexdump -C
   

   Common magic numbers: JPEG FF D8 FF、gzip 1F 8B、squashfs hsqs、cramfs 45 3D CD 28(little-endian, big-endian reversed), U-Boot 27 05 19 56、jffs2 85 19、ELF 7F 45 4C 46. Once one is identified, process it according to that format.

3. Endianness determination：

   file firmware.bin                        # 输出含 LSB/MSB 提示
   readelf -h <解出的ELF>                    # Machine 字段 + Data 字段（大小端）
   strings firmware.bin | head               # 可读串确认字节序
   

   Common in big-endian ARM/MIPS firmware: in big-endian, magic numbers and strings are encoded in big-endian (e.g., hsqs appears reversed); once confirmed, the entire process proceeds according to that endianness.

4. Decompress nested containers layer by layer：

   • unblob / binwalk -Me automatically recurses, but nesting (zip inside tar, gzip wrapped in a custom header) often stops midway
   • Manual layer by layer: first extract the outer layer → file confirm the inner type → use the corresponding tool (tar/gzip are built into the system; use sasquatch for squashfs; for others use the 7z/unsquashfs prepared by the [[re-fw-rootfs]] tool) to extract again, until a filesystem or ELF appears
   • Additional data after the end marker: after the end marker of formats such as images (PNG IEND、JPEG FFD9), containers/compressed streams are often appended (the parser stops upon reading the end marker, so the additional data is invisible to normal viewing)——file will report the entire file as an image.Locating the end marker cannot rely on raw-byte first/last searches（IEND can appear inside tEXt chunk payloads,FF D9 can appear inside JPEG APP/COM segments, and the first/last hit may both be false markers): for PNG, starting from the 8-byte signature, traverse by Length/Type/Data/CRC to the IENDthat is structurally valid and has length 0; for JPEG, start from SOI and parse by marker/segment, afterSOS handle by FF00 stuffing and RSTn , take thewhose syntactic position is validof EOI; use the parsed true end offset to cut the additional data (raw search results are only candidates)
   • zip can be repaired even when the signature is missing: an appended zip may lack the PK\x03\x04at the start of the local file header (4 bytes stripped)——verify with self-consistent fields: after adding the signature back, version (commonly 20/45), mod date (year 1980+), compressed/uncompressed size (consistent with EOCD/central directory entries) are all reasonable, and the EOCD is intact at the tail → adding the PK\x03\x04 prefix makes it complete and extractable (unzip reporting "missing 4 bytes" or zipfile OSError: Invalid argument is a typical sign)
   • Automating the nesting-doll pattern: when the same technique repeats (e.g., each layer is "image + trailing zip containing the next layer"), write a loop to strip automatically——extract the tail → repair → decompress → locate the next layer → repeat, until there is no additional data; keep each intermediate product named per layer (to allow rollback)

5. Manual splitting when automatic fails (dd by offset)：

   binwalk firmware.bin                     # 找内嵌文件偏移
   dd if=firmware.bin of=part1.bin bs=1 skip=<偏移1> count=<长度1>
   dd if=firmware.bin of=part2.bin bs=1 skip=<偏移2>
   file part*.bin                           # 每块验证
   

   Vendor custom headers are the most common: the standard format only begins after skipping the first N bytes of the header (first eyeball the header length with hexdump, then cut).

MCU image analysis (8051 / AVR / PIC / MSP430)

When to enter this section

• Prerequisite: steps 1-5 have been completed (automatic unpacking, magic scanning, endianness determination, nested unpacking, dd splitting), and it is confirmed that the image contains no filesystem/ELF/executable container
• Criteria (enter this section if satisfied):
  • No filesystem: binwalk/unblob has zero hits or only a few false positives, with no squashfs/cramfs/jffs2 magic numbers, etc.
  • Single image: small size (on the order of tens of KB), no partition table, no bootloader boot header, no multi-segment structure
  • MCU characteristics:file reports data/unknown; strings has no OS or library characteristic strings (no "libc", no RTOS kernel name); at the low addresses at the start of the image there is a short jump/vector data rather than a readable header
• Exception routing: RTOS characteristics appear in the image (scheduler/task creation strings, TCB form) → after locating the reset vector and startup code, switch to [[re-rtos]]

Image formats: Intel HEX and binary

• Intel HEX（.hex）: ASCII text, recording line by line :LLAAAATT<数据><校验和>—LL data length, AAAA intra-segment address, TT record type (00 data / 01 end / 02·04 extended address / 03·05 start address), last-byte checksum
  • Addresses may be segmented and non-contiguous: gaps are left between multiple segment records, and disassembling the entire file as bin will inevitably be misaligned
  • First convert to a memory image: place the data at the actual addresses according to the records (02/04 extended address records determine the upper 16 bits of the address), then import into the disassembler using the flash start base address
• Binary (.bin): raw memory image, no address information, manually provide the base address when importing
• Program memory layout (generalized, specific values vary by chip):
  • flash start: 0x0000 or the chip mapping base address (e.g., on the order of 0x8000/0x10000); inferring from the location of the reset vector is more reliable
  • Interrupt vector table: a fixed region at low addresses; the reset vector is at a fixed offset within the table (see the next section)

Vector table location

• Reset vector → startup code: take the entry address from the reset vector location; disassembling from there is the startup sequence (disable interrupts, initialize stack/clock/peripherals, jump to main)
• The reset vector location differs by architecture: 8051 at 0x0000, AVR at 0x0000, MSP430 at 0xFFFE (high-address end), PIC mostly at 0x0000 (some series have another fixed offset)—look up the location by architecture, do not guess
• Firmware running an RTOS: reset vector → startup code → kernel initialization/task creation → scheduler start; connect to [[re-rtos]] and split by task table

ISA identification and disassembly

• Identification points (judged by combining instruction word width, addressing characteristics, and endianness):
  • 8051: Harvard architecture, separate program/data spaces (MOVC reads the program area, MOVX accesses the external data area); SFR special function registers (0x80-0xFF direct addressing); the 0x20-0x2F area is bit-addressable; accumulator A and DPTR dominate data movement
  • AVR: RISC, 32 general-purpose registers (R0-R31) register file, most instructions single-cycle; Harvard architecture
  • PIC: Harvard architecture, registers are divided into banks, switched via status bits (bank switching); instruction word width varies by series (12/14/16 bits), use the corresponding width when importing into the disassembler
  • MSP430: von Neumann (unified address space), 16-bit; the interrupt vector table is concentrated in the high-address area (0xFFE0-0xFFFF)
  • Endianness: AVR/PIC/MSP430 are little-endian; 8051 has no native endianness instructions and multi-byte data is conventionally stored big-endian (Keil C51 defaults to big-endian), opposite to the other three—when importing into the disassembler, confirm with the actual data (multi-byte constants and the byte arrangement of jump target addresses serve as the basis for judgment)
• Disassembly: select and import the processor module for the corresponding architecture (Ghidra includes processor support for 8051/AVR/PIC/MSP430, [[re-ghidra]]; other disassemblers likewise have corresponding processors; the tool can be replaced, the method is the core)
• Fuse bits/configuration bits (generalized concept): programming-time configuration items (called differently across architectures: fuse bits/configuration bits/security bits), written once during programming, affecting clock source selection, watchdog, code/read protection, etc. They are not reflected in the image contents, but determine the chip's subsequent hardware behavior—when the image cannot be read, suspect them first

Common pitfalls and traps in this section

• Disassembling multi-segment Intel HEX directly as bin: Symptom—disassembly is misaligned, addresses do not match the code, all invalid instructions; Cause—HEX segment records have non-contiguous addresses, and ignoring 02/04 extended address records will place high-address segment data at low addresses; Countermeasure—first place data at actual addresses according to record type (extended address records participate in address synthesis), generate a contiguous memory view, then import according to the flash base address
• Guessing the vector table offset wrong: Symptom—disassembling directly from the file header yields a bunch of data/invalid instructions, and no entry point can be found; Cause—the reset vector location differs by architecture (8051/AVR at 0x0000, MSP430 at 0xFFFE, PIC mostly at 0x0000 or a series-fixed offset); guessing wrong means disassembling starting from data; Countermeasure—first confirm the architecture, then take the entry point from the fixed reset vector location for that architecture, and follow the startup code from the reset vector
• Fuse bits lock read protection, image unavailable: Symptom—chip read protection is enabled, read operations return all F/reject responses, and a complete image cannot be obtained; Cause—programming-time configuration bits enabled read protection (code protection), and the normal read interface is locked; Countermeasure—use alternative paths: the chip's built-in UART bootloader interaction (if not disabled), logic analyzer capture of firmware upgrade/programming traffic ([[re-hardware-io]]) and restore according to the protocol; when the image is unreadable, do not force it, switch to behavior-side analysis (serial output/protocol interaction)
• Confusing data EEPROM with program flash: Symptom—a large number of invalid instructions are disassembled, or after analyzing for a long time it turns out to be data; Cause—the image contains data EEPROM/calibration table/configuration table images (non-volatile data area, independent of the program area), which are mistaken for code; Countermeasure—use the reset vector/vector table to locate the start of the program area, first exclude high-entropy/all-F/repeating-pattern segments outside the program area as data areas, then disassemble

Cross-domain combination

• [[re-firmware]]: workflow step 2 always calls this skill
• Extracted filesystem → [[re-fw-rootfs]]; extracted executable program → [[re-fw-emulate]]
• Extracted ELF → [[re-binary-core]] ([[re-format-elf]] / [[re-ghidra]]); malicious sample discovered → [[re-malware]]
• This skill is called by [[re-analyze]]'s triage "analyze firmware / IoT device" path (re-firmware → re-fw-extract)

Common pitfalls and traps

• Automatic unpacking failure ≠ nothing there: Symptom—binwalk output is blank or extracts 0-byte files; Cause—vendor custom headers/encryption layers cause magic number mismatches, and the signature library misses detection; Countermeasure—manually scan for custom headers with hexdump (step 2), skip the header offset, then split with dd (step 5)
• Wrong endianness → everything wrong: Symptom—extracted files/ELF/strings are all garbled; Cause—big-endian ARM/MIPS firmware is treated as little-endian; Countermeasure—first file + readelf -h confirm the endianness (step 3), and keep it consistent throughout subsequent unpacking/analysis
• Big-endian MIPS header missed by binwalk: Symptom — binwalk cannot identify a known filesystem (e.g. squashfs); Cause — the signature is matched by little-endian characteristics, and the big-endian magic number is reversed; Countermeasure — use hexdump to manually find the reversed magic number, dd at the offset, and unpack with sasquatch
• Unpacked output mixed with garbage: Symptom — the extracted "file" is random data,file reports data; Cause — a false signature match or a misaligned split offset; Countermeasure — for each output file + look at the first 16 bytes to verify the magic; if invalid, retry the offset
• Extension does not match content (disguised extension): Symptom —.jpg reports PNG,.so reports tar,.c reports XZ; Cause — the author deliberately used an unrelated extension (common in steganography/nesting/check-evasion scenarios); Countermeasure — always rely on file/magic, treat the extension only as a clue;file when the output includes hints such as "with extra data prepended", just follow them directly
• zip errors but can list files: Symptom —unzip reports "missing 4 bytes"/"invalid zip with overlapped components" (false zip bomb detection) or Python zipfile OSError: [Errno 22] Invalid argument; Cause — the local file header is missing the PK\x03\x04 signature, or there is prefix garbage before the data; Countermeasure — first unzip -l to see whether it can list (if it can list, the EOCD/central directory is intact), patch the signature (step 4), or use UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE to force
• ME firmware (Intel ME / CSME) starts from the boot chain: Symptom — after extracting the ME firmware partitions (FTPR/RBE/BUP), you do not know where to start, and the string of ROM_SVC_* calls in the code is incomprehensible; Cause — ME is an independent processor firmware, and its boot chain differs from ordinary firmware: ROM (fixed in the PCH, not extractable) verifies the signature of FTPR → decompresses RBE (initializes PCH/SPI, UTOK verification, boot_cfg) → kernel (ThreadX RTOS) → BUP (hardware bringup, reads boot_cfg to decide whether to enable DCI) → 38 user processes such as pm/vfs/heci/crypto; Countermeasure — first parse the FPT (Flash Partition Table: the partition routing table at a fixed offset in the ME region, recording each partition's location/size/type — ROM/RBE have no filesystem and rely entirely on it to locate partitions; the UEFIExtract source code has a parsing reference), then follow the order RBE→kernel→BUP; the debug interface/hardware initialization logic is in BUP
• UTOK is the OEM backdoor for ME debug unlock: Symptom — you want to enable the ME debug interface (DCI/DFX) but cannot find the switch; Cause — an unlock mechanism Intel reserved for OEMs: a separate UTOK partition in the SPI Flash FPT partition table; writing data into it unlocks the DFX debug interface (no hardware fuse needed); Countermeasure — reverse the RBE's rbe_utok_check to locate the verification logic: debug capability starts only when the flag UTOK[0x298]==1 (factory default FF does not start it), and DCI exposure is controlled by a multi-layer verification chain (boot_mode==2, boot_cfg parsing, CT handler gating, FPF Debug Auth, etc.), so changing the flag alone is not enough; Locating tip: the ME runtime base is 0x4000, and string offsets are converted accordingly
• ME firmware exploit is restricted by the ROM cookie: Symptom — a buffer overflow is found in ME firmware (e.g. Intel-SA-00086: CT file num_records(uint16 at offset +0x06) no canary, record count exceeding 100 overwrites stack cookie/return address), yet ROP cannot be made to work; reason — ME's cookie is generated by ROM (mask ROM, non-extractable) during the SRAM pre-fill process context table, and cannot be bypassed via TLS like on TXE platforms; countermeasure — first confirm the target platform's cookie source (if ROM-generated, overflow exploitation is limited), fully reverse the authentication chain (UTOK→boot_cfg→CT handler multi-layer verification), then look for a non-overflow path
• Block-wise periodic transformation: symptom — a single whole-file XOR/rotation only works at the beginning; reason — the transformation resets per block size (commonly 256/512/1024/2048/4096); countermeasure — first check whether the transformation resets periodically per common Flash/transfer block size, then decode explicitly in blocks
• Block-header self-contained mask obfuscation: symptom — high-entropy firmware is mistaken for strong encryption; reason — each block carries its own mask obfuscation (e.g. mask = block[0]、plain[i] = ROR8(packed[i], n) XOR mask(the mask byte itself does not participate in the transformation, i starts at 1)); countermeasure — first use a strong crib (vector table SRAM stack pointer/Thumb Reset Vector) to constrain and recover the first block, then verify the model
• Mode crib is unreliable: symptom — using the modal byte as a mask on text-dense blocks still yields garbage; reason — the most common plaintext byte is not necessarily zero; countermeasure — switch to a block-header-byte model, with validation criteria: garbage disappears + reproduced on a second firmware of the same series + byte-for-byte round-trip consistency
• CRC string ≠ CRC field: symptom — seeing a CRC name string and assuming the end is a standard CRC; reason — the string is merely a metadata key; countermeasure — systematically rule out common CRC/hardware CRC/Adler/accumulator families and keep it as an unknown field, without forcing a name (source: reverse-skill field-journal, MIT)