SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsre-protocol

re-protocol

General & other

View repository

Updated 10-16 05:07

About this skill

Protocol Reverse Engineering

Complete Workflow

Execute in order; record evidence paths + sha256 for each step's artifacts (pcap/keys/decryption script/parsing script) for report citation (see [[re-ioc]] (capability:threat-intel））。

  1. Capture: [[re-netcap]] (capability:network-capture） — first determine the capture point (local host/gateway/man-in-the-middle), in-sandbox capture preferred ([[re-sandbox]] (capability:sandbox-setup) Network isolation: INetSim / fake DNS / network disconnection, to prevent real external connections, see [[re-analyze/platform-tips]] highest principle); tcpdump filter to keep only the target stream before saving to disk, for HTTPS/TLS prepare in advance with mitmproxy CA
  2. Crypto identification: [[re-crypto-id]] (capability:crypto-identification） — determine whether traffic is plaintext or ciphertext: entropy >7.0 / unstructured / no ASCII → ciphertext; then do constant table fingerprints (AES S-box / CRC table), XOR/ROL/ROR single-byte patterns, common algorithm flow characteristics
  3. Keys: [[re-crypto-keys]] (capability:key-extraction） — static first (strings / cross-references to find hardcoded, resource files, near Crypt* in import table), if static fails then dynamic ([[re-memdump]] (capability:memory-dump) by default after dump search for 16/32-byte entropy blocks and printable passwords), PBKDF-like restored according to derivation function
  4. Decryption: [[re-crypto-decrypt]] (capability:crypto-decryption） — locate the decryption function (cross-reference ciphertext input point) → decompile and restore algorithm → rewrite as standalone python script → verify with known plaintext/known header → decrypt the captured ciphertext stream into a plaintext traffic stream
  5. State machine reconstruction: [[re-proto-rev]] (capability:protocol-recovery） — only plaintext traffic proceeds to this step: packet grouping statistics and clustering (length/direction/timing) → locate fixed header (magic/length field) → field inference (type/length/CRC) → write parser with Scapy → state machine deduction (handshake/heartbeat/end)

Pre-check: Do not enter state machine reconstruction before ciphertext is decrypted (would mistake gibberish for structure); plaintext traffic skips steps 2-4.

When to use which atomic skill (selection tree)

Branch by input characteristics/objective:

  • Traffic available (pcap / live capture) → [[re-netcap]] (capability:network-capture; capture) → see if ciphertext:
    • Ciphertext (high entropy/unstructured) → [[re-crypto-id]] (capability:crypto-identification) → [[re-crypto-keys]] (capability:key-extraction) → [[re-crypto-decrypt]] (capability:crypto-decryption) → after plaintext then [[re-proto-rev]] (capability:protocol-recovery）
    • Plaintext → [[re-proto-rev]] (capability:protocol-recovery) directly reconstruct state machine
  • Industrial control/SCADA protocols (Modbus/DNP3/OPC UA, ports 502/20000/4840) → [[re-ics]] (capability:protocol-recovery; industrial control traffic parsing and point table; for security testing boundaries see [[re-sandbox]] (capability:sandbox-setup））
  • IoT device protocols (MQTT/CoAP/BLE/Zigbee, 1883/5683/2.4GHz band) → [[re-iot-proto]] (capability:protocol-recovery; device semantic parsing + firmware linkage [[re-firmware]] (capability:firmware-extraction、emulation、rtos-analysis、hardware-interface、uefi-analysis））
  • Only binary samples, no traffic("What is the protocol implementation logic") → Find the encryption implementation from static analysis [[re-crypto-id]] (capability:crypto-identification) → [[re-crypto-keys]] (capability:key-extraction) → [[re-crypto-decrypt]] (capability:crypto-decryption); for deeper logic go to [[re-binary-core]] (capability:decompilation、debugging、memory-dump、elf-parser、pe-parser、macho-parser; [[re-ghidra]] (capability:decompilation、debugging) / [[re-ida]] (capability:decompilation、debugging) / [[re-radare2]] (capability:decompilation））
  • To understand the interaction semantics("How do the client and server talk" "handshake process") → [[re-proto-rev]] (capability:protocol-recovery）
  • If you only need to decrypt a blob of a known algorithm(algorithm/key known) → directly [[re-crypto-decrypt]] (capability:crypto-decryption）
  • If you only need to find the key("Is there a hardcoded key in the sample") → [[re-crypto-keys]] (capability:key-extraction; static first, see the static-first approach of the highest principle in [[re-analyze/platform-tips]])
  • White-box cryptography(large blocks of table lookup code, no standard library calls, key hidden in the table) → [[re-whitebox]] (capability:crypto-identification、key-extraction; identification → table extraction → key recovery, connecting to the encryption trio)
  • Traffic capture environment not ready → first [[re-sandbox]] (capability:sandbox-setup) network isolation (INetSim / fake DNS) then come back to [[re-netcap]] (capability:network-capture）
  • Standard TLS/encrypted traffic depth(ClientHello fingerprint, SSLKEYLOG decryption, TLS 1.2/1.3) → [[re-tls]] (capability:tls-analysis、crypto-identification; standard TLS stack; for self-implemented encryption switch to the crypto trio)

Cross-domain combination

  • C2 communication analysis ([[re-malware]]): This gateway is step 4 of the re-malware workflow (capture callback traffic → reconstruct protocol → identify and decrypt communication encryption); the encryption trio (crypto-id / crypto-keys / crypto-decrypt) is directly referenced by the C2 decryption path; capture depends on [[re-sandbox]]'s INetSim / fake DNS network isolation
  • Firmware communication analysis ([[re-firmware]]): This gateway is step 6 of the re-firmware workflow (firmware callback / custom protocol / encrypted communication), [[re-netcap]] captures packets from the virtual NIC of the emulation environment, and the protocol clues extracted by binwalk are for [[re-proto-rev]] to reference
  • Behavior analysis connection ([[re-behavior]]): After behavior analysis records network connections (callback domain/IP/port), switch to this gateway for protocol-layer analysis
  • IOC output ([[re-ioc]]): C2 domain/IP/port, protocol fingerprint, and decrypted config plaintext go into the IOC list and YARA signatures
  • Binary deep dive ([[re-binary-core]]): Decompilation of encryption implementation/decryption functions goes through [[re-ghidra]] / [[re-ida]] / [[re-radare2]]; for keys in memory go through [[re-memdump]]
  • White-box cryptography ([[re-whitebox]]): When encryption identification/key chain encounters a white-box implementation (large blocks of table lookup encryption without an explicit key), switch to it—the branch where the conventional [[re-crypto-keys]] path fails
  • TLS specialty ([[re-tls]]): In-depth analysis of standard TLS traffic—JA3/JA4 fingerprinting, SSLKEYLOG decryption (TLS 1.2/1.3), certificate chain analysis; the decrypted plaintext traffic continues to [[re-proto-rev]]
  • Entry dispatch: This gateway is invoked by the "analyze network traffic / unknown protocol" path in [[re-analyze]]'s triage.md (re-protocol → netcap → crypto-* → proto-rev)

Common Pitfalls and Traps

  • Capturing packets without isolating the sandbox network → real external connections: Symptom—the sample actually accessed an external C2, and the captured traffic cannot distinguish malicious callbacks from normal external connections; Cause—skipping [[re-sandbox]] network isolation (INetSim / fake DNS / network disconnection) and running directly online; Countermeasure—isolate the network before running any sample (the highest principle of [[re-analyze/platform-tips]]), and choose the capture point inside the isolated environment
  • Treating ciphertext as plaintext and directly reconstructing the state machine: Symptom—proto-rev clustering/field inference results are all chaotic, and the "structures" parsed by the parser are all random bytes; Cause—the traffic carries an encryption layer (entropy >7.0) that was not identified first; Countermeasure—in step 2, [[re-crypto-id]] first confirm ciphertext, follow the decryption chain, and then use [[re-proto-rev]]
  • Skipping key extraction and hard-coding a decryption script: Symptom—the decryption script cannot decrypt the captured traffic into plaintext, or only decrypts half of it; Cause—the key is dynamically generated/changes per session, so hard-coded assumptions fail; Countermeasure—follow [[re-crypto-keys]] to obtain the real key from a memory dump ([[re-memdump]]) or from the key derivation point, and keep the key parameterized in the script
  • Taking decryption results as conclusions without verification: Symptom—the "plaintext" written in the report is actually an incorrect decryption (padding misalignment/IV error); Cause—no comparison with known plaintext (protocol header magic, readable strings, known field values); Countermeasure—step 4 must use known plaintext/headers for verification, and the decrypted plaintext must undergo another readability check
  • Capturing packets without filtering → the file is huge and impossible to analyze: Symptom—the pcap is tens of GB, and tshark statistics freeze; Cause—no filtering at the capture point (all interfaces, all protocols); Countermeasure—in step 2 of [[re-netcap]], first write a filter expression (host/port/protocol), and filter before saving to disk
  • C2 over DoH → network isolation and content analysis are completely blind: Symptom—INetSim has no DNS query records at all, and the pcap only contains 443 ciphertext connecting to dns.google/cloudflare-dns.com, leaving no raw material for protocol reconstruction; Cause—the sample has its own DNS client and directly uses DoH to connect to an external resolver, bypassing the system resolver—fake DNS/INetSim are completely blind, and DoH traffic cannot be distinguished from normal browser behavior by content (the DoHdoor backdoor encodes C2 commands into the DoH query string); Countermeasure—identify the signals "443→known DoH endpoint / TLS handshake without SNI (hard-coded IP direct connection) / a connection on the process side but no DNS event (Sysmon 3 without 22)", shift to the behavioral side (beacon period, owning process) to locate C2, redirect the DoH endpoint IP, or man-in-the-middle 443 (see the [[re-netcap]] pitfall)

General path for protocol recovery

  • Declarative definitions take priority; packet capture as a fallback: Symptom—protocol recovery starts from packet capture/behavior observation and takes a long detour; Cause—the client often embeds declarative definitions (proto/schema/interface description files), which can be extracted in plaintext or semi-plaintext; Countermeasure—when unpacking, first search for declarative definitions (syntax = "proto3"、.fbs, interface descriptions, etc.),recovery priority: declarative definitions > decompiled packing/unpacking implementation > packet capture
  • Directly probe official APIs in operation: Symptom—repeated trial and error by statically guessing the response format; Countermeasure—when the official service is still in operation, directly hit the real API (read-only, low frequency) according to the client request parameters, obtain the envelope ({code,msg,data}) and error code semantics (401 wrong parameter/404 wrong id/406 wrong op) all at once, then confirm the fields against the client parsing code; this is an order of magnitude faster than pure static recovery
  • For encrypted "not implemented", perform a full search to confirm: Phenomenon—the protocol flag bit has 0x40000000=Encrypt so an encryption layer is assumed; Cause—the flag bit is infrastructure, the feature may never have been enabled; Countermeasure—search exhaustively for algorithm implementations (AES/RC4/XXTEA, etc.) and call sites, and only conclude "unencrypted" if none are found in either the C# or native layer (a common case: the encryption bit is reserved, and only the compression bit is actually used)