re-proto-rev

Protocol State Machine Reconstruction

When to Use / When Not to Use

Use: you have plaintext traffic (pcap or live stream) and need to figure out "what the messages look like and how the two sides converse"
Use: inferring the field structure of custom protocols / proprietary protocols / unknown protocols
Use: deducing the interaction patterns of C2 protocols and firmware communication protocols (handshake/heartbeat/teardown)
Don't use: the traffic is ciphertext (first go [[re-crypto-id]] → [[re-crypto-keys]] → [[re-crypto-decrypt]], recover the plaintext, then come back)
Don't use: standard protocols (HTTP/TLS/DNS)—just use the Wireshark dissectors (see [[re-netcap]] step 3)
Don't use: you only need to capture traffic (that's [[re-netcap]])

Tool Preparation

Verify all tools before use. This skill only handles data and parsing scripts, does not run samples, and can skip the sandbox ([[re-analyze/platform-tips]] highest principle); the traffic capture step happens within [[re-netcap]].

python3 — runtime environment for parsing scripts

Linux: apt install python3 python3-pip / dnf install python3 python3-pip / pacman -S python python-pip(mostly built in)
macOS: brew install python(or bundled with Xcode CLT)
Windows: python.org installer (check Add to PATH); Linux version inside WSL
Verify: python3 --version

scapy — the workhorse for pcap parsing and protocol construction

All platforms: pip install scapy(Python 3, recommended)
Linux: some distros have a package (apt install python3-scapy）
Verify: python3 -c "import scapy; print(scapy.__version__)"
Read a pcap file: rdpcap is enough, pure Python parsing (no libpcap needed; libpcap is only needed for live capture sniff() ; to force pure Python parsing use conf.use_pcap = False）
Verify (cross-platform, write a temporary pcap and read it back, no dependency on /dev/null): python3 -c "import tempfile,os; from scapy.all import rdpcap, wrpcap, IP, TCP, Raw; f=os.path.join(tempfile.gettempdir(),'t.pcap'); wrpcap(f,[IP()/TCP()/Raw(b'test')]); print(len(rdpcap(f)))" — outputting 1 means it's normal

tshark — packet statistics and initial field exploration (installation see [[re-netcap]])

tshark -r out.pcap -q -z conv,tcp session statistics,-z io,stat,1 time series—used for clustering in step 1
Verify: tshark --version

binwalk — clues for known formats (finding formats when a pcap has an embedded blob)

Installation and verification see [[re-fw-extract]] tool preparation
Purpose: when parsed fields are suspected to contain an embedded file/image (gzip/zip/PNG etc.), use binwalk blob.bin to confirm the format

Operating Steps

Execute in order, record the result of each step. Prerequisite:the traffic must be plaintext(ciphertext first goes through [[re-crypto-decrypt]]).

Packet statistics and clustering (length/direction/timing)：
```
tshark -r out.pcap -T fields -e tcp.len -e ip.src -e ip.dst | awk '{print $1}' | sort -n | uniq -c   # 长度分布
tshark -r out.pcap -q -z conv,tcp          # 会话列表：谁和谁、包数、字节数
tshark -r out.pcap -q -z io,stat,1         # 每秒流量：找规律性心跳
```
- Clustering goal: group by direction (A→B / B→A) and look at the length distribution; lengths showing a few fixed values → fixed-length messages (possibly no length field); lengths varying continuously → variable-length messages (usually requiring some kind of framing; candidates: embedded length field, delimiter, fixed grammar, chunking, connection close)
- Look for regularly spaced packets (a fixed-length small packet every 30s) → heartbeat messages (see pitfall 3)
TCP stream reconstruction → application-layer framing (mandatory, cannot be skipped)：
- TCP is a byte stream and does not preserve application-layer message boundaries (RFC 9293 §3.7)—a segment may be 0, 1, or multiple messages, and a message may also span multiple segments (MSS splitting). Treating bytes(p[TCP].payload) as one message per packet will systematically go wrong: back-to-back continuous send() are often merged into one chunk, while large data or slow links get split; "one write equals exactly one segment" is just a timing coincidence
- By connection + direction（(src,sport,dst,dport) quad-tuple normalized and split into A→B / B→A) reassemble separately: order by TCP SEQ, deduplicate (retransmissions/overlapping segments), fill out-of-order holes, and record capture gaps (positions of missing bytes caused by packet loss)—candidate messages spanning a gap are discarded; do not force-assemble from erroneous bytes
- After reassemblingunidirectional byte streamthen infer framing: magic+length / delimiter / fixed length / TLV / framing by connection close. Do not infer fields before framing is determined
- Scapy route:TCPSession sessionization + tcp_reassemble() hooks (can handle duplicates/overlaps); without writing code, first use tshark -r out.pcap -q -z follow,tcp,raw,<流号> to obtain a unidirectional byte stream and inspect it manually
- Datagram transports such as UDP do not use this model: datagrams preserve message boundaries; one datagram is one message (or contains a whole message); it may still be multiple messages concatenated or further fragmented at the application layer, so only do length/boundary statistics, not SEQ reassembly
Locate the fixed header (magic / length field)：
```
# 对重组后的单向字节流扫描候选 magic 前缀（不是逐包取前 8 字节）
from collections import Counter
stream = b""  # 步骤 2 按 (四元组, 方向) 重组出的单向字节流
cnt = Counter()
for off in range(max(0, len(stream) - 8)):
    cnt[stream[off:off + 8]] += 1
for k, v in cnt.most_common(10):
    print(v, k.hex())
```
- High-frequency prefix → magic candidate (e.g. 0xAA55、\xDE\xAD\xBE\xEF, ASCII brand name); in the same stream the 2nd-4th bytes vary but the prefix stays the same → variable-length message, and the varying part is the length field
- A candidate magic must pass the length constraint to count: suppose the magic is at offset m, and the 2nd-4th bytes after it are the length field L, then the next magic should fall at m + 头部长度 + L ; only candidates where most occurrence positions satisfy this recurrence are true magics (this excludes byte pairs that happen to appear in the data)
- Manual verification: use hexdump to look at a few representative positions (tshark -r out.pcap -q -z follow,tcp,raw,<流号>), confirm the magic and the length field byte order (see pitfall 1)
Field inference (type / length / CRC)：
- Length field: take the value at a fixed offset vs the difference between the start offsets of adjacent frames in that directionPlot a scatter/sort and verify; try byte order based on value plausibility (try both big-endian and little-endian, see pitfall 1)
- Type field: a fixed set of first bytes within the same length family (e.g. 0x01/0x02/0x03) → message type; combine with timing to see whether the type sequence looks like request-response pairing
- CRC/checksum: the last 2-4 bytes of a packet are sensitive to payload changes → checksum (identify it first, do not parse it as data—see pitfall 2); use crcmod（pip install crcmod) to try common CRC16/CRC32 polynomials, or XOR with the payload to check whether it is a simple XOR checksum
- When the structure is uncertain, use the differential method: two packets differing by only a single byte, XOR byte by byte → the bytes that change are the positions of the "key fields"
Write a parser in Scapy (custom protocol class)：
```
from scapy.all import Packet, ByteField, ShortField, XByteField, StrFixedLenField
class C2(Packet):
    name = "C2"
    fields_desc = [
        XShortField("magic", 0xAA55),
        ByteField("type", 0),
        ShortField("len", 0),
        StrFixedLenField("payload", b"", 64),
    ]
# 测试解析（hex 与 fields_desc 对齐：aa55 | 01 | 000a | 54657374696e672e2e2e，len=0x000a=10 = "Testing..." 的 10 字节）
pkt = C2(bytes.fromhex("aa5501000a" + "54657374696e672e2e2e" + "00" * 54))   # aa55 | 01 | 000a | "Testing..."（10B）+ 54B 零填充，凑满 64B payload
pkt.show()
```
- Field order/width follow the inferences from steps 3 and 4; if the parser can display field by field (show()), that verifies the field layout is correct
- Batch processing:run a continuous buffer parser on the unidirectional byte stream reassembled in step 2(not per-packet C2(load)on packets)—loop from the current offset to parse one message and advance by the parsed length until the remaining bytes are insufficient to form a complete message. The parser must be able to handle 0 / 1 / N messages corresponding to one segment Three cases; upon encountering a gap or a parse failure, stop and record at that position—do not skip the erroneous bytes and continue aligning (otherwise the misalignment will propagate silently). On the Scapy side you can directly use TCPSession to make C2 the class instantiate on the reassembled stream
- Compare with the raw bytes in Wireshark: pick a few frame start/end offsets,follow,tcp,raw to confirm the bytes match
State machine derivation (handshake / heartbeat / teardown)：
- Draw in chronological order the sequence of "who sends what message": client → server first message (connection establishment/handshake request) → response → subsequent business → periodic heartbeat → disconnect. The unit of the sequence isreassembled messages(steps 2/5), not TCP packets
- For each message type record: direction, type value, fields, trigger condition (what is received before what is sent)
- Verify the derivation: go back into the reassembled stream and look for counterexamples—whether there are messages that do not follow the "state machine," a connection that resets halfway through a message, or missing frames caused by a capture gap (if so, add states/transition conditions, or explain the exception branch); the final output is a state diagram (text or mermaid) + message format table, written into the report for reference by [[re-ioc]]

Cross-domain joint

[[re-protocol]]: this skill is step 5 of its workflow (state machine reconstruction)—the final output stage for cleartext traffic
[[re-malware]]: C2 protocol reconstruction—re-malware step 4 (netcap → proto-rev → crypto-*), the reconstructed protocol semantics (handshake/heartbeat/commands) are the basis for behavior judgment and IOCs
[[re-firmware]]: firmware communication protocol reconstruction—re-firmware step 6 (firmware call-home / custom protocol)
[[re-netcap]]: raw material source (cleartext pcap and tshark statistics)
[[re-crypto-id]] / [[re-crypto-decrypt]]: prerequisite—when traffic is ciphertext, decrypt first before entering this skill
Outputs (message format table / state diagram) go back to [[re-ioc]] for the report and YARA signatures

Common pitfalls and traps

Wrong byte order in the length field → parsing is all wrong: symptom—the parser reads the length as little-endian, the length value is huge/abnormal, and all subsequent fields are misaligned; cause—the protocol length field is big-endian (network order), but it was read as little-endian (host order); countermeasure—in step 4 try both big-endian and little-endian for the length field, decide with "is the length value reasonable (≤ the actual span of that frame in the reassembled stream)," and in the parser explicitly use !H/!I network order
Identify CRC/checksum before parsing: symptom—the checksum at the end of the packet is parsed as business data, fields are misaligned and messages do not match; cause—in an unknown protocol, checksums and data cannot be distinguished by name; countermeasure—if the last 2-4 bytes of the packet vary sensitively with the payload, first assume it is a checksum, identify it with crcmod/xor verification, and in the parser skip it or make it a separate field
Heartbeat message confusion: symptom—periodic heartbeats are treated as business messages, the state machine shows a false "one extra sent" branch, and the command sequence is thrown into disorder; cause—heartbeats have the same structure as business messages (same magic, same type prefix), and differ only in timing patterns; countermeasure—in step 1's io,stat find fixed-period small packets, mark them as heartbeats and handle them separately, and treat them in the state machine as an independent state transition
Direction confusion (C2 interaction mode): Symptom—the server-to-client direction is reversed, command messages are treated as responses, and downstream data is treated as reply packets; Cause—in C2, client-initiated polling (requests carrying commands) and server-initiated push (traffic even without requests) are mixed together; Countermeasure—first clarify the A/B roles by IP:port ownership, then tally by direction (step 1), combine with timing to confirm which side is the initiator, and annotate the direction of each transition in the state diagram
Scapy built-in layer binding hijacks custom payload: Symptom—custom protocol payloads are automatically parsed by Scapy into other built-in layers such as DNS, and all fields are garbled (e.g., a UDP 53 payload always parses to default values); Cause—bind_layers() default bindings (UDP/53→DNS, etc.) are matched first in guess_payload_class() ; Countermeasure—split_layers() remove the default bindings, have the custom protocol class override guess_payload_class() for explicit dispatch, or in the parsing script force construction with the custom class for every packet
Short packets/fixed-length fields silently degrade to Raw: Symptom—for some packets the parser outputs Raw with no error / fields are misaligned, and with more packets the error becomes hidden; Cause—when a packet is shorter than the minimum field length, Scapy silently treats the whole packet as Raw (no Wireshark-style malformed marker), andStrFixedLenField interacting with the packet cache also produces inconsistencies; Countermeasure—explicitly check before parsing that the payload length ≥ the sum of field lengths, record short packets separately instead of force-parsing; perform round-trip verification for each custom class (bytes(pkt) then parse back, compare against the on-wire bytes) before running in bulk
After reinjection, the peer reports an FCS error / incorrect frame length: Symptom—after reinjecting a captured frame, the peer's Wireshark reports "ETHERNET FRAME CHECK SEQUENCE INCORRECT" or the frame length differs from expectation; Cause—you must distinguish two lengths: Ethernet FCS is 4 bytes, whereasa typical Ethernet header is 14 bytes(6 destination MAC + 6 source MAC + 2 type)—"short by 14 bytes" points to a header/link-layer encapsulation problem, not the FCS; the capture library's frame abstraction (Padding handling, whether the FCS field is parsed, whether a trailer is included) varies by version and configuration, andcannot be broadly attributed to "not preserving the trailer"; Countermeasure—first measureexactly how many bytes are actually missingand then locate it: if 4 bytes are missing, check whether the FCS was stripped; if 14 bytes are missing, check whether the Ethernet header was lost/duplicated or injected at the wrong layer; for reinjection, preferraw bytesrather than a round-tripped packet, or inject at L3 on the local loopback (lo) to avoid the Ethernet layer
Read-only serialization function, layout confirmed on only one side: Symptom—the field order inferred from serialize does not match real traffic; Cause—packing/unpacking may be asymmetric (conditional fields, version branches); Countermeasure—cross-validate serialization/deserialization functions against each other(serialize vs parse); only when both directions match is it the true layout; protocol libraries usually contain both packing and unpacking functions
Missing protocol behavior anchors: Symptom — special commands (heartbeat/time correction) take an independent assembly path and get completely garbled when parsed with the generic frame structure; Cause — the hardcoded special-case strings/commands in the protocol library are the behavioral anchors; Countermeasure — grep the command strings in the native library and configuration (heartbeat/correcttime types), locate the special path first, then parse with the generic structure
Java client ignores the JNI contract: Symptom — no entry point can be found when reverse engineering the native library; Cause — the Java layer calls native through JNI, and statically registered methods will remain as Java_包名_类名_方法名 in the dynamic export table (strip does not remove .dynsym), which can serve as interface documentation; but RegisterNatives dynamically registered librarieshave no Java_* exports; Countermeasure —nm -D list the JNI exports; if present, the function names directly give the contract (division of labor for encode/parse/setKey) and reverse engineering starts from the exported functions; if absent, locate the entry point from the JNI_OnLoad/RegisterNatives call sites, JNI function table xrefs, strings, and runtime hooks ([[re-frida]])
IRC/text protocol fragmentation in the middle of words: Symptom — "half words" appear in protocol messages extracted from packet captures (e.g. "giant s", "undead eye"), and parsing materials/fields by message boundary causes misalignment; Cause — the IRC server (InspIRCd) hard-splits long messages at ~400 bytes (without respecting word boundaries), and after the client receives themconcatenate all fragments and re-split by delimiter; Countermeasure — handle it according to the client logic: concatenate adjacent messages (Join(msgs, "")) → then split by delimiter (, , etc.), rather than parsing each message boundary independently; TSHARK follow's ascii mode has a ~451-byte display truncation, so long data must be extracted in raw (hex) mode
Message accumulation trigger mechanism: Symptom — the protocol client processes long messages in fragments and only triggers complete processing under specific conditions; Cause — the client accumulates messages by an "end marker" (e.g. only processes the whole segment when it ends with ./? ), and intermediate fragments are merely buffered; Countermeasure — decompile the client message handler to find the accumulation condition (contains its own name + end character, etc.), and construct messages in that format when replaying, otherwise messages are silently dropped
Fake server replay to verify the protocol: Symptom — the protocol logic is complex and static reverse engineering is slow; Cause — the client program (bot/C2 beacon) logic is triggered during interaction; Countermeasure — forge a server (fake IRC/HTTP + DNS hooked to local) and replay the real sequence from the pcap to observe client behavior (messages sent/call chains), quickly verifying the understanding of the protocol
Web CAPTCHA/modern JS protocol: reverse-infer from the Verify interface: Symptom — there is no way to start directly reverse engineering the CAPTCHA encryption algorithm, or validation still fails after restoring it; Cause — all the data actually submitted by the CAPTCHA is encapsulated in a single field (e.g. captchaBody), and the algorithm is only the last step; Countermeasure — pipeline: capture the Verify request → locate the core field → instrument the JS to find the plaintext plain before SDK encryption → analyze the source of each plain field one by one (returned by an interface and no computation needed / gap recognition: download the image + OpenCV to find the gap and calculate the drag distance / browser environment collection generated in SDK format / trajectory coordinates) → once the plaintext is completely consistent, encryption is instead the simplest step (follow the SDK call chain to execute the same encryption flow as the browser); the core principle isthe input is completely consistent rather than the algorithm being equivalent
Compute the additional signature layer synchronously when protocolizing: Symptom — the core field (captchaBody) is restored correctly but Verify is still rejected; Cause — besides Verify, there is also a request protection signature automatically generated by the page SDK (omitted during protocol reimplementation); Countermeasure — confirm from the page SDK's complete call chain that all automatically generated parameters are included; for similar scenarios, first list all "non-manual fields included in the request" before implementing
Private protocol has no parser: Symptom — Wireshark only shows Data; Cause — the private protocol has no dissector; Countermeasure — write a Lua dissector (<100 lines) or do offline analysis in Python
Data has no pattern → suspect encryption first: Symptom — every frame is different; Cause — a compression/encryption layer; Countermeasure — entropy analysis (>7.5 → candidate for encryption/compression/high randomness, corroborated by frame length patterns and entropy distribution; high entropy is not a determination), and for the encryption layer find the nonce/IV fields; when length fields cannot be solved, set up equations by "length may be little/big-endian, includes/excludes itself" to solve the semantics
Replay rejected: Symptom — the data is correct but the server does not respond; Cause — the protocol carries an incrementing seq/nonce; Countermeasure — figure out how seq is computed (hash of the previous frame or an incrementing counter) and advance it correctly during reproduction
Closed-loop verification: Symptom — there is no criterion for whether the protocol restoration holds; Cause — static analysis cannot prove the restoration is correct by itself; Countermeasure — use the restored frame structure to write a client and send one frame; it only counts as valid if the server responds consistently (Source: reverse-skill field-journal, MIT)