SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsre-tls

re-tls

Data & analytics

View repository

Updated 10-15 21:35

About this skill

TLS/Encrypted Traffic Deep Analysis

When to Use / When Not to Use

  • Use: when TLS/HTTPS traffic needs to be "decrypted to see plaintext" (with keylog or man-in-the-middle) — C2 communication, firmware callbacks, encrypted application protocols
  • Use: when only ciphertext pcaps are available but depth is still needed — ClientHello/certificate fingerprint clustering, anomalous TLS feature determination
  • Use: certificate chain validation/certificate content analysis (openssl x509)
  • Don't use: when only capturing packets without going deeper (that's [[re-netcap]])
  • Don't use: when traffic is already decrypted and there is no TLS layer (go directly to [[re-proto-rev]] to rebuild the state machine)
  • Don't use: for custom encryption that is not standard TLS (go through [[re-crypto-id]] → [[re-crypto-keys]] → [[re-crypto-decrypt]])
  • Note: this skill handles traffic and key files and does not run samples, so by default the sandbox can be skipped; the packet capture step should be performed in an isolated environment per [[re-netcap]] ([[re-analyze/platform-tips]] highest principle)

Tool Preparation

Verify all tools before use.

tshark / wireshark — the main workhorse for parsing, fingerprinting, and decryption (installation see [[re-netcap]])

  • Linux: apt install wireshark tshark / dnf install wireshark-cli wireshark / pacman -S wireshark-cli wireshark-qt
  • macOS: brew install --cask wireshark(includes tshark CLI); Windows: choco install wireshark
  • Verify: tshark --version；tshark -r out.pcap -Y 'tls.handshake.type == 1' | head -1 can parse ClientHello
  • JA3 field (built into Wireshark 3.6+) and JA4 field (native in Wireshark 4.2+, client only): verify tshark -r out.pcap -T fields -e tls.handshake.ja4 | head -1 produces output; for local field existence, use tshark -G fields | grep -iE 'ja3|ja4' as the standard

openssl — certificate parsing and TLS interaction (cross-OS)

  • Linux: apt install openssl / dnf install openssl / pacman -S openssl(mostly preinstalled)
  • macOS: brew install openssl(keg-only, add to PATH /opt/homebrew/opt/openssl@3/bin(Apple Silicon) or /usr/local/opt/openssl@3/bin（Intel））
  • Windows: choco install openssl.light(PATH must be added manually, see package instructions) or Git for Windows's bundled openssl.exe
  • Verify: openssl version

python3 — fingerprint clustering/batch analysis scripts (optional)

  • For installation and verification see [[re-proto-rev]] Tool Preparation (python3)

SSLKEYLOG environment — Chrome/Firefox key export (not a standalone tool)

  • Chrome: set the environment variable before launch SSLKEYLOGFILE=/tmp/keys.log(Linux/macOS) or set SSLKEYLOGFILE=C:\keys.log（Windows）
  • Firefox: same environment variable (restart the browser after setting)
  • Wireshark: Edit → Preferences → Protocols → TLS → (Pre)-Master-Secret log filename, point it to that file; on the tshark side use -o tls.keylog_file:...
  • Verify: after setting it, visit an HTTPS site and keys.log will show CLIENT_HANDSHAKE_TRAFFIC_SECRET and similar lines (TLS 1.3) or CLIENT_RANDOM lines (TLS 1.2)

mitmproxy — man-in-the-middle decryption (optional, installation see [[re-netcap]])

  • Scenario: third-party programs without keylog access (self-implemented TLS stack, certificate pinning see pitfall 4)
  • Verify: mitmdump --version

Operating Steps

Execute in order, and archive each step's output (fingerprint table/certificate/decrypted traffic) with sha256 + path ([[re-ioc]] evidence chain requirement).

  1. ClientHello fingerprinting (JA3/JA4)：

    # JA3（ClientHello 协商参数哈希，Wireshark 3.6+ 内置）
tshark -r out.pcap -Y 'tls.handshake.type == 1' -T fields \
  -e tls.handshake.ja3 -e ip.src -e tls.handshake.extensions_server_name \
  | sort | uniq -c | sort -rn
# JA4（Wireshark 4.2+ 原生，客户端侧；含协议/版本/SNI/密码套件数/ALPN 维度）
tshark -r out.pcap -Y 'tls.handshake.type == 1' -T fields -e tls.handshake.ja4 \
  | sort | uniq -c | sort -rn
    • Purpose: cluster by "client fingerprint" — the same fingerprint = the same client stack (browser/library/malicious tool), a new cluster = suspicious client
    • Cross-reference: compare against public JA3/JA4 intelligence sources to hit known malicious clients; no hit = weak signal, requires step 5 synthesis (pitfall 3)
    • Field note: SNI (tls.handshake.extensions_server_name) clustered together with IP — the same fingerprint connecting to different targets is also a signal

  2. Certificate Chain Analysis (openssl x509 Parsing)：

    # 一个 Certificate 消息常含多张证书，该字段是逗号分隔的十六进制串（hex 带冒号）——先按逗号切行再逐行转 DER
# 链的次序即发送次序，第一行是叶子（服务器）证书；只取叶子在切行后加 | head -1（放在 tr -d ':' 之前）
tshark -r out.pcap -Y 'tls.handshake.type == 11' -T fields -e tls.handshake.certificate \
  | head -1 | tr ',' '\n' | tr -d ':' | xxd -r -p > cert_chain.der
openssl x509 -in cert_chain.der -inform DER -text -noout | head -60
# 逐张分别导出（cert_1.der、cert_2.der…）用循环或 csplit 按行切开；
# 不要一次 xxd 把整条链写进同一个文件——多张 DER 拼接不是一个合法的 DER 对象，openssl 只读第一张且可能报错
# 实时抓取（对已知主机）
openssl s_client -connect <host>:443 -servername <host> -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -text
# 证书链校验结果
openssl s_client -connect <host>:443 -servername <host> -verify_return_error </dev/null 2>&1 \
  | grep -i 'verify return'
    • Analysis Points: Issuer/Subject (CN/SAN), Validity Period (Anomalous Dates/Expiration), Self-Signed vs Trusted CA, SPKI/Fingerprint (openssl x509 -noout -fingerprint -sha256), Certificate Reuse (Same Certificate Across Multiple C2 Domains = Batch Issuance)
    • Note: TLS 1.3's Certificate Message (type 11) Is Inside the Encrypted Handshake,-Y 'tls.handshake.type == 11' Cannot Be Matched—TLS 1.3 Traffic Requires Prior Decryption per Step 3 (SSLKEYLOG) or Real-Time Capture with openssl s_client
    • Platform Note: </dev/null Is POSIX Syntax, Not Applicable to Windows cmd—on Windows Use echo | openssl s_client ... or Remove </dev/null to Exit Interactively
    • Malicious Characteristics: Self-Signed + Newly Registered Domain + Short Validity Period (90 Days-1 Year Batch Automated Issuance Is Common)

  3. SSLKEYLOG Decryption (TLS 1.2/1.3 Key Export)：

    # tshark 侧解密（keylog 文件 + 重新解析）
tshark -r enc.pcapng -o tls.keylog_file:keys.log -Y 'http' \
  -T fields -e http.host -e http.request.uri
# 或 GUI: 首选项 → Protocols → TLS → (Pre)-Master-Secret log filename
    • TLS 1.2: keylog Records CLIENT_RANDOM <hex> <master secret>—Master Secret Can Directly Decrypt All Records in This Session
    • TLS 1.3: keylog Records the traffic secret of Each Phase (CLIENT_HANDSHAKE_TRAFFIC_SECRET / SERVER_HANDSHAKE_TRAFFIC_SECRET / CLIENT_TRAFFIC_SECRET_0 / SERVER_TRAFFIC_SECRET_0 / EXPORTER_SECRET)—There Is No Master Secret as in 1.2; Manual Derivation Uses HKDF-Expand(-Label) from Each traffic secret to Derive Record Keys (Extract Is an Earlier-Phase Operation, See Pitfall 1); Wireshark/tshark Reading the keylog File Directly Can Automatically Handle Both Protocol Generations
    • Verify Successful Decryption: After Decryption, Plaintext HTTP/Application Protocol Fields Are Visible (Raw Material for Semantic Restoration in Step 4); If Decryption Fails, First Check Whether keys.log Contains the CLIENT_RANDOM or traffic secret Entry for That Session (Pitfall 2)
    • Real-Time Capture Scenario: After Setting SSLKEYLOGFILE, Start Chrome/Firefox (Tool Preparation); Packet Capture and keylog Are Generated Synchronously

  4. Encrypted Traffic Semantic Restoration (Application-Layer Protocol Recovery)：

    # 解密后按应用层协议还原（TLS 只是载体，语义在 ALPN/上层协议）
tshark -r decrypted.pcapng -Y 'http' -T fields -e http.request.method -e http.host -e http.request.uri
tshark -r decrypted.pcapng -Y 'tls.handshake.extensions_alpn_str' -T fields -e tls.handshake.extensions_alpn_str | sort | uniq -c   # ALPN 应用协议：ClientHello 里是客户端候选列表（offered）；最终协商结果（negotiated）TLS ≤1.2 在 ServerHello、TLS 1.3 在 EncryptedExtensions
    • Process: Decryption (Step 3) → Identify Upper-Layer Protocol (ALPN: http/1.1, h2, Custom; or by Port/Characteristics) → Session Reassembly → Semantic Extraction (C2 Commands/Configuration/Plaintext Credentials)
    • offered and negotiated Must Be Distinguished: tls.handshake.extensions_alpn_str Matches Both ClientHello (Client Candidate List) and the Handshake Message Carrying the Final Result; Direct Statistics Will Treat Candidates as Conclusions. TLS 1.3's ALPN Result Is Not in ServerHello—ServerHello Only Returns the Extension Type; the Actual ALPN Extension Is in EncryptedExtensions (Visible Only After Handshake Key/keylog Decryption); When Undecrypted, This Field Can Only Show the ClientHello Candidates. To Determine the Final Protocol, Take the negotiated Side, or Directly Look at the Decrypted Application-Layer Record Type
    • Custom Protocol: After Plaintext Restoration, Convert to [[re-proto-rev]] for State Machine Reconstruction ([[re-protocol]] Workflow Step 5)
    • Boundary: TLS Decryption Only Unlocks Traffic of the "Standard TLS Stack"—Self-Implemented/Modified TLS (No ClientHello Structure, No Corresponding Entry in keylog) Fails to Decrypt and Can Only Go Through Characteristic Analysis (Step 5)

  5. Malicious TLS Characteristics (Fingerprint Clustering/Anomalous Certificates)：

    • Characteristic Checklist: ① Fingerprint Clustering—JA3/JA4 New Clusters or Hits on Known Malicious Libraries (Native Stacks of curl/python requests Are Also Common Signals) ② Anomalous Certificates—Self-Signed/Short Validity Period/SNI Mismatch with IP/Certificate CN Is an IP ③ Beacon Rhythm—Fixed-Interval Short Connections (Align with [[re-behavior]] Timeline) ④ SNI Anomalies—Random Subdomains/High-Entropy SNI/No SNI (Hardcoded IP Direct Connection, Same Origin as [[re-netcap]] DoH Pitfall) ⑤ Mixed Protocols—Non-TLS Running on 443
    • Clustering method: fingerprint table from step 1 + session statistics (connection duration/packet size/direction ratio) layered; baseline comparison against normal browser traffic (capture a baseline segment inside the sandbox first)
    • Conclusion discipline: fingerprints and certificates are both "weak signals" and can be forged (pitfall 3) — final conclusions must be cross-validated on the behavioral side (process attribution/beacons/download behavior), and the signal type must be annotated when entering IOCs
    • Output: malicious TLS session list (time/IP/fingerprint/certificate/SNI) + features into [[re-ioc]] YARA (certificate bytes/fingerprint patterns)

Cross-domain correlation

  • [[re-netcap]]: capture and filtering (the input material for this skill; for capture strategies in DoH/encrypted traffic blind spots see its pitfalls)
  • [[re-proto-rev]]: state machine reconstruction on decrypted plaintext traffic (downstream of step 4 of this skill)
  • [[re-protocol]]: this skill is the TLS specialization of its gateway — the deep analysis branch for standard TLS traffic; custom encryption goes through the crypto trio
  • [[re-crypto-id]] / [[re-crypto-keys]] / [[re-crypto-decrypt]]: non-standard/self-implemented encrypted channels (the transition when TLS fingerprinting and decryption fail)
  • [[re-malware]]: TLS feature determination for malicious C2 (refinement of step 4 of callback traffic analysis)
  • [[re-frida]]: certificate pinning bypass (pitfall 4) — hook the verification function together with mitmproxy
  • [[re-sandbox]]: packet capture environment isolation and baseline traffic collection ([[re-analyze/platform-tips]] highest principle)
  • References [[re-analyze/platform-tips]] static-first approach and the sandbox network isolation branch

Common pitfalls and traps

  • TLS 1.3 key export differs from 1.2: Symptom — using the TLS 1.2 "CLIENT_RANDOM → master secret" approach to manually decrypt TLS 1.3 traffic fails, and there is no master secret entry in keys.log either; Cause — TLS 1.3 forward secrecy + per-session independent derivation, and what keylog records is the traffic secret (CLIENT_HANDSHAKE_TRAFFIC_SECRET, etc.), with no master secret/RSA premaster; Countermeasure — hand the keylog file directly to Wireshark/tshark (-o tls.keylog_file:...) to handle both protocol generations automatically; when deriving manually, use HKDF-Expand(-Label) from each traffic secret to derive the record keys for each phase (handshake and application phase keys differ)
  • Forward secrecy (cannot be decrypted without keylog): Symptom — only a pcap and no keys, and no matter what, the plaintext cannot be decrypted; Cause — the ECDHE session key exists only in the memory of both ends of the session, not in any static location; Countermeasure — acknowledge that it cannot be decrypted and switch to fingerprint/metadata analysis (SNI/certificate/length/timing, step 5); prepare two paths in advance: set SSLKEYLOGFILE before capture (self-controlled client) or mitmproxy man-in-the-middle (third-party program, combined with certificate pinning bypass from pitfall 4)
  • Fingerprints can be forged: Symptom — JA3/JA4 clustering groups a malicious client into the "normal browser" cluster, or two unrelated clients share the same fingerprint; Cause — the fingerprint only reflects ClientHello negotiation parameters, and curl/requests/malicious code can customize and imitate them (fingerprint spoofing is a routine technique of C2 infrastructure); Countermeasure — treat fingerprints as weak signals for clustering and correlation, not for identity determination; whether hit or miss, cross-validate with certificates, SNI, and the behavioral side (process attribution/beacons), and note the evidence strength in the conclusion
  • Certificate pinning bypass requires man-in-the-middle cooperation: Symptom — after mitmproxy man-in-the-middle, the target program refuses to connect / certificate error, and plaintext cannot be obtained; Cause — the application embeds a public key/SPKI fingerprint for certificate pinning and does not trust the system CA; Countermeasure — use [[re-frida]] to hook the verification function (SSL_CTX_set_verify / X509_verify_cert / self-implemented pin verification) to return success, or patch the program to skip verification, then route through mitmproxy to obtain plaintext; operate inside a sandbox ([[re-sandbox]]), and note the bypass method and version in the conclusion
  • ECH (extension 65037, encrypted ClientHello) and latency-based proxy detection: Symptom — JA3/JA4 fingerprints are normal but requests are still rejected, or the fingerprint can be imitated but the proxy is identified as soon as it is enabled; Cause — anti-crawling escalation: ①ECH (Encrypted Client Hello, extension 65037) encrypts the SNI/extensions into the ClientHello, so fingerprint analysis cannot obtain the SNI (as of 2025+ it has appeared in the anti-crawling detection chain at verify=3); ②proxy detection does not use blacklists/whitelists/reputation scores/behavior analysis, but directlymeasure request latency(server round-trip time difference) to identify VPNs/proxies — because going through a proxy/tunnel necessarily introduces extra latency, and fingerprints can be forged but latency cannot be eliminated. When evaluating data, pay attention to the metric definitions: under normal conditions, the precision/recall of single-connection classification can reach 99%, and its comprehensive F1 is about 92%(F1 is the harmonic mean of precision and recall, not equal to "detection rate", nor is it "accuracy"); while simple traffic-scheduling evasion can reduce the recall of traditional RTT detection from 99% to 8% — this is exactly why latency-based detection needs more robust methods; Countermeasure — analysis side: packet capture tools/man-in-the-middle need to support ECH decryption (the key is on the server side and in the browser ECH key configuration); evasion side: latency-based detection cannot be bypassed by forging fingerprints, so one can only switch to direct connections/low-latency tunnels, and during analysis first confirm whether the target uses fingerprint or latency detection