re-ctf

CTF practice (challenge-type identification / automated solving / constraint solving)

Complete workflow

Execute in order; for each step's artifact (challenge-type conclusion / key function address / constraint model / flag), record the evidence path + sha256 (see [[re-triage]] (capability:triage)), for writeup citation.

1. Challenge-type identification — don't rush to use tools first; read the challenge + do initial reconnaissance to determine the challenge type (see pitfall 1):
   • Reverse challenges (rev / babyre): given a binary, required to recover the algorithm / find the input / find the flag — follow steps 2-4
   • Crypto / algorithm challenges: given an encryption function / ciphertext, required to solve for the plaintext or invert the algorithm — involves [[re-crypto-id]] (capability:crypto-identification) / [[re-crypto-decrypt]] (capability:crypto-decryption), use [[re-z3]] (capability:constraint-solving）
   • Serial number / registration-type challenges: recover the verification function → keygen (see [[re-license]] (capability:license-analysis) / [[re-keygen]] (capability:license-analysis), for brute-forcing the comparison chain use [[re-z3]] (capability:constraint-solving））
   • pwn / stack overflow challenges: for introductory exploitation go to [[re-pwn]] ([[re-gdb]] (capability:debugging) / [[re-radare2]] (capability:decompilation) dynamic debugging, run in [[re-sandbox]] (capability:sandbox-setup））
   • Initial reconnaissance commands:file / checksec / entropy / strings ([[re-triage]] (capability:triage)), identify the architecture, whether it is packed, whether it is statically linked
2. Simple challenges go directly to [[re-binary-core]] (capability:decompilation、debugging、memory-dump、elf-parser、pe-parser、macho-parser） — when the challenge does not require automation, follow general analysis (initial reconnaissance → format parsing → decompilation → debugging): [[re-ghidra]] (capability:decompilation、debugging) / [[re-ida]] (capability:decompilation、debugging) / [[re-radare2]] (capability:decompilation) decompile the main logic, manually recover the flag / registration algorithm.First estimate the complexity: a challenge with a few lines of comparison / simple XOR is enough to do manually in 15 minutes, and using angr is actually slower (see pitfall 2)
3. Requires automation → [[re-angr]] (capability:symbolic-execution) / [[re-z3]] (capability:constraint-solving）：
   • Input is verified byte by byte in a long loop / deep comparison chain (manual reverse derivation bit by bit is time-consuming and error-prone) → [[re-angr]] (capability:symbolic-execution） symbolic execution (symbolize input → find target address → solve)
   • Verification is "a set of mathematical equations / comparison chain" (satisfying the constraints means flag) → [[re-z3]] (capability:constraint-solving） model and solve (lighter, faster, and more stable than angr)
   • For dispatch details see "selection tree"
4. Obfuscation → [[re-deobfuscate]] (capability:deobfuscation） — the decompilation output has junk instructions / control flow flattening / string encryption:restore first, then automateRunning angr directly on the obfuscated function will cause path explosion / fail to solve (see pitfall 3); after restoration with [[re-deobfuscate]] (capability:deobfuscation), return to step 2/3
5. Verification and output：
   • The solved input must, when running the original program (inside the sandbox, [[re-analyze/platform-tips]] highest principle), print flag{...}(see pitfall 4: flag format / consistency with the solved result)
   • Record: challenge type, key function addresses, constraint model, solving script, flag, reproduction commands — writeup and [[re-ioc]] (capability:threat-intel) features (such as automated solving features)

When to use which atomic skill (selection tree)

By challenge-type features / target branch:

• Just got the challenge, type not yet determined → Step 1: [[re-triage]] (capability:triage) initial survey + decompile and glance at the main logic, judge manually first (most simple challenges can be done manually, don't rush to automate)
• Simple XOR / shifts / table lookup transforms (≤ one screen of pseudocode) → [[re-binary-core]] (capability:decompilation、debugging、memory-dump、elf-parser、pe-parser、macho-parser) manual restoration ([[re-ghidra]] (capability:decompilation、debugging) / [[re-radare2]] (capability:decompilation)), or small-scale constraints directly modeled with [[re-z3]] (capability:constraint-solving)
• Byte-by-byte / character-by-character long loop verification (every byte must satisfy the condition, manual reverse derivation is tedious) → [[re-angr]] (capability:symbolic-execution; symbolic input + find the address where verification passes)
• "Satisfying a set of equations / comparison chain yields flag / key" (no loop or loop already manually expanded) → [[re-z3]] (capability:constraint-solving; BitVec modeling + solver)
• Input location is complex (argv / file / stdin / network) → [[re-angr]] (capability:symbolic-execution; symbolize any input point, [[re-z3]] (capability:constraint-solving) requires manually locating the variable source first)
• Junk instructions / flattening / string encryption visible → first [[re-deobfuscate]] (capability:deobfuscation) restore, then proceed by challenge type with angr / z3 / manual (pitfall 3)
• Packed challenge(rare in CTF, common in "unpacking challenges") → first [[re-anti-analysis]] (capability:unpack、deobfuscation、evasion-analysis; packer-id → unpack-*), after unpacking return to this gateway
• Encryption / cryptography challenge(AES / RSA / custom encryption + ciphertext) → [[re-crypto-id]] (capability:crypto-identification) identify → [[re-crypto-decrypt]] (capability:crypto-decryption) restore; for custom mathematical relations use [[re-z3]] (capability:constraint-solving) to solve
• pwn challenge → [[re-pwn]] (capability:exploit-development; exploit basics: stack overflow / format string / ret2libc) → [[re-gdb]] (capability:debugging) / [[re-radare2]] (capability:decompilation) + [[re-sandbox]] (capability:sandbox-setup) dynamic debugging (this gateway focuses on reverse engineering challenges; pwn is handed off to the debugging domain as an adjacent challenge type)
• flag solved but format is incorrect → check case / newline / encoding (pitfall 4)

Cross-domain combination

• Entry dispatch: this gateway is called by the triage "CTF challenge" path of [[re-analyze]] (re-ctf → challenge type identification → re-angr / re-z3 / deobfuscation re-deobfuscate → foundation re-binary-core)
• CTF challenges = application of core skills (spec 2.4 combined scenario): CTF challenges are essentially combined applications of core skills → simple challenges / decompilation workbench go directly through [[re-binary-core]] ([[re-triage]] / [[re-ghidra]] / [[re-ida]] / [[re-radare2]]); the [[re-binary-core]] gateway also declares "CTF reverse engineering challenges → this gateway serves as the foundation"
• Automated solving: [[re-angr]] (symbolic execution), [[re-z3]] (constraint solving) — the two atomic skills of this gateway, dispatched by step 3 of this gateway's workflow; step 1 of the [[re-deobfuscate]] workflow also produces a clean binary that can be consumed by angr
• Obfuscation countermeasures: For obfuscated challenges, first restore with [[re-deobfuscate]] (its "cross-domain joint" declaration states that CTF deobfuscation challenges always reference this gateway)
• Cryptography-type challenges: [[re-crypto-id]] / [[re-crypto-keys]] / [[re-crypto-decrypt]] (algorithm identification, key locating, decryption and restoration); use [[re-z3]] for key derivation modeling
• Serial number-type challenges: [[re-license]] / [[re-keygen]] (verification locating and keygen); use [[re-z3]] for brute-forcing comparison chains
• pwn / dynamic: [[re-gdb]] (debugging / breakpoints to inspect verification branches), [[re-radare2]] (command-line rapid attacks), [[re-sandbox]] (sandbox for running and flag verification, the highest principle of [[re-analyze/platform-tips]])
• Outputs: Automated solving script characteristics / challenge-type fingerprints can go into the [[re-ioc]] list

Common pitfalls and traps

• Going straight to angr without examining the challenge: Symptom — symbolic execution applied to a simple XOR challenge / plaintext comparison challenge, solving takes forever or yields no solution; Cause — misjudging the challenge type, automated tools instead introduce path explosion / modeling errors; Countermeasure — in step 1, first do reconnaissance + decompilation to estimate complexity: challenges that can be manually restored within one screen go directly to [[re-binary-core]], leave automation for long loops / deep comparison chains
• Blindly reversing long loops by hand: Symptom — byte-by-byte verification with hundreds of iterations, manual reversing goes wrong midway and time spirals out of control; Cause — underestimating the scale of mechanized computation; Countermeasure — dispatch according to step 3: use [[re-angr]] for long byte-by-byte loops, use [[re-z3]] for sets of equations; verify while writing the solving script (test with small inputs first)
• Automating before deobfuscating: Symptom — angr directly runs flattened / junk-instruction-laden functions, path explosion prevents completion, or the solved "input" does not pass when run against the original program; Cause — obfuscation distorts the constraint model and expands the state space; Countermeasure — first restore with [[re-deobfuscate]] (D-810 / manual), then symbolic execution or manual analysis; in angr, confirm the symbolized function decompiles clearly before running (see pitfall 1 of [[re-angr]])
• flag format / boundary inconsistency: Symptom — an apparently correct input is solved, but the program outputs non-flag content or errors out; Cause — the flag has prefix/suffix format (flag{...}), case / newline / encoding (ASCII vs wide characters) inconsistency, or the input length is fixed and needs alignment; Countermeasure — use [[re-gdb]] to break at the print location and see the actual output; replay the solving script exactly according to the input channel (stdin / argv / file) and format the program expects, and run the original program in the sandbox for verification
• Missed detection of packing / static linking: Symptom — strings are all garbled, xrefs are all packer code, decompilation gives no clue; Cause — the challenge is packed (unpacking challenge) or statically linked (no import table clues); Countermeasure — [[re-triage]] reconnaissance first checks for packing (entropy / section names / EP); for packing go to [[re-anti-analysis]]; for statically linked challenges, locate the logic starting point from main / entry-point characteristics / strings (the library fingerprint approach of [[re-imports]])
• PyInstaller challenge: entry script has no behavior ≠ unpacking failed: Symptom — after unpacking, main.pyc only has sys.exit(0), but the challenge statement hints completely fail to match; the reason — the real logic is not in the entry script: it may be hidden in a runtime hook,frozen module（_frozen array, such as a modified version of the os module), a modified Python runtime library (a recompiled python313.dll), bootloader pre-logic, or a code object generated at runtime; countermeasure —competing hypotheses + minimal experiment: hypothesis 1 = a modified frozen module, hypothesis 2 = bootloader pre-logic, first run the experiment with the lowest cost and highest discriminating power (download the official DLL of the same version for comparison);frozen module hash diffing: parse the PE export table and read the _frozen array (code/size), after marshal.loads compare SHA-256 one by one against the official 29 frozen modules, compressing the 7MB runtime library down to a single 47KB marshal object; note that the marshal format and bytecodeversion are strongly correlated(must use the Python version corresponding to the challenge, such as the magic of 3.13.1)
• periodic XOR + known-plaintext key recovery: phenomenon — ciphertext from a 6-character key periodic XOR, brute-force search yields no printable candidate; reason — when each slot of the key is covered by plaintext an insufficient number of times, it cannot be solved; countermeasure — use naturally known plaintext (such as a plaintext success message inside the program),密文 ^ 明文 to obtain the key stream,count the coverage count of each slot(only trustworthy if each slot is consistent at least 2 times); after recovery, cross-validate by decrypting the success message