yakit-data-extract-plugin

SKILL: Yakit historical data extraction and dictionary-driven scanning

AI LOAD INSTRUCTION: This skill targets a specific pain point—"a bunch of traffic for a domain has piled up in History, I want to extract all unique paths, obtain the login Cookie, and then continue scanning/Fuzzing." It provides two end-to-end runnable examples: extract traffic by domain, deduplicate, and extract the Cookie; feed deduplicated paths into a Payload dictionary and perform real request-sending scanning with the login state. Both examples yak <file> self-test passed, and the scanning example starts a real local target site and sends real HTTP requests as evidence.

0. Related routes

Main entry:yak
Database API (KV / Payload / query):yaklang-database
If you want to collect directly during the MITM stage instead of retrieving from the database afterward:mitm-hotpatch
Training material cross-references:yaklang-ai-training-materials/library-usage/db/db-practice.yak

1. Pain point breakdown

A common workflow gets stuck on "data shuttling":

flowchart LR
    H[History 大量流量] --> Q[按 domain 捞取]
    Q --> D[URI 去重]
    Q --> C[提取登录 Cookie]
    D --> P[沉淀 Payload 字典]
    P --> S[带 Cookie 逐路径发包扫描]
    C --> S

Doing this manually is painful: duplicate paths, query interference with deduplication, Cookie scattered across different requests, and forgetting to carry the login state during scanning. This skill uses db + poc to script every step.

2. Retrieve History by domain + URI deduplication + extract Cookie

Core data forms (verify first, do not speculate):

Source	Form
`db.QueryHTTPFlowsByKeyword(domain)`	Returns an `*HTTPFlow` iterator (channel)
`flow.Url`	Full URL string
`flow.Request`	`codec.StrconvQuote` the string after, requiring `codec.StrconvUnquote` to get the raw bytes
Login Cookie	`poc.GetHTTPPacketHeader(req, "Cookie")`

collectFromHistory = func(domain) {
    pairs = []
    for flow in db.QueryHTTPFlowsByKeyword(domain) {
        req = codec.StrconvUnquote(flow.Request)~
        pairs = append(pairs, {"url": flow.Url, "req": []byte(req)})
    }
    return analyzePairs(pairs)   // 归一化去重 + 收集 cookie
}

The deduplication key uses "host+path after removing scheme and query", avoiding ?page=1 / ?page=2 being treated as two entries.

Complete example:examples/collect-domain-urls.yak

3. Feed deduplicated paths into Payload dictionary + send requests with login state for scanning

Distill the paths into a Yakit Payload dictionary (persisted, reusable in Web Fuzzer / scanner), then send requests one by one with the Cookie, using status codes to distinguish alive/nonexistent:

db.SavePayload(group, dedupPaths(paths))~        // 去重后入库
for path in db.YieldPayload(group) {             // 迭代字典逐条扫描
    raw = sprintf("GET %s HTTP/1.1\r\nHost: %s\r\nCookie: %s\r\nConnection: close\r\n\r\n", path, hostport, cookie)
    rsp, _ = poc.HTTP(raw, poc.timeout(5), poc.save(false))~
    code = poc.GetStatusCodeFromResponse(rsp)    // 200 存活 / 404 不存在
}

Complete example:examples/path-scan-with-payload.yak (The self-test starts a real target site on a random local port, actually sends HTTP requests, and verifies status codes and Cookie passthrough)

4. Pitfalls (verify before writing)

flow.Request / flow.Response is a string after strconv-quote; using it directly as the original text will include escaping; be sure to codec.StrconvUnquote(...)~。
db.YieldPayload(group) directly produces the payload content string, not an object with a .Content field; directly use p。
db.SavePayload writes to the profile persistent store; in scripts use a group with a unique prefix and defer db.DeletePayloadByGroup(group) clean up to avoid contamination.
When deduplicating, always strip the query first, otherwise different pages of the same endpoint will be treated as different URIs.
When sending requests for scanning, be sure to poc.timeout(...) + poc.save(false)(do not write to the database during self-test/batch runs), to avoid hangs and contaminating History.

5. How to verify the extraction/scanning script you wrote

Directly yak your-script.yakto let the YAK_MAIN self-test block run the logic through with synthetic data + a local target site.
To verify real History retrieval, you can first use mitm-hotpatch to collect a few traffic entries and then run it.
For the toolchain and command-line verification methods, see yaklang-toolchain。