gau-urlfetch

gau URL History Collection Methodology

gau (getallurls) collects known URLs from historical data sources. Core advantages:Aggregation of Four Major Data Sources（Wayback/CommonCrawl/OTX/URLScan）+ Purely Passive + Pipeline-Friendly。

Project address:https://github.com/lc/gau

Phase 1: Basic Usage

# 收集目标域名的所有已知 URL
gau target.com

# 包含子域名
gau --subs target.com

# 输出到文件
gau target.com -o urls.txt

Filter Static Resources

gau --blacklist png,jpg,gif,svg,woff,ttf,ico,css target.com

## Phase 2: 数据源控制

```bash
# 指定数据源
gau --providers wayback target.com
gau --providers wayback,commoncrawl target.com

# 时间范围
gau --from 202301 --to 202612 target.com

# 控制线程
gau --threads 10 target.com

# 从 stdin
echo target.com | gau
cat domains.txt | gau

Phase 3: Pipeline Integration

# URL 收集 → 提取带参数的 URL → XSS 扫描
gau target.com | grep "=" | sort -u | dalfox pipe --silence

# URL 收集 → 提取 JS 文件
gau target.com | grep "\.js$" | sort -u

# URL 收集 → 提取 API 端点
gau target.com | grep -E "/api/|/v[0-9]/" | sort -u

# URL 收集 → 存活检测
gau target.com | httpx -silent

# URL 收集 → 敏感路径
gau target.com | grep -iE "admin|backup|config|\.env|\.git|debug"

Common Scenarios Quick Reference

Scenario	Command
Full Collection	gau --subs target.com -o all_urls.txt
Parameter URLs	gau target.com | grep "=" | sort -u
JS Files	gau target.com | grep "\\.js$" | sort -u
API Discovery	gau target.com | grep -E "/api/" | sort -u