apify-osint-threat-intel

OSINT Threat Intelligence

Real-time security intelligence powered by live threat data via Apify actors. Never answer security questions from training knowledge alone. CVEs, breaches, and threat actor activity change daily — always gather live data first, then analyze.

Prerequisites

Apify CLI v1.5.0+: npm i -g apify-cli
Authenticated: apify login or export APIFY_TOKEN=your_token
Token: https://console.apify.com/settings/integrations

CLI rules (always follow)

Always pass --user-agent apify-awesome-skills/apify-osint-threat-intel on every apify CLI call — it's critical for telemetry, never omit it.

apify actors call "ACTOR_ID" -i 'INPUT_JSON' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null
apify datasets get-items DATASET_ID --format json --user-agent apify-awesome-skills/apify-osint-threat-intel > /tmp/results.json 2>/dev/null
jq '.[] | "\(.field1) | \(.field2)"' /tmp/results.json
apify actors info "ACTOR_ID" --input --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null   # check schema

Actor Routing Table

Data Need	Actor ID	Notes
CVE lookup	`apify/google-search-scraper`	Query: `site:nvd.nist.gov [product] [version]`
NVD full record	`apify/website-content-crawler`	URL: `nvd.nist.gov/vuln/detail/CVE-XXXX-XXXXX`
CISA known exploited	`apify/rag-web-browser`	URL: `cisa.gov/known-exploited-vulnerabilities-catalog`
GitHub advisories	`apify/rag-web-browser`	URL: `github.com/advisories?query=[product]`
Exploit-DB search	`apify/google-search-scraper`	Query: `site:exploit-db.com [product] [version]`
Security news	`data_xplorer/google-news-scraper-fast`	Keywords: `"[target]" vulnerability OR exploit OR breach`
Reddit threat discussion	`harshmaur/reddit-scraper`	`searchTerms` + `withinCommunity` — one subreddit per run (`netsec`, then a second run for `cybersecurity`); a value like `netsec OR cybersecurity` silently drops the filter and searches all of Reddit. Always set `postedAfter` (`YYYY-MM-DD`) for recency — `searchTime` is not enforced and the Actor pads the cap with years-old posts. Pay-per-event: $0.02 per run + $0.002 per post; `maxPostsCount` is per search term.
Threat intel Twitter/X	`apidojo/tweet-scraper`	Keywords: `#threatintel [target]`, search mode
Breach mention search	`apify/google-search-scraper`	Query: `"[domain]" site:pastebin.com OR intext:breach`
Vendor security advisory	`apify/website-content-crawler`	Direct vendor security page URL
Shodan exposure hints	`apify/google-search-scraper`	Query: `site:shodan.io "[domain OR org name]"`
Threat actor research	`apify/rag-web-browser`	MITRE ATT&CK: `attack.mitre.org/groups/`

Prefer apify/google-search-scraper and apify/rag-web-browser over website-content-crawler for speed.
Use website-content-crawler only when you need the full page body (e.g. NVD detail, vendor advisory).
Do NOT use website-content-crawler on: reddit.com, twitter.com, pastebin.com, linkedin.com.

Core Workflow

Step 0 — Clarify scope before running anything

Ask the user:

Target type: domain, IP, software/version, CVE ID, threat actor name, or keyword?
Goal: one-time lookup vs. ongoing monitoring brief?
Autonomy: full autopilot, or checkpoint before each actor call?

Step 1 — Identify module

User says	Module	Steps
"Find CVEs for [product]"	CVE Intelligence	2a
"Is [domain] breached / exposed"	Domain Threat Profile	2b
"Research [threat actor / malware]"	Threat Actor Profile	2c
"Security news about [topic]"	Security News Brief	2d
"Attack surface of [company]"	Attack Surface Discovery	2b + 2d
"Full threat report on [target]"	Multi-Module	2a + 2b + 2c + 2d

Step 2a — CVE Intelligence

Gather live CVE data for a product or version:

# 1. Search NVD via Google
apify actors call "apify/google-search-scraper" -i '{
  "queries": "site:nvd.nist.gov CVE [PRODUCT] [VERSION]",
  "maxPagesPerQuery": 1
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 2. Pull full NVD record for each CVE ID found
apify actors call "apify/website-content-crawler" -i '{
  "startUrls": [{"url": "https://nvd.nist.gov/vuln/detail/CVE-XXXX-XXXXX"}],
  "proxyConfiguration": {"useApifyProxy": true},
  "maxCrawlPages": 1
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 3. Check if CVE is in CISA's Known Exploited Vulnerabilities list
apify actors call "apify/rag-web-browser" -i '{
  "query": "[CVE-ID] site:cisa.gov/known-exploited-vulnerabilities-catalog",
  "maxResults": 3
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 4. Check Exploit-DB for public PoC
apify actors call "apify/google-search-scraper" -i '{
  "queries": "site:exploit-db.com [PRODUCT] [VERSION]",
  "maxPagesPerQuery": 1
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

Synthesize: severity (CVSS), exploitability (CISA KEV = active exploitation), public PoC exists (yes/no), patch available (yes/no).

Step 2b — Domain Threat Profile

# 1. Search for breach mentions
apify actors call "apify/google-search-scraper" -i '{
  "queries": "\"[DOMAIN]\" breach OR leak OR hacked OR \"data exposed\"",
  "maxPagesPerQuery": 1
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 2. Check paste sites for credential leaks
apify actors call "apify/google-search-scraper" -i '{
  "queries": "\"[DOMAIN]\" site:pastebin.com OR site:ghostbin.com OR site:rentry.co",
  "maxPagesPerQuery": 1
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 3. Check Shodan exposure hints via Google
apify actors call "apify/google-search-scraper" -i '{
  "queries": "site:shodan.io \"[DOMAIN OR ORG]\"",
  "maxPagesPerQuery": 1
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 4. Scan r/netsec for mentions — one subreddit per run; repeat with "withinCommunity": "cybersecurity"
#    postedAfter = today minus 365 days (YYYY-MM-DD). 3 terms × 5 posts = 15 posts ≈ $0.05.
#    Use `postUrl` as the Source and `createdAt` for the date stamp.
apify actors call "harshmaur/reddit-scraper" -i '{
  "searchTerms": ["[DOMAIN] breach", "[DOMAIN] hack", "[DOMAIN] vulnerability"],
  "withinCommunity": "netsec",
  "postedAfter": "[YYYY-MM-DD]",
  "maxPostsCount": 5,
  "crawlCommentsPerPost": false
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

Step 2c — Threat Actor Profile

# 1. MITRE ATT&CK lookup
apify actors call "apify/rag-web-browser" -i '{
  "query": "[THREAT ACTOR NAME] site:attack.mitre.org",
  "maxResults": 3
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 2. Recent activity via news
apify actors call "data_xplorer/google-news-scraper-fast" -i '{
  "keywords": ["[THREAT ACTOR NAME] attack OR campaign OR malware"],
  "timeframe": "30d",
  "maxArticles": 15
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 3. Community threat intel on Twitter/X
apify actors call "apidojo/tweet-scraper" -i '{
  "searchTerms": ["#threatintel [THREAT ACTOR]", "[THREAT ACTOR] TTPs"],
  "maxItems": 20,
  "sort": "Latest"
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 4. Reddit discussion — postedAfter = today minus 365 days; `createdAt` of the newest post = "Last seen"
apify actors call "harshmaur/reddit-scraper" -i '{
  "searchTerms": ["[THREAT ACTOR NAME]"],
  "withinCommunity": "netsec",
  "postedAfter": "[YYYY-MM-DD]",
  "maxPostsCount": 10,
  "crawlCommentsPerPost": false
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

Step 2d — Security News Brief

# 1. Google News for topic
apify actors call "data_xplorer/google-news-scraper-fast" -i '{
  "keywords": ["[TOPIC] vulnerability OR CVE OR breach OR exploit"],
  "timeframe": "7d",
  "maxArticles": 20
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

# 2. Reddit r/netsec latest — sort goes into the URL (/new/); `searchSort` does not apply to startUrls
apify actors call "harshmaur/reddit-scraper" -i '{
  "startUrls": [{"url": "https://www.reddit.com/r/netsec/new/"}],
  "maxPostsCount": 15,
  "crawlCommentsPerPost": false
}' --user-agent apify-awesome-skills/apify-osint-threat-intel --json 2>/dev/null

Step 3 — Triage and assess

For every finding, apply this classification:

Severity	Criteria
Critical	CVSS ≥ 9.0 OR on CISA KEV list OR public PoC + unpatched
High	CVSS 7.0–8.9 OR active exploitation reported in news
Medium	CVSS 4.0–6.9 OR breach mention without active exploit
Low	CVSS < 4.0 OR historical, patched, no active exploitation
Informational	Exposure hints without confirmed vulnerability

Step 4 — Deliver structured report

Output format:

## Threat Intelligence Report — [TARGET]
Date: [today]

### Executive Summary
[2–3 sentence risk verdict]

### Critical Findings
- [CVE/Finding] — Severity: [X] — Status: [Patched/Unpatched/Active exploit]
  Source: [URL]

### Breach/Exposure Indicators
- [Finding] — Source: [URL]

### Threat Actor Activity (if applicable)
- [Actor] — TTPs: [list] — Last seen: [date]

### Recommended Actions
1. [Immediate action]
2. [Short-term action]
3. [Monitoring recommendation]

### Data Sources
[Bullet list of all URLs cited]

Data Quality Rules

Every claim needs a source URL — no ungrounded assertions
Empty results are intelligence — report them explicitly ("no paste mentions found")
Date-stamp all findings — CVE severity, patch status, and breach reports are time-sensitive
Confidence tiers:
- [Confirmed] — primary source (NVD, CISA, vendor advisory)
- [Reported] — news + community corroboration
- [Unverified] — single secondary source, flag clearly
Parallelize independent actor calls (CVE search + news + Reddit can run simultaneously; the two Reddit runs — netsec, cybersecurity — too)
Budget: warn user if >10 actor calls needed; get approval before proceeding

Troubleshooting

Problem	Fix
`google-search-scraper` returns 0 results	Simplify query, remove `site:` filter, try broader terms
`website-content-crawler` times out on NVD	Use `rag-web-browser` as fallback with direct CVE URL
`harshmaur/reddit-scraper` returns 0 items, or posts from unrelated subreddits	Read the `RUN-SUMMARY` record in the run's key-value store: `inputWarnings` says when `withinCommunity` was dropped (more than one name) or a date was unparseable, `emptyReason` explains 0 items. Shorten the term (Reddit search is literal). Fallback: `fatihtahta/reddit-scraper-search-fast` with `{"subredditName": "netsec", "subredditKeywords": ["[TERM]"], "subredditTimeframe": "month", "maxPosts": 10}` ($0.00149 per post, no start fee; fields `title`, `url`, `subreddit`, `created_utc`, `score`, `num_comments`)
`tweet-scraper` returns sparse results	Broaden to `#cybersecurity [term]` or drop hashtag requirement
CISA KEV page too large to crawl	Use `rag-web-browser` with specific CVE ID as query

Example prompts

"Check if example.com has any known vulnerabilities or appears in recent breach data."
"What's the latest threat intel on CVE-2026-1234 — is it actively exploited?"
"Profile the APT28 group — recent campaigns, TTPs, and infrastructure."

Boundary: This skill researches organizations, infrastructure and named threat groups. It won't build cross-platform profiles of private individuals.