SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsrecon-sector-expansion

recon-sector-expansion

Security & auditing

View repository

Updated 10-16 03:19

About this skill

RECON-SECTOR-EXPANSION — Batch Discovery Across Untapped Sectors

When to Use

Use within authorized engagements to expand coverage into fresh sectors — not dig deeper into a single known target, but find and test 20+ new company domains across 5-10 different sectors. Target list must be from authorized scope. Batch expansion is for scoped programs, not indiscriminate scraping.

Signals:

  • "Discover 20 more domains in non-regulated sectors"
  • "We need to expand coverage; what haven't we tested?"
  • "Find fresh targets outside the usual sectors"

The Sector Expansion Workflow

Phase 1 — Coverage Audit

Before discovering new targets, know what's already been tested:

# 1. Check existing target list
ALL_TARGETS="${OUTPUT_DIR:-./output}/sector-expansion/all_targets.txt"
ALL_MASSIVE="${OUTPUT_DIR:-./output}/sector-expansion/all_massive.txt"

# Extract all domains from both files
grep -ohE '[a-zA-Z0-9][a-zA-Z0-9.-]*\.[a-z]{2,}' "$ALL_TARGETS" "$ALL_MASSIVE" | sort -u > /tmp/targets_deduped.txt
echo "Already tested: $(wc -l < /tmp/targets_deduped.txt) domains"

# 2. Check what findings files exist
ls "${OUTPUT_DIR:-./output}"/sector-expansion/*_findings.md 2>/dev/null | wc -l

# 3. Identify which sectors have been covered by inspecting findings files
# Check for sector-specific keywords
for sector in "dentist" "dental" "gym" "fitness" "baker" "auto_body" "carpet" "laundry" "daycare" "pest" "tree" "pet"; do
  count=$(grep -l "$sector" "${OUTPUT_DIR:-./output}"/sector-expansion/*_findings*.md 2>/dev/null | wc -l)
  echo "Sector '$sector': $count findings files"
done

Phase 2a — Source 1: crt.sh Certificate Transparency (Preferred for Local Business Sectors)

For local service sectors (roofing, landscaping, plumbing, pool services, HVAC) where no national chains exist, crt.sh is the primary discovery source. These businesses often have descriptive domain names like [name][SECTOR_EXAMPLE] or [city][SECTOR_EXAMPLE].

crt.sh Query Pattern

Use simple sector keywords with the HTML output mode (JSON API returns 502/503 on broad queries):

QUERY="roofing"
curl -s --max-time 40 --connect-timeout 10 \
  "https://crt.sh/?q=${QUERY}&excluded=expired&dedup=Y" \
  -H 'User-Agent: Mozilla/5.0' 2>/dev/null | \
  grep -oE '>[A-Za-z0-9][A-Za-z0-9.-]*\.com<' | \
  sed 's/^>//;s/<$//' | \
  sort -u >> /tmp/all_crt_domains.txt

Parameters explained:

  • &excluded=expired — only current certificates
  • &dedup=Y — deduplicate certificates
  • -oE regex — BusyBox-compatible (no -P flag available)
  • No &output=json — JSON API is unreliable for broad queries

Sector Keyword Catalog

Query ALL these sectors to gather candidates:

SECTORS="roofing landscaping pestcontrol dentist dentist fitness \
cleaningservice movingcompany photography vetclinic realtor hvac \
treeservice lawncare plumbing poolcleaning windowcleaning petsalon \
barbershop daycare carpetcleaning handyman lawfirm concrete \
autorepair petgrooming autobody remodeling"
for sector in $SECTORS; do
  curl -s --max-time 40 --connect-timeout 10 "https://crt.sh/?q=${sector}&excluded=expired&dedup=Y" \
    -H 'User-Agent: Mozilla/5.0' 2>/dev/null | \
    grep -oE '>[A-Za-z0-9][A-Za-z0-9.-]*\.com<' | \
    sed 's/^>//;s/<$//' >> /tmp/all_crt_domains.txt
  sleep 2  # avoid rate limiting
done

Filtering Noise from crt.sh Results

crt.sh returns many subdomains and infrastructure hosts. Strip these patterns:

import re

# Common noise patterns from crt.sh
skip_patterns = [
    '^autodiscover\\.', '^vpn\\.', '^api\\.', '^mail\\.', '^remote\\.',
    '^webmail\\.', '^crm\\.', '^ftp\\.', '^test\\.', '^dev\\.',
    '^exchange', '^hostmaster', '^owa\\.', '^smtp\\.',
    # Generic placeholder domains
    'roofingcompany\\.com$', 'landscapingcompany\\.com$', 'vetclinic\\.com$',
]

Then associate domains with sectors by matching their name:

SECTOR_KEYWORDS = {
    'roofing': ['roofing', 'roof', 'roofer'],
    'landscaping': ['landscaping', 'landscape', 'lawn'],
    'hvac': ['hvac', 'heating', 'cooling', 'air'],
    'dental': ['dentist', 'dental'],
    'pest-control': ['pest', 'mosquito'],
    'daycare': ['daycare'],
    'legal': ['lawfirm', 'lawfirm'],
    'fitness': ['gym', 'fitness'],
    'auto-repair': ['autorepair', 'auto', 'collision'],
    'cleaning': ['cleaning', 'maid'],
    'veterinary': ['vet', 'pet'],
    'pool-services': ['pool', 'spa'],
}

Noise Subdomain Filtering

When using grep on crt.sh HTML output, you'll get entries from large hosting providers. Exclude:

  • autodiscover.*, vpn.*, api.*, mail.*, remote.* — infrastructure
  • .*\.hvacrightnow\.com, .*\.hvacbrain\.com — SaaS platforms (not actual targets)
  • .*\.theroofingprofessor\.com, .*\.bakerroofing\.com — large org subdomains
  • .*\.singaporepools\.com, .*\.devsingaporepools\.com — non-US

Piping through grep -v with exclusion patterns before dedup saves time.

Phase 2b — Source 2: Google / Manual (Preferred for National Chains)

For sectors dominated by national/regional chains (dental, gyms, bakeries, auto body), use Google searches because chain domain names rarely contain sector keywords:

Sector How to Find Candidates Example Chains
Dental chains Google "top dental chains USA" Aspen Dental, Gentle Dental, Coast Dental, DentalWorks, Heartland Dental, Pacific Dental
Gym/fitness Google "largest fitness chains USA" F45, Barry's, CrossFit, Gold's Gym, 24 Hour Fitness, LA Fitness, OrangeTheory
Bakery chains Google "largest bakery chains USA" Cinnabon, Crumbl, Insomnia Cookies, Sprinkles, Nothing Bundt Cakes
Auto body/collision Google "auto body repair chains USA" ABRA, Caliber, Gerber, Service King, Fix Auto, Sterling, Crash Champions
Carpet cleaning Google "carpet cleaning companies USA" Chem-Dry, ServPro, Stanley Steemer, Rainbow International, Heaven's Best
Laundry services Google "laundry delivery service USA" Poplin, Rinse, Washlava, LaundryHeap
Daycare/childcare Google "largest daycare chains USA" Bright Horizons, KinderCare, Goddard, Primrose, Children of America
Pet grooming Google "pet grooming chains USA" Petco, PetSmart, Camp Bow Wow, Dogtopia
Pest control Google "pest control companies USA" Terminix, Orkin, Ehrlich, Bulwark, Arrow
Tree services Google "tree service companies USA" Davey, SavATree, Arbor Care, Bartlett

Phase 3 — Batch Alive-Check (Curl Probe)

Once you have 50-60 candidate domains, filter against already-tested domains, then quick-check HTTP:

# Filter against known targets
grep -v -f /tmp/targets_deduped.txt /tmp/candidates.txt | sort -u > /tmp/fresh_candidates.txt

# Quick alive check with curl (serial, ~2.5 min for 60 targets)
echo "=== Checking all candidates with curl ==="
for domain in $(cat /tmp/fresh_candidates.txt); do
  code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 5 --max-time 8 "https://${domain}" 2>/dev/null)
  echo "$domain => $code"
done > /tmp/curl_alive.txt

# Extract alive domains (anything not 000/timeout)
grep -v "=> 000" /tmp/curl_alive.txt | grep -v "^$" | cut -d' ' -f1 > /tmp/alive_domains.txt
echo "Alive: $(wc -l < /tmp/alive_domains.txt) / $(wc -l < /tmp/fresh_candidates.txt) candidates"

Phase 4 — Full Testing Pipeline

Then run the full 6-step pipeline on alive domains. The most efficient approach at scale (20+ targets) is a Python batch script:

#!/usr/bin/env python3
"""Batch test targets: httpx → WP REST → CORS → XMLRPC → ports"""
import subprocess, json, sys, os, re, time

OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "./output") + "/sector-expansion"

def run(cmd, timeout=15):
    try:
        r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
        return r.stdout + r.stderr
    except subprocess.TimeoutExpired:
        return f"TIMEOUT ({timeout}s)"
    except Exception as ex:
        return f"ERROR: {ex}"

def test_domain(domain, sector):
    findings = {"domain": domain, "sector": sector, "wp": False, "wp_users": 0,
                "cors_reflected": False, "xmlrpc_open": False}
    
    # STEP 1: httpx probe
    r = run(f"httpx -sc -title -tech-detect -server -no-color -u https://{domain} -timeout 10")
    findings["http"] = r.strip()
    
    # STEP 2: WordPress REST API check (3 paths)
    for path in ["/wp-json/wp/v2/users", "/wp-json/", "/?rest_route=/wp/v2/users"]:
        r = run(f'curl -sk -o /dev/null -w "%{{http_code}}" --max-time 10 --connect-timeout 10 "https://{domain}{path}"')
        if r in ["200"]:
            findings["wp_rest_status"] = f"{path} => HTTP {r}"
            findings["wp"] = True
            # Enumerate users
            user_resp = run(f'curl -sk --max-time 10 --connect-timeout 10 "https://{domain}{path}"', timeout=12)
            try:
                users = json.loads(user_resp)
                if isinstance(users, list):
                    findings["wp_users"] = len(users)
                    findings["wp_users_list"] = [u.get("name","") for u in users]
            except:
                pass
            break
    
    # Fallback: check page source for WP markers
    if not findings["wp"]:
        r = run(f'curl -sk --max-time 10 --connect-timeout 10 "https://{domain}"', timeout=12)
        if "/wp-content/" in r or "/wp-includes/" in r:
            findings["wp"] = True
    
    # STEP 3: CORS test
    target_url = f"https://{domain}/wp-json/wp/v2/users" if findings["wp"] else f"https://{domain}/"
    r = run(f'curl -sk -H "Origin: https://evil.com" -D- --max-time 10 --connect-timeout 10 "{target_url}" | grep -i access-control', timeout=12)
    if "Access-Control-Allow-Origin" in r:
        findings["cors_reflected"] = True
        findings["cors_credentialed"] = "Access-Control-Allow-Credentials" in r
    
    # STEP 4: XMLRPC check
    r = run(f'curl -sk -o /dev/null -w "%{{http_code}}" --max-time 10 --connect-timeout 10 -X POST "https://{domain}/xmlrpc.php"', timeout=12)
    findings["xmlrpc_open"] = (r.strip() == "200")
    
    # STEP 5: Port scan (top 20)
    r = run(f'nmap --top-ports 20 -T4 --open -n {domain}', timeout=120)
    ports = [line.split('/')[0] for line in r.split('\n') if re.search(r'^\d+/tcp\s+open', line)]
    findings["ports"] = ports
    
    # Severity calculation
    score = sum([findings["wp"], bool(findings["wp_users"] > 0), findings["cors_reflected"], findings["xmlrpc_open"]])
    findings["vuln"] = "HIGH" if score >= 3 else ("MEDIUM" if score >= 2 else ("LOW" if score >= 1 else "NONE"))
    
    return findings

# Usage
domains = {"[TARGET_1]": "dental", "[TARGET_2]": "gym_fitness", ...}
for domain, sector in domains.items():
    f = test_domain(domain, sector)
    # Write findings.md
    content = f"""## {domain}
| Field | Value |
|-------|-------|
| HTTP | {f['http'][:200]} |
| WordPress | {'Yes' if f['wp'] else 'No'}, users: {f['wp_users']} |
| CORS | {'Reflected' + (' WITH CREDENTIALS' if f.get('cors_credentialed') else '') if f['cors_reflected'] else 'Not reflected'} |
| XMLRPC | {'Open' if f['xmlrpc_open'] else 'Closed'} |
| Ports | {', '.join(f['ports']) if f['ports'] else 'none found'} |
| Sector | {sector} |
| Vuln | {f['vuln']} |
"""
    with open(f"{OUTPUT_DIR}/{domain}_findings.md", 'w') as out:
        out.write(content)

Phase 5 — Results Analysis

Review findings for these high-signal patterns:

Pattern How to Spot Severity
Credentialled CORS Access-Control-Allow-Origin: http://evil.com + Access-Control-Allow-Credentials: true on /wp-json/wp/v2/users HIGH
WordPress users exposed REST API at /wp-json/wp/v2/users returns array of user objects with names, slugs, avatars MEDIUM
Wildcard CORS Access-Control-Allow-Origin: * (no credentials) MEDIUM
XMLRPC open /xmlrpc.php returns HTTP 200 (may be blocked at method-list level by WAF) LOW-MEDIUM
Open ports on non-standard services 21 (FTP), 3306 (MySQL), 3389 (RDP), 5900 (VNC), 6379 (Redis) VARIES
PHPInfo exposed /info.php, /test.php, etc reveal 400+ config entries LOW

Sector-Specific Chains Domain List

Curated list of US national/regional chain domains (verified alive as of 2026-06):

Dental

aspendental.com  gentledental.com  dentalworks.com  coastdental.com
brightnow.com    sagedental.com    western-dental.com  castledental.com
heartlanddental.com  pacificdental.com  dentalcarealliance.com

Gym / Fitness

[FITNESS_CHAIN]  barrys.com  crossfit.com  goldsgym.com
24hourfitness.com  lafitness.com  orangetheory.com  planetfitness.com
anytimefitness.com  snapfitness.com  crunch.com

Bakery

cinnabon.com  crumbl.com  insomniacookies.com  sprinkles.com
bakery.example.com  restaurant.example.com  donut.example.com

Auto Body / Collision Repair

abracollision.com  serviceking.com  sterlingautobody.com  crashchampions.com
fixauto.com  true2form.com  auto-body-shop.com
calibercollision.com  carstar.com  gerbercollision.com  maaco.com

Carpet Cleaning

chemdry.com  servpro.com  rainbowintl.com  heavenlycarpet.com
stanleysteemer.com  zerorez.com  coit.com

Laundry

poplin.com  rinse.com  washlava.com  laundryheap.com
speedqueen.com  spincyclelaundry.com

Daycare

brighthorizons.com  childrenofamerica.com  kindercare.com  goddardschool.com
primroseschools.com  montessori.com  lajollamontessori.com  tutortime.com

Pest Control

terminix.com  orkin.com  echols.com  bulwarkpest.com
arrowpest-control.example.com  dodsonbros.com  massey-services.com

Tree Services

treecare.com  arborcare.com  davey.com  savatree.com
bartlett.com  allstatetree.com  firstchoicetree.com

Roofing (Local/Regional — crt.sh Discovered)

aastro[SECTOR_EXAMPLE]  abcarnes[SECTOR_EXAMPLE]  americaschoiceroofers.com  arvada[SECTOR_EXAMPLE]
bacon[SECTOR_EXAMPLE]  baker[SECTOR_EXAMPLE]  bricor[SECTOR_EXAMPLE]  capitol[SECTOR_EXAMPLE]
charlottepro[SECTOR_EXAMPLE]  coloradopro[SECTOR_EXAMPLE]  eh[SECTOR_EXAMPLE]  errdaddy[SECTOR_EXAMPLE]
fifthwall[SECTOR_EXAMPLE]  freeman[SECTOR_EXAMPLE]  ghatley.com  hambro[SECTOR_EXAMPLE]
hollis[SECTOR_EXAMPLE]  integrityrc.com  kbfamily[SECTOR_EXAMPLE]  mcas[SECTOR_EXAMPLE]
na[SECTOR_EXAMPLE]  pabco[SECTOR_EXAMPLE]  roofon.com  roofwithfoster.com
sandiegocounty[SECTOR_EXAMPLE]  springfield[SECTOR_EXAMPLE]  tomtheroofer.com
universityroof.com  vanguardroofingltd.com  wallace[SECTOR_EXAMPLE]

Landscaping (Local/Regional — crt.sh Discovered)

delliquadrilandscape.com  dundeedig.com  guardyouryardpa.com
landscapingsi.com  miles[SECTOR_EXAMPLE]  moss[SECTOR_EXAMPLE]
mountainlandscapingkc.com  sarasota[SECTOR_EXAMPLE]  trinas[SECTOR_EXAMPLE]
moss[SECTOR_EXAMPLE]  eo[SECTOR_EXAMPLE]  brucewilson[SECTOR_EXAMPLE]

HVAC (Local/Regional — crt.sh Discovered)

airzonahvac.com  bigfishhvac.com  dormarhvac.com  specializedhvac.com

General Contractors & Other (crt.sh Discovered)

allysonsflowers.com  [INSURANCE_CO]  canopyroofers.com
gelinc.com  rvroofrepairflorida.com  spartanroofingbc.com

Pitfalls

  1. Don't rely on a single key file for "tested" domains. all_targets.txt and all_massive.txt may differ. Merge both and deduplicate with sort -u.

  2. CloudFlare blocks WP REST API for some sites. If you get HTTP 403 on /wp-json/wp/v2/users, try /?rest_route=/wp/v2/users or check page source for /wp-content/ markers.

  3. XMLRPC may return 200 but block system.listMethods. CloudFlare and other WAFs block the method-listing XML body but leave the endpoint open for system.multicall. Test separately.

  4. Some 301/302 sites may still have CORS. Redirect responses can carry CORS headers (brighthorizons.com had wildcard CORS on its 301 response). Test the redirect target URL if the landing page differs.

  5. BusyBox grep limitation. Minimal environments may provide BusyBox, which lacks grep -P. Use grep -oE with extended patterns. BusyBox find also lacks -printf; use -exec echo {} \; or shell loops instead.

  6. crt.sh rate limiting. Returns HTTP 502/503 when hammered. Spread queries 3 seconds apart and if you get repeated errors, wait 15-30s. The JSON API (&output=json) is especially unreliable for broad queries — use HTML output mode (&excluded=expired&dedup=Y) instead.

  7. Port scan timing. nmap --top-ports 20 -T4 takes ~3-5s per target. Parallelize across 5-7 targets in background.

  8. CORS on non-WordPress sites. Generic static sites on CloudFront or S3 may also reflect CORS. Always test on both / and any known API endpoint.

  9. Python JSON parsing edge case. The WP REST API user endpoint may return an object instead of a list for protected endpoints (HTTP 401 vs 200 response codes). Always check isinstance(users, list).

Verification

  1. Sector domain generation — confirm sector keywords produce candidate domains: 
    echo 'roofing landscaping pestcontrol' | tr ' ' '

' | wc -l

2. **crt.sh accessibility** — confirm crt.sh CDX endpoint reachable:
```bash
curl -s --max-time 10 --connect-timeout 10 -o /dev/null -w '%{http_code}' 'https://crt.sh/?q=test&excluded=expired&dedup=Y'

All tests verify sector expansion readiness.

Related Skills

  • web2-recon — The per-target technical testing pipeline (httpx, WP enum, CORS, XMLRPC, port scan, JS analysis)
  • recon-sector — Unified parameterized sector recon with 25 sectors in sectors.yaml