multi-search-engine

multi-search-engine

A unified CLI for querying several web search engines in parallel and returning a normalized result list. Built on httpx and beautifulsoup4 (both already in OpenSquilla default dependencies, so no extra install beyond pip install opensquilla).

Use cases

Building a deep-research round with diverse engine coverage
Fact-check a claim against >1 engine
Compare scholarly metadata coverage with general web results
Find citable publication metadata without requiring an API key

Limitations

A single engine sufficient → call its API directly instead
Need headless-browser DOM rendering → this skill is HTTP-only

Quick start

python {baseDir}/scripts/search.py \
    --query "openclaw skill registry" \
    --engines crossref,duckduckgo,brave \
    --limit 10 \
    --json

Output:

{
  "query": "...",
  "results": [
    {
      "engine": "crossref",
      "title": "...",
      "url": "https://...",
      "snippet": "...",
      "rank": 1,
      "doi": "10.1234/example",
      "year": 2024,
      "authors": ["Ada Example", "Edge Research Consortium"],
      "corporate_authors": ["Edge Research Consortium"]
    }
  ],
  "errors": [
    {"engine": "brave", "reason": "BRAVE_SEARCH_API_KEY/BRAVE_API_KEY not set; skipping"}
  ]
}

doi, year, authors, and corporate_authors are optional. They are emitted only when an engine supplies verifiable values, so the existing five-field web-result shape remains compatible for general engines. The parallel corporate-author list lets BibTeX consumers protect institution names from person-name parsing.

Engines

Engine	Needs key	Key env var	Strength
`crossref`	no	optional `CROSSREF_MAILTO`	Scholarly works with canonical DOI, year, and author metadata
`duckduckgo`	no	—	No-key, privacy-oriented broad web baseline
`brave`	yes	`BRAVE_SEARCH_API_KEY` or legacy `BRAVE_API_KEY`	High-quality results, generous free tier
`tavily`	yes	`TAVILY_API_KEY`	Designed for AI agents, returns clean JSON

The script never errors out when an API-key engine's key is missing — it records a per-engine errors entry and continues with the rest. Pass --strict to fail fast when any requested engine is unavailable.

Routing tips

The host should pick engines by language and availability:

Academic queries → crossref first, then brave or tavily for broader context
General web queries → duckduckgo plus brave or tavily for triangulation
Time-sensitive (last 24h) → brave (recency filter) or tavily
Long-tail academic → start with crossref; supplement with direct arXiv when needed

engines.md has the full per-engine guidance.

Boundaries

HTTP-only. JS-rendered pages will not be readable; use a headless-browser skill if needed.
DuckDuckGo scraping is best-effort — HTML structure changes can break it. The script logs parse failures individually and keeps the run going.
Timeout, HTTP 429, and transient HTTP 5xx responses receive at most two retries with bounded backoff. Repeated failures remain per-engine soft errors. Callers must still avoid high-rate loops.
Captcha-protected results are not bypassed. If an engine returns a challenge page, the parser will return zero results for that engine and log a warning.

Crossref is a public metadata service, not a full-text search index. The client sends query.bibliographic, respects the requested result limit, and uses canonical https://doi.org/<doi> URLs. Set CROSSREF_MAILTO to identify your application through Crossref's polite-pool convention. See the Crossref REST API etiquette and rate-limit documentation.