SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillssearch-layer

search-layer

General & other

View repository

Updated 10-15 20:06

About this skill

Search Layer v2.2 — Intent-Aware Multi-Source Retrieval Protocol

Four sources at the same level: Brave (web_search) + Exa + Tavily + Grok. Automatically selects strategy, adjusts weights, and performs synthesis based on intent.

Execution Flow

用户查询
    ↓
[Phase 1] 意图分类 → 确定搜索策略
    ↓
[Phase 2] 查询分解 & 扩展 → 生成子查询
    ↓
[Phase 3] 多源并行检索 → Brave + search.py (Exa + Tavily + Grok)
    ↓
[Phase 4] 结果合并 & 排序 → 去重 + 意图加权评分
    ↓
[Phase 5] 知识合成 → 结构化输出

Phase 1: Intent Classification

After receiving a search request,first determine the intent type, then decide the search strategy. Do not ask the user which mode to use.

Intent Recognition Signal Mode Freshness Weight Bias
Factual "What is X", "definition of X", "What is X" answer — Authority 0.5
Status "latest developments of X", "current state of X", "latest X" deep pw/pm Freshness 0.5
Comparison "X vs Y", "difference between X and Y" deep py Keywords 0.4 + Authority 0.4
Tutorial "how to do X", "X tutorial", "how to X" answer py Authority 0.5
Exploratory "in-depth understanding of X", "X ecosystem", "about X" deep — Authority 0.5
News "X news", "X this week", "X this week" deep pd/pw Freshness 0.6
Resource "X official website", "X GitHub", "X documentation" fast — Keywords 0.5

For the detailed classification guide, see references/intent-guide.md

Decision Rules：

  1. Scan the query for signal words
  2. When multiple types match, choose the most specific one
  3. When unable to determine, default to exploratory

Phase 2: Query Decomposition & Expansion

Based on the intent type, expand the user query into a set of sub-queries:

General Rules

  • Technical synonyms are automatically expanded：k8s→Kubernetes, JS→JavaScript, Go→Golang, Postgres→PostgreSQL
  • Chinese technical queries: also generate English variants (e.g., "Rust async programming" → additionally search "Rust async programming")

Expand by Intent

Intent Expansion Strategy Example
Factual Add "definition", "explained" "WebTransport" → "WebTransport", "WebTransport explained overview"
Status Add year, "latest", "update" "Deno developments" → "Deno 2.0 latest 2026", "Deno update release"
Comparison Split into 3 sub-queries "Bun vs Deno" → "Bun vs Deno", "Bun advantages", "Deno advantages"
Tutorial Add "tutorial", "guide", "step by step" "Rust CLI" → "Rust CLI tutorial", "Rust CLI guide step by step"
Exploratory Split into 2-3 angles "RISC-V" → "RISC-V overview", "RISC-V ecosystem", "RISC-V use cases"
News Add "news", "announcement", date "AI news" → "AI news this week 2026", "AI announcement latest"
Resource Add specific resource type "Anthropic MCP" → "Anthropic MCP official documentation"

Phase 3: Multi-Source Parallel Retrieval

Step 1: Brave (all modes)

Call web_searchfor each sub-query. If the intent has freshness requirements, pass the freshness parameter:

web_search(query="Deno 2.0 latest 2026", freshness="pw")

Step 2: Exa + Tavily + Grok (Deep / Answer mode)

Call search.py for the sub-query, passing in the intent and freshness:

python3 /home/node/.openclaw/workspace/skills/search-layer/scripts/search.py \
  --queries "子查询1" "子查询2" "子查询3" \
  --mode deep \
  --intent status \
  --freshness pw \
  --num 5

Source Participation Matrix by Mode：

Mode Exa Tavily Grok Description
fast ✅ ❌ fallback Exa preferred; use Grok when no Exa key
deep ✅ ✅ ✅ Three sources in parallel
answer ❌ ✅ ❌ Tavily only (includes AI answer)

Parameter Description：

Parameter Description
--queries Multiple sub-queries executed in parallel (a single query can also be passed as a positional argument)
--mode fast / deep / answer
--intent Intent type, affects scoring weights (if not passed, no scoring is done and behavior is the same as v1)
--freshness pd(24h) / pw(week) / pm(month) / py(year)
--domain-boost Comma-separated domains; matching results get authority score +0.2
--num Number of results per source per query

Exa source description (two-layer roles)：

  • Retrieval lane (default main path)：
    • By default it still goes through /search, but no longer hardcodes type=auto
    • Current minimal mapping:
      • resource → instant
      • status / news → fast
      • exploratory + mode=deep → deep
      • Other → auto
    • By default includes contents.highlights.maxCharacters=1200, improving snippet quality and preventing Exa results from being underestimated in local ranking due to empty abstracts
    • freshness maps to Exa startPublishedDate, making status/news queries more consistent with the Tavily/Grok time windows
    • Keeps meta.exaTypein the result metadata, making it easy to observe the actually resolved type
  • Research lane (selective escalation)：
    • Only when the query hits a complex comparison / exploratory / status / news scenario, after standard candidate recall, append a segment of Exa type=deep research block, and attach it to the output with the research field
    • research is an additional contract; it does not replace results, ensuring old callers can still read only results
    • Current boundaries: comparison requires explicit comparison words/judgment words/3+ subqueries; exploratory requires judgment/causal/comparison words; status/news requires judgment/causal words, and it is not mistakenly triggered by ordinary multi-query expansion
  • For now, do not connect deep-reasoning / outputSchema to the default main path, to avoid turning the basic search-layer into a heavyweight research/synthesis engine
  • The Exa endpoint by default is https://api.exa.ai/search; if you need a self-hosted/proxy setup, you can override it via the EXA_API_BASE/EXA_API_URL in ~/.openclaw/credentials/search.json or exaApiBase (example:https://exa.example.com）

Grok source description：

  • Call the Grok model via the completions API (grok-4.1-fast), using its real-time knowledge to return structured search results
  • Automatically detect time-sensitive queries and inject the current time context
  • Execute in parallel with Exa and Tavily in deep mode
  • Requires configuring Grok's ~/.openclaw/credentials/search.json in apiUrl、apiKey、model(or via the environment variable GROK_API_URL、GROK_API_KEY、GROK_MODEL）
  • If the Grok configuration is missing, automatically fall back to the dual-source Exa + Tavily setup

Step 3: Merge

Merge Brave results with the search.py output. Deduplicate by canonical URL and mark the source.

If search.py returns the score field, use it for sorting; for Brave results without a score, recalculate it using the same intent weight formula.

Phase 3.5: Citation tracing (Thread Pulling)

When the search results contain GitHub issue/PR links and the intent is Status or Exploratory, automatically trigger citation tracing.

Automatic trigger conditions

  • Intent is status or exploratory
  • Search results contain github.com/.../issues/ or github.com/.../pull/ URL

Method 1: search.py --extract-refs (batch)

Extract the citation graph directly from the search results, with no extra call needed:

python3 search.py "OpenClaw config validation bug" --mode deep --intent status --extract-refs

The output will have an extra refs field, containing the citation list for each result URL.

You can also skip the search and extract citations directly for known URLs:

python3 search.py --extract-refs-urls "https://github.com/owner/repo/issues/123" "https://github.com/owner/repo/issues/456"

Method 2: fetch-thread (single-URL deep fetch)

Fetch the complete discussion thread + structured citations for a single URL:

python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --format json
python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --format markdown
python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --extract-refs-only

GitHub scenario (issue/PR): Fetch the body + all comments + timeline events (cross-references, commits) via the API, and extract:

  • Issue/PR references (#123, owner/repo#123)
  • Duplicate markers
  • Commit references
  • Linked PR/issues (timeline cross-references)
  • External URLs

General web scenario: web fetch + regex extraction of reference links.

Agent execution flow

Step 1: search-layer 搜索 → 获取初始结果
Step 2: search.py --extract-refs 或 fetch-thread → 提取线索图
Step 3: Agent 筛选高价值线索（LLM 判断哪些值得追踪）
Step 4: fetch-thread 深度抓取每个高价值线索
Step 5: 重复 Step 2-4，直到信息闭环或达到深度限制（建议 max_depth=3）

Phase 4: Result ranking

Scoring formula

score = w_keyword × keyword_match + w_freshness × freshness_score + w_authority × authority_score

Weights are determined by intent (see the Phase 1 table). Each sub-score:

  • keyword_match (0-1): Coverage of query terms in title + summary
  • freshness_score (0-1): Based on publication date, newer is higher (no date = 0.5)
  • authority_score (0-1): Based on domain authority tier
    • Tier 1 (1.0): github.com, stackoverflow.com, official documentation sites
    • Tier 2 (0.8): HN, dev.to, well-known tech blogs
    • Tier 3 (0.6): Medium, Juejin, InfoQ
    • Tier 4 (0.4): Others

For the full domain scoring table, see references/authority-domains.json

Domain Boost

Manually specify domains to be weighted via the --domain-boost parameter (authority score +0.2 for matching results):

search.py "query" --mode deep --intent tutorial --domain-boost dev.to,freecodecamp.org

Recommended pairing:

  • Tutorial → dev.to, freecodecamp.org, realpython.com, baeldung.com
  • Resource → github.com
  • News → techcrunch.com, arstechnica.com, theverge.com

Phase 5: Knowledge synthesis

Choose a synthesis strategy based on the number of results:

Small result set (≤5 items)

Show item by item, each with a source label and score:

1. [Title](url) — snippet... `[brave, exa]` ⭐0.85
2. [Title](url) — snippet... `[tavily]` ⭐0.72

Medium result set (5-15 items)

Cluster by topic + summary per group:

**主题 A: [描述]**
- [结果1] — 要点... `[source]`
- [结果2] — 要点... `[source]`

**主题 B: [描述]**
- [结果3] — 要点... `[source]`

Large result set (15+ items)

High-level overview + Top 5 + deeper-dive hints:

[一段综述，概括主要发现]

**Top 5 最相关结果：**
1. ...
2. ...

共找到 N 条结果，覆盖 [源列表]。需要深入哪个方面？

Synthesis rules

  • Give the answer first, then list sources(don't start with "what I searched for")
  • Aggregate by topic, not by source(don't do "Brave results: ... Exa results: ...")
  • Explicitly flag conflicting information: clearly point out when different sources contradict each other
  • Confidence expression：
    • Multiple sources agree + fresh → state directly
    • Single source or older → "According to [source], ..."
    • Conflicting or uncertain → "There are differing accounts: A believes..., B believes..."

Fallback strategy

  • Exa 429/5xx → continue with Brave + Tavily + Grok
  • Tavily 429/5xx → continue with Brave + Exa + Grok
  • Grok timeout/error → continue with Brave + Exa + Tavily
  • search.py fails entirely → use only Brave web_search(always available)
  • Never block the main flow because a source failed

Backward compatibility

Without the --intent parameter, search.py behaves exactly the same as v1 (no scoring, output in original order).

Existing callers (such as github-explorer) need no changes.

Quick reference

Scenario Command
Quick facts web_search + search.py --mode answer --intent factual
Deep research web_search + search.py --mode deep --intent exploratory
Latest updates web_search(freshness="pw") + search.py --mode deep --intent status --freshness pw
Comparative analysis web_search × 3 queries + search.py --queries "A vs B" "A pros" "B pros" --intent comparison
Find resources web_search + search.py --mode fast --intent resource