SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillssearch-layer

search-layer

General & other

View repository

Updated 10-15 11:01

About this skill

Search Layer v2.2 — Intent-Aware Multi-Source Retrieval Protocol

Four sources at the same level: Brave (web_search) + Exa + Tavily + Grok. Automatically selects strategy, adjusts weights, and synthesizes based on intent.

Execution Flow

用户查询
    ↓
[Phase 1] 意图分类 → 确定搜索策略
    ↓
[Phase 2] 查询分解 & 扩展 → 生成子查询
    ↓
[Phase 3] 多源并行检索 → Brave + search.py (Exa + Tavily + Grok)
    ↓
[Phase 4] 结果合并 & 排序 → 去重 + 意图加权评分
    ↓
[Phase 5] 知识合成 → 结构化输出

Phase 1: Intent Classification

After receiving a search request,first determine the intent type, then decide the search strategy. Do not ask the user which mode to use.

Intent Recognition Signals Mode Freshness Weight Bias
Factual "What is X", "the definition of X", "What is X" answer — Authority 0.5
Status "latest developments of X", "current state of X", "latest X" deep pw/pm Freshness 0.5
Comparison "X vs Y", "difference between X and Y" deep py Keywords 0.4 + Authority 0.4
Tutorial "how to do X", "X tutorial", "how to X" answer py Authority 0.5
Exploratory "in-depth understanding of X", "X ecosystem", "about X" deep — Authority 0.5
News "X news", "X this week", "X this week" deep pd/pw Freshness 0.6
Resource "X official website", "X GitHub", "X documentation" fast — Keywords 0.5

For the detailed classification guide, see references/intent-guide.md

Decision Rules：

  1. Scan the query for signal words
  2. When multiple types match, choose the most specific one
  3. When it cannot be determined, default to exploratory

Phase 2: Query Decomposition & Expansion

Based on the intent type, expand the user query into a set of sub-queries:

General Rules

  • Automatic expansion of technical synonyms：k8s→Kubernetes, JS→JavaScript, Go→Golang, Postgres→PostgreSQL
  • Chinese technical queries: also generate English variants (e.g., "Rust 异步编程" → additionally search "Rust async programming")

Expansion by Intent

Intent Expansion Strategy Example
Factual Add "definition", "explained" "WebTransport" → "WebTransport", "WebTransport explained overview"
Status Add year, "latest", "update" "Deno 进展" → "Deno 2.0 latest 2026", "Deno update release"
Comparison Split into 3 sub-queries "Bun vs Deno" → "Bun vs Deno", "Bun advantages", "Deno advantages"
Tutorial Add "tutorial", "guide", "step by step" "Rust CLI" → "Rust CLI tutorial", "Rust CLI guide step by step"
Exploratory Split into 2-3 angles "RISC-V" → "RISC-V overview", "RISC-V ecosystem", "RISC-V use cases"
News Add "news", "announcement", date "AI 新闻" → "AI news this week 2026", "AI announcement latest"
Resource Add specific resource types "Anthropic MCP" → "Anthropic MCP official documentation"

Phase 3: Multi-Source Parallel Retrieval

Step 1: Brave (all modes)

Call web_searchfor each sub-query. If the intent has a freshness requirement, pass the freshness parameter:

web_search(query="Deno 2.0 latest 2026", freshness="pw")

Step 2: Exa + Tavily + Grok (Deep / Answer modes)

Call search.py for sub-queries, passing intent and freshness:

python3 /home/node/.openclaw/workspace/skills/search-layer/scripts/search.py \
  --queries "子查询1" "子查询2" "子查询3" \
  --mode deep \
  --intent status \
  --freshness pw \
  --num 5

Source Participation Matrix by Mode：

Mode Exa Tavily Grok Description
fast ✅ ❌ fallback Exa preferred; use Grok when there is no Exa key
deep ✅ ✅ ✅ Three sources in parallel
answer ❌ ✅ ❌ Tavily only (with AI answer)

Parameter Description：

Parameter Description
--queries Multiple sub-queries execute in parallel (a single query can also be passed as a positional argument)
--mode fast / deep / answer
--intent Intent type, affects scoring weights (if not passed, no scoring is done and behavior is consistent with v1)
--freshness pd(24h) / pw(week) / pm(month) / py(year)
--domain-boost Comma-separated domains; matching results get authority score +0.2
--num Number of results per source per query

Grok Source Description：

  • Call the Grok model via the completions API (grok-4.1-fast) and use its real-time knowledge to return structured search results
  • Automatically detect time-sensitive queries and inject the current time context
  • Execute in parallel with Exa and Tavily in deep mode
  • Requires configuring Grok's ~/.openclaw/credentials/search.json in apiUrl、apiKey、model(or via the environment variable GROK_API_URL、GROK_API_KEY、GROK_MODEL）
  • If the Grok configuration is missing, automatically fall back to the dual-source Exa + Tavily

Step 3: Merge

Merge the Brave results with the search.py output. Deduplicate by canonical URL and mark the source.

If search.py returns a score field, use it for sorting; for Brave results without a score, calculate it using the same intent weight formula.

Phase 3.5: Reference Tracking (Thread Pulling)

When the search results contain GitHub issue/PR links and the intent is Status or Exploratory, automatically trigger reference tracking.

Automatic Trigger Conditions

  • The intent is status or exploratory
  • The search results contain github.com/.../issues/ or github.com/.../pull/ URL

Method 1: search.py --extract-refs (batch)

Extract the reference graph directly from the search results, with no extra calls needed:

python3 search.py "OpenClaw config validation bug" --mode deep --intent status --extract-refs

The output will have an extra refs field containing the reference list for each result URL.

You can also skip the search and extract references directly for a known URL:

python3 search.py --extract-refs-urls "https://github.com/owner/repo/issues/123" "https://github.com/owner/repo/issues/456"

Method 2: fetch-thread (deep fetch for a single URL)

Fetch the full discussion thread + structured references for a single URL:

python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --format json
python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --format markdown
python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --extract-refs-only

GitHub scenario (issue/PR): fetch the body + all comments + timeline events (cross-references, commits) via the API, and extract:

  • Issue/PR references (#123, owner/repo#123)
  • Duplicate markers
  • Commit references
  • Related PRs/issues (timeline cross-references)
  • External URLs

General web scenario: web fetch + regex extraction of reference links.

Agent Execution Flow

Step 1: search-layer 搜索 → 获取初始结果
Step 2: search.py --extract-refs 或 fetch-thread → 提取线索图
Step 3: Agent 筛选高价值线索（LLM 判断哪些值得追踪）
Step 4: fetch-thread 深度抓取每个高价值线索
Step 5: 重复 Step 2-4，直到信息闭环或达到深度限制（建议 max_depth=3）

Phase 4: Result Ranking

Scoring Formula

score = w_keyword × keyword_match + w_freshness × freshness_score + w_authority × authority_score

The weights are determined by the intent (see the Phase 1 table). Each component score:

  • keyword_match (0-1): Coverage of query terms in the title + summary
  • freshness_score (0-1): Based on the publication date, the newer the higher (no date = 0.5)
  • authority_score (0-1): Based on the domain authority tier
    • Tier 1 (1.0): github.com, stackoverflow.com, official documentation sites
    • Tier 2 (0.8): HN, dev.to, well-known tech blogs
    • Tier 3 (0.6): Medium, Juejin, InfoQ
    • Tier 4 (0.4): Others

See the full domain scoring table in references/authority-domains.json

Domain Boost

Use the --domain-boost parameter to manually specify the domains to weight (matching results get authority score +0.2):

search.py "query" --mode deep --intent tutorial --domain-boost dev.to,freecodecamp.org

Recommended combinations:

  • Tutorial → dev.to, freecodecamp.org, realpython.com, baeldung.com
  • Resource → github.com
  • News → techcrunch.com, arstechnica.com, theverge.com

Phase 5: Knowledge Synthesis

Choose a synthesis strategy based on the number of results:

Small result set (≤5 items)

Show each item individually, each with a source tag and score:

1. [Title](url) — snippet... `[brave, exa]` ⭐0.85
2. [Title](url) — snippet... `[tavily]` ⭐0.72

Medium result set (5-15 items)

Cluster by topic + a summary for each group:

**主题 A: [描述]**
- [结果1] — 要点... `[source]`
- [结果2] — 要点... `[source]`

**主题 B: [描述]**
- [结果3] — 要点... `[source]`

Large result set (15+ items)

High-level overview + Top 5 + in-depth tips:

[一段综述，概括主要发现]

**Top 5 最相关结果：**
1. ...
2. ...

共找到 N 条结果，覆盖 [源列表]。需要深入哪个方面？

Synthesis rules

  • Give the answer first, then list the sources(don't start by saying "what I searched")
  • Aggregate by topic, not by source(don't do "Brave results: ... Exa results: ...")
  • Explicitly flag conflicting information: clearly point out when different sources contradict each other
  • Confidence expression：
    • Multiple sources agree + fresh → state directly
    • Single source or older → "According to [source], ..."
    • Conflicting or uncertain → "There are differing accounts: A believes ..., B believes ..."

Fallback strategy

  • Exa 429/5xx → continue with Brave + Tavily + Grok
  • Tavily 429/5xx → continue with Brave + Exa + Grok
  • Grok timeout/error → continue with Brave + Exa + Tavily
  • search.py fails entirely → use Brave only web_search(always available)
  • Never block the main flow just because one source fails

Backward compatibility

When the --intent parameter is not provided, search.py behaves exactly the same as v1 (no scoring, output in the original order).

Existing callers (such as github-explorer) require no changes.

Quick reference

Scenario Command
Quick facts web_search + search.py --mode answer --intent factual
In-depth research web_search + search.py --mode deep --intent exploratory
Latest updates web_search(freshness="pw") + search.py --mode deep --intent status --freshness pw
Comparative analysis web_search × 3 queries + search.py --queries "A vs B" "A pros" "B pros" --intent comparison
Find resources web_search + search.py --mode fast --intent resource