SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillssearch-layer

search-layer

General & other

View repository

Updated 10-15 11:46

About this skill

Search Layer v2.2 — Intent-Aware Multi-Source Retrieval Protocol

Four sources at the same level: Brave (web_search) + Exa + Tavily + Grok. Automatically selects strategy, adjusts weights, and performs synthesis based on intent.

Execution Flow

用户查询
    ↓
[Phase 1] 意图分类 → 确定搜索策略
    ↓
[Phase 2] 查询分解 & 扩展 → 生成子查询
    ↓
[Phase 3] 多源并行检索 → Brave + search.py (Exa + Tavily + Grok)
    ↓
[Phase 4] 结果合并 & 排序 → 去重 + 意图加权评分
    ↓
[Phase 5] 知识合成 → 结构化输出

Phase 1: Intent Classification

After receiving a search request,first determine the intent type, then decide the search strategy. Do not ask the user which mode to use.

Intent Recognition Signal Mode Freshness Weight Bias
Factual "What is X", "definition of X", "What is X" answer — Authority 0.5
Status "latest developments of X", "current state of X", "latest X" deep pw/pm Freshness 0.5
Comparison "X vs Y", "difference between X and Y" deep py Keywords 0.4 + Authority 0.4
Tutorial "how to do X", "X tutorial", "how to X" answer py Authority 0.5
Exploratory "in-depth understanding of X", "X ecosystem", "about X" deep — Authority 0.5
News "X news", "X this week", "X this week" deep pd/pw Freshness 0.6
Resource "X official site", "X GitHub", "X documentation" fast — Keywords 0.5

For the detailed classification guide, see references/intent-guide.md

Decision Rules：

  1. Scan the query for signal words
  2. When multiple types match, choose the most specific one
  3. When unable to determine, default to exploratory

Phase 2: Query Decomposition & Expansion

Based on the intent type, expand the user query into a set of sub-queries:

General Rules

  • Automatic technical synonym expansion：k8s→Kubernetes, JS→JavaScript, Go→Golang, Postgres→PostgreSQL
  • Chinese technical queries: also generate English variants (e.g. "Rust 异步编程" → additionally search "Rust async programming")

Expansion by Intent

Intent Expansion Strategy Example
Factual Add "definition", "explained" "WebTransport" → "WebTransport", "WebTransport explained overview"
Status Add year, "latest", "update" "Deno 进展" → "Deno 2.0 latest 2026", "Deno update release"
Comparison Split into 3 sub-queries "Bun vs Deno" → "Bun vs Deno", "Bun advantages", "Deno advantages"
Tutorial Add "tutorial", "guide", "step by step" "Rust CLI" → "Rust CLI tutorial", "Rust CLI guide step by step"
Exploratory Split into 2-3 angles "RISC-V" → "RISC-V overview", "RISC-V ecosystem", "RISC-V use cases"
News Add "news", "announcement", date "AI 新闻" → "AI news this week 2026", "AI announcement latest"
Resource Add specific resource type "Anthropic MCP" → "Anthropic MCP official documentation"

Phase 3: Multi-Source Parallel Retrieval

Step 1: Brave (all modes)

Call for each sub-query web_search. If the intent has a freshness requirement, pass the freshness parameter:

web_search(query="Deno 2.0 latest 2026", freshness="pw")

Step 2: Exa + Tavily + Grok (Deep / Answer modes)

Call search.py for sub-queries, passing the intent and freshness:

python3 /home/node/.openclaw/workspace/skills/search-layer/scripts/search.py \
  --queries "子查询1" "子查询2" "子查询3" \
  --mode deep \
  --intent status \
  --freshness pw \
  --num 5

Source Participation Matrix by Mode：

Mode Exa Tavily Grok Description
fast ✅ ❌ fallback Exa preferred; use Grok when there is no Exa key
deep ✅ ✅ ✅ Three sources in parallel
answer ❌ ✅ ❌ Tavily only (including AI answer)

Parameter Description：

Parameter Description
--queries Multiple sub-queries executed in parallel (a single query can also be passed as a positional argument)
--mode fast / deep / answer
--intent Intent type, affects scoring weights (if not passed, no scoring is performed, and behavior is consistent with v1)
--freshness pd(24h) / pw(week) / pm(month) / py(year)
--domain-boost Comma-separated domains; matching results get authority score +0.2
--num Number of results per source per query

Grok Source Description：

  • Call the Grok model via the completions API (grok-4.1-fast), using its real-time knowledge to return structured search results
  • Automatically detect time-sensitive queries and inject current time context
  • Execute in parallel with Exa and Tavily in deep mode
  • Requires configuring Grok's ~/.openclaw/credentials/search.json in apiUrl、apiKey、model(or via the environment variable GROK_API_URL、GROK_API_KEY、GROK_MODEL）
  • If Grok configuration is missing, automatically degrade to Exa + Tavily dual-source

Step 3: Merge

Merge Brave results with search.py output. Deduplicate by canonical URL, mark the source.

If search.py returns the score field, use it for sorting; for Brave results without a score, calculate it using the same intent weight formula.

Phase 3.5: Citation Tracking (Thread Pulling)

When search results contain GitHub issue/PR links and the intent is Status or Exploratory, automatically trigger citation tracking.

Automatic Trigger Conditions

  • Intent is status or exploratory
  • Search results contain github.com/.../issues/ or github.com/.../pull/ URL

Method 1: search.py --extract-refs (batch)

Extract the citation graph directly on the search results, no additional calls needed:

python3 search.py "OpenClaw config validation bug" --mode deep --intent status --extract-refs

The output will have an extra refs field, containing the citation list for each result URL.

You can also skip the search and extract citations directly for known URLs:

python3 search.py --extract-refs-urls "https://github.com/owner/repo/issues/123" "https://github.com/owner/repo/issues/456"

Method 2: fetch-thread (single URL deep fetch)

Fetch the complete discussion thread + structured citations for a single URL:

python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --format json
python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --format markdown
python3 fetch_thread.py "https://github.com/owner/repo/issues/123" --extract-refs-only

GitHub scenario (issue/PR): fetch the body + all comments + timeline events (cross-references, commits) via the API, and extract:

  • Issue/PR citations (#123, owner/repo#123)
  • Duplicate marker
  • Commit citations
  • Related PR/issue (timeline cross-references)
  • External URL

Generic web scenario: web fetch + regex extraction of citation links.

Agent Execution Flow

Step 1: search-layer 搜索 → 获取初始结果
Step 2: search.py --extract-refs 或 fetch-thread → 提取线索图
Step 3: Agent 筛选高价值线索（LLM 判断哪些值得追踪）
Step 4: fetch-thread 深度抓取每个高价值线索
Step 5: 重复 Step 2-4，直到信息闭环或达到深度限制（建议 max_depth=3）

Phase 4: Result Ranking

Scoring Formula

score = w_keyword × keyword_match + w_freshness × freshness_score + w_authority × authority_score

Weights are determined by intent (see the Phase 1 table). Each sub-score:

  • keyword_match (0-1): coverage of query terms in title + summary
  • freshness_score (0-1): based on publish date, newer is higher (no date = 0.5)
  • authority_score (0-1): based on domain authority tier
    • Tier 1 (1.0): github.com, stackoverflow.com, official documentation sites
    • Tier 2 (0.8): HN, dev.to, well-known tech blogs
    • Tier 3 (0.6): Medium, Juejin, InfoQ
    • Tier 4 (0.4): others

See the full domain scoring table at references/authority-domains.json

Domain Boost

Use the --domain-boost parameter to manually specify domains that need weighting (matching results get authority score +0.2):

search.py "query" --mode deep --intent tutorial --domain-boost dev.to,freecodecamp.org

Recommended combinations:

  • Tutorial → dev.to, freecodecamp.org, realpython.com, baeldung.com
  • Resource → github.com
  • News → techcrunch.com, arstechnica.com, theverge.com

Phase 5: Knowledge Synthesis

Choose a synthesis strategy based on the number of results:

Small result set (≤5 items)

Display one by one, each with a source label and score:

1. [Title](url) — snippet... `[brave, exa]` ⭐0.85
2. [Title](url) — snippet... `[tavily]` ⭐0.72

Medium result set (5-15 items)

Cluster by topic + summarize each group:

**主题 A: [描述]**
- [结果1] — 要点... `[source]`
- [结果2] — 要点... `[source]`

**主题 B: [描述]**
- [结果3] — 要点... `[source]`

Large result set (15+ items)

High-level summary + Top 5 + deep dive tips:

[一段综述，概括主要发现]

**Top 5 最相关结果：**
1. ...
2. ...

共找到 N 条结果，覆盖 [源列表]。需要深入哪个方面？

Synthesis rules

  • Give the answer first, then list the sources(Don't start with "what I searched")
  • Aggregate by topic, not by source(Don't do "Brave results: ... Exa results: ...")
  • Explicitly flag conflicting information: clearly point out when different sources contradict each other
  • Confidence expression：
    • Multiple sources agree + fresh → state directly
    • Single source or older → "According to [source], ..."
    • Conflicting or uncertain → "There are different accounts: A believes ..., B believes ..."

Fallback strategy

  • Exa 429/5xx → continue with Brave + Tavily + Grok
  • Tavily 429/5xx → continue with Brave + Exa + Grok
  • Grok timeout/error → continue with Brave + Exa + Tavily
  • search.py overall failure → use only Brave web_search(always available)
  • Never block the main flow because a source fails

Backward compatibility

When no --intent parameter is provided, search.py behaves exactly the same as v1 (no scoring, output in original order).

Existing callers (such as github-explorer) require no changes.

Quick reference

Scenario Command
Quick facts web_search + search.py --mode answer --intent factual
Deep research web_search + search.py --mode deep --intent exploratory
Latest updates web_search(freshness="pw") + search.py --mode deep --intent status --freshness pw
Comparative analysis web_search × 3 queries + search.py --queries "A vs B" "A pros" "B pros" --intent comparison
Find resources web_search + search.py --mode fast --intent resource