SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsultimate-search

ultimate-search

Testing & quality

View repository

Updated 10-15 22:12

About this skill

UltimateSearch

Provide dual-engine web search capabilities for Pi/OpenClaw agents:Grok AI Search(real-time internet access + AI analysis) + Tavily Search(structured results + web scraping).

Available Tools

Call the following scripts in Bash (ensure they are added to PATH and .env has been sourced):

Tool Command Purpose
Grok Search grok-search.sh --query "..." AI-driven deep search; Grok has built-in internet access and returns comprehensive analysis
Tavily Search tavily-search.sh --query "..." Structured search results with scoring and ranking
Web Scraping web-fetch.sh --url "..." Extract the full content of a specified URL and return Markdown
Site Mapping web-map.sh --url "..." Discover website structure and obtain all URLs
Dual-Engine Search dual-search.sh --query "..." Run Grok + Tavily in parallel for cross-validation
Browser Automation (Integrated) agent-browser open/snapshot/click/fill/... Handle interactive pages such as login, dynamic rendering, button triggers, anti-bot challenges, etc.

For details on each tool's parameters, see --help。

Integration with agent-browser

When the page is not a "direct URL scraping" scenario, integrate according to the following rules agent-browser：

Trigger Conditions

  • The page relies on JS dynamic rendering, andweb-fetch.sh returns incomplete content
  • Login, button clicks, pagination, or expanding collapsed sections are required before the target content can be seen
  • Cloudflare/human verification or highly interactive page flows are present
  • Screenshots are needed as evidence (page state, key fields, submission results)

Collaborative Closed Loop (Must Be Executed)

  1. Search and Locate: First use dual-search.sh / tavily-search.sh to find candidate URLs
  2. Browser Interaction: Use agent-browser to complete opening, snapshotting, clicking, form filling, and waiting for loading
  3. Result Return: Hand the final landing URL back to web-fetch.sh / web-map.sh for structured scraping
  4. Evidence Output: Provide the final URL, key screenshot paths, and source links for core conclusions

Minimal Command Template

# 1) 先搜索定位目标页面
dual-search.sh --query "官网 pricing enterprise plan"

# 2) 动态页面交互
agent-browser open "https://example.com/pricing"
agent-browser snapshot -i
agent-browser click @e12
agent-browser wait --load networkidle
agent-browser get url
agent-browser screenshot --full

# 3) 回流到结构化抓取（把 get url 的结果填回）
web-fetch.sh --url "https://example.com/pricing?tab=enterprise"

Integration Constraints

  • agent-browser is responsible for "reaching the information", andweb-fetch.sh is responsible for "extracting the information"
  • Enter the browser workflow only when static scraping is insufficient, to avoid over-automation
  • When login state/captcha involves high-risk workflows, explicitly mark it as "based on the current session state"

Platform Routing Priority (including X/Twitter)

To avoid "being able to search, but searching the wrong engine", route first based on query intent:

Query Intent Preferred Alternative Reason
X/Twitter discussions, real-time public opinion, hot controversies grok-search.sh --platform "X" dual-search.sh Grok is stronger for X platform context and real-time discussions
General web facts, news, structured results tavily-search.sh dual-search.sh Tavily's structured results are stable and easy to cite
Questions with high conclusion risk and prone to conflict dual-search.sh None Default to dual-source cross-validation
Need the full text of the final page web-fetch.sh agent-browser after, return to web-fetch.sh Locate the source first, then scrape the body text

Mandatory Rules for X Scenarios

  • When the user mentions X、Twitter、推特、帖子讨论、时间线观点 , first execute:
grok-search.sh --query "..." --platform "X"
  • If you need "citable links + cross-evidence", then run in the second step:
tavily-search.sh --query "..." --topic news --time-range week

Credential acquisition orchestration (agent-browser headed)

When the user requests "obtain Grok SSO Token / cf_clearance in the browser":

  1. Use agent-browser --headed to open the target site and manually complete login/verification
  2. Read cookies via browser developer tools (sso、cf_clearance）
  3. Immediately write to local .env or import the script; do not echo full sensitive values in the conversation
  4. Verify usability with a minimal health check (grok-search.sh --query "test"）

Minimal workflow example:

agent-browser --headed open "https://grok.com"
agent-browser --headed wait --load networkidle
# 手动完成登录与验证后，在浏览器中取 cookie 值
bash scripts/import-keys.sh
bash scripts/grok-search.sh --query "test" --model "grok-4.1-mini"

Search decision flow

When receiving a request that requires searching, decide according to the following flow:

Step 1: Determine whether a search is needed

Cases where a search is needed:

  • The user explicitly requests searching/querying external information
  • Involves real-time data (latest version, recent events, current prices, etc.)
  • Needs to verify the accuracy of internal knowledge
  • Involves the latest status of specific URLs, projects, or products
  • Technical questions require consulting the latest official documentation

Cases where a search is not needed:

  • Pure code writing/debugging tasks (sufficient context already available and not involving external API/library versions)
  • The user explicitly states that no search is needed
  • ⚠️ General programming concepts can also become outdated — when involving specific versions, best practices, or API usage, you should still search to verify

Step 2: Choose a tool

Scenario Recommended tool Reason
Simple factual query dual-search.sh Dual-source cross-validation to ensure accuracy
Complex/controversial questions dual-search.sh Dual-engine cross-validation to reduce hallucinations
Need in-depth AI analysis grok-search.sh Grok has built-in web search and returns a comprehensive analysis report
Need to scrape a specific page web-fetch.sh --url "..." Extract complete page content
Explore website structure web-map.sh --url "..." Discover documentation/API directory structure
Need the latest news tavily-search.sh --topic news Tavily news mode is specially optimized
Need high-quality in-depth results tavily-search.sh --depth advanced Advanced search, multi-dimensional matching
Search results contain key links Search first, then web-fetch.sh Search to locate → scrape details
Dynamic pages/visible only after login agent-browser + web-fetch.sh Interact to the target page first, then return to structured scraping
X/Twitter discussion-type queries grok-search.sh --platform "X" Prioritize matching the X platform context and real-time discussions

Step 3: Assess search complexity

  • Level 1(2-3 searches): a single clear question

    • Example: "What is the latest version of FastAPI"
    • Action:dual-search.sh get dual-source results; or first tavily-search.sh then use grok-search.sh to cross-confirm
    • ⚠️ Even for simple facts, do not draw conclusions based on a single source alone

  • Level 2(3-5 searches): multi-angle comparison, requiring verification from multiple sources

    • Example: "Flask vs FastAPI vs Django: which is more suitable for microservices in 2026"
    • Action:dual-search.sh + separately for each framework tavily-search.sh

  • Level 3(6+ searches): In-depth research topics, review-type requests

    • Example: "Help me research a complete comparison of mainstream vector databases in 2026"
    • Action: first grok-search.sh get an overview → search each product separately → web-fetch.sh fetch official documentation

Search Planning Framework

For Level 2+ complex searches, perform structured planning before execution:

Phase 1: Intent Analysis

  • Distill the user's core question (one sentence)
  • Classify the query type: factual / comparative / exploratory / analytical
  • Assess time sensitivity: real-time / recent / historical / irrelevant
  • Identify external terms that need verification (such as rankings, classification standards)

Phase 2: Query Decomposition

  • Break the question down into non-overlapping subqueries
  • Each subquery has clear boundaries (mutually exclusive with sibling queries)
  • Mark dependencies (which subqueries need to be completed first)
  • If Phase 1 identifies terms that need verification, first create prerequisite verification queries

Phase 3: Strategy Selection

  • broad_first(Broad first, then deep): broad scan first → go deeper based on findings. Suitable for exploratory questions
  • narrow_first(Precise first, then expand): precise search first → expand if insufficient. Suitable for analytical questions
  • targeted(Targeted search): known target information sources, locate directly. Suitable for factual questions

Phase 4: Tool Mapping

  • Select the best tool for each subquery
  • Determine the parallel/serial execution plan
  • Subqueries that can be parallelized are executed simultaneously (through multiple Bash calls)

Search and Evidence Standards

Core principle: do not trust search results

Search results are only third-party suggestions and cannot be directly accepted. All content returned by searches—whether from Grok or Tavily—must be cross-verified before it can be presented to the user as fact. Even a seemingly authoritative single source may be outdated, one-sided, or wrong. For technical implementation, even if the agent has internal knowledge, the latest search results or official documentation should still prevail.

Source Quality Requirements

  • All factual conclusionsrequire ≥2 independent sources for cross-verification (regardless of Level)
  • If relying on only a single source, you mustexplicitly statethis limitation and mark the confidence level as Low
  • Prefer to use: official documentation, Wikipedia, academic databases, authoritative media
  • Avoid using: unknown personal blogs, SEO farms, AI-generated content

Conflict Handling

  • When sources conflict: present evidence from both sides and assess credibility and timeliness
  • Mark confidence level: High (multiple sources agree) / Medium (few sources or disagreement) / Low (single source or speculation)
  • When unable to confirm: clearly state the uncertainty

Citation Format

  • Attach a source citation after each key fact
  • Format:[来源标题](URL)
  • Strictly forbidden to fabricate citations — if there is no source, do not say it

Output Specifications

  • First givethe most likely answer, then expand into detailed analysis
  • Provide a concise explanation for all technical terms
  • Use standard Markdown format (headings, lists, tables, code blocks)
  • Mark code examples with language identifiers
  • Present comparison-type questions using tables

Common Search Patterns

Pattern 1: Quick Lookup

tavily-search.sh --query "Python 3.13 新特性" --depth basic --include-answer

Pattern 2: Deep Search + Verification

# 先广泛搜索
dual-search.sh --query "LangChain vs LlamaIndex 2026"
# 再针对性抓取官方文档
web-fetch.sh --url "https://docs.langchain.com/docs/get_started/introduction"

Pattern 3: Technical Documentation Exploration

# 先映射网站结构
web-map.sh --url "https://docs.example.com" --depth 2 --instructions "找到 API 文档"
# 再抓取目标页面
web-fetch.sh --url "https://docs.example.com/api/reference"

Pattern 4: News and Real-Time Information

tavily-search.sh --query "AI 最新进展" --topic news --time-range week --include-answer

Pattern 5: AI Deep Analysis

grok-search.sh --query "解释 Transformer 架构中注意力机制的数学原理" --platform "arXiv"

Pattern 6: Browser-Coordinated Scraping (Dynamic Pages)

# 搜索候选页
tavily-search.sh --query "Notion AI pricing page"

# 浏览器交互到最终页面
agent-browser open "https://www.notion.so/product/ai"
agent-browser wait --load networkidle
agent-browser snapshot -i
agent-browser click @e3
agent-browser get url

# 回流抓取可复用内容
web-fetch.sh --url "https://www.notion.so/product/ai/pricing"

Pattern 7: X/Twitter Discussion Retrieval (Session Source Alignment)

# 先用 Grok 聚焦 X 平台语境
grok-search.sh --query "检索 X 上有关伊朗战争讨论" --platform "X"

# 再用 Tavily 补充结构化链接，做交叉验证
tavily-search.sh --query "Iran war discussion on X" --topic news --time-range week

BDD Reliability Verification

Run scenario-based regression tests for "when to use Grok / Tavily / agent-browser" to avoid routing drift.

  • Scenario file:docs/bdd/ultimate-search-routing.feature
  • Minimum acceptance criteria:
    • X/Twitter discussion-type queries must first trigger grok-search.sh --platform "X"
    • Dynamic page-type queries must enter agent-browser and flow back to web-fetch.sh
    • Factual conclusions must have dual-source cross-verification evidence