SuperAPI AI Directory

web-search

General & other

View repository

Updated 10-18 10:18

About this skill

Web Search

Overview

Search the web using ddgs to find information across web pages, news articles, images, videos, and books. Use scripts/fetch.py after search when the answer requires reading a page rather than only listing results. The bundled scripts mirror the Nymbo-Tools Web_Search and Web_Fetch MCP tools: backend fallbacks, page/offset pagination, smart news fallback, rich result formatting, clean Markdown extraction, raw HTML, link scraping, and best-effort in-process rate limiting.

Prerequisites

Install dependencies if they are missing:

pip install ddgs requests beautifulsoup4 lxml markdownify

This library provides a simple Python interface to DuckDuckGo's search API without requiring API keys or authentication.

Core Capabilities

1. Basic Web Search

Search for web pages and information:

python scripts/search.py "<query>"

Example:

python scripts/search.py "python asyncio tutorial"

Returns the top 10 web results with titles, URLs, and descriptions in a clean text format.

2. Limiting Results

Control the number of results returned:

python scripts/search.py "<query>" --max-results <N>

Example:

python scripts/search.py "machine learning frameworks" --max-results 20

Useful for:

  • Getting more comprehensive results (increase limit)
  • Quick lookups with fewer results (decrease limit)
  • Balancing detail vs. processing time

3. Time Range Filtering

Filter results by recency:

python scripts/search.py "<query>" --date-filter <day|week|month|year>

Time range options:

  • d - Past day
  • w - Past week
  • m - Past month
  • y - Past year

Example:

python scripts/search.py "artificial intelligence news" --date-filter week

Great for:

  • Finding recent news or updates
  • Filtering out outdated content
  • Tracking recent developments

4. News Search

Search specifically for news articles:

python scripts/search.py "<query>" --type news

Example:

python scripts/search.py "climate change" --type news --date-filter week --max-results 15

News results include:

  • Article title
  • Source publication
  • Publication date
  • URL
  • Article summary/description

If news search returns no results, the script automatically tries general web search and labels the output as a smart fallback.

5. Image Search

Search for images:

python scripts/search.py "<query>" --type images

Example:

python scripts/search.py "sunset over mountains" --type images --max-results 20

Image results include:

  • Image title
  • Image URL (direct link to image)
  • Thumbnail URL
  • Source website
  • Dimensions (width x height)

6. Video Search

Search for videos:

python scripts/search.py "<query>" --type videos

Example:

python scripts/search.py "python tutorial" --type videos --max-results 15

Video results include:

  • Video title
  • Publisher/channel
  • Duration
  • Publication date
  • Video URL
  • Description

7. Fetch a Result Page

After search identifies a useful URL, fetch readable page content:

python scripts/fetch.py "https://example.com/article" --mode markdown --max-chars 5000
python scripts/fetch.py "https://example.com/article" --mode links
python scripts/fetch.py "https://example.com/article" --mode html --max-chars 0

If output is truncated, use the printed Next cursor with --offset.

8. Pagination

Use page-based or exact-offset pagination:

python scripts/search.py "OpenAI news" --type news --page 2 --max-results 5
python scripts/search.py "OpenAI news" --type news --offset 5 --max-results 5

--offset overrides --page when it is greater than 0. Search output includes the next page and exact next offset when more results are available.

9. Output Formats

Choose how results are formatted:

Text format (default):

python scripts/search.py "quantum computing"

Clean, readable plain text with numbered results.

Markdown format:

python scripts/search.py "quantum computing" --format markdown

Formatted markdown with headers, bold text, and links.

JSON format:

python scripts/search.py "quantum computing" --format json

Structured JSON data for programmatic processing.

10. Saving Results to File

Save search results to a file:

python scripts/search.py "<query>" --output <file-path>

Example:

python scripts/search.py "artificial intelligence" --output ai_results.txt
python scripts/search.py "AI news" --type news --format markdown --output ai_news.md
python scripts/search.py "AI research" --format json --output ai_data.json

The file format is determined by the --format flag, not the file extension.

Output Format Examples

Text Format

1. Page Title Here
   URL: https://example.com/page
   Brief description of the page content...

2. Another Result
   URL: https://example.com/another
   Another description...

Markdown Format

## 1. Page Title Here

**URL:** https://example.com/page

Brief description of the page content...

## 2. Another Result

**URL:** https://example.com/another

Another description...

JSON Format

[
  {
    "title": "Page Title Here",
    "href": "https://example.com/page",
    "body": "Brief description of the page content..."
  },
  {
    "title": "Another Result",
    "href": "https://example.com/another",
    "body": "Another description..."
  }
]

Common Usage Patterns

Research on a Topic

Gather comprehensive information about a subject:

# Get overview from web
python scripts/search.py "machine learning basics" --max-results 15 --output ml_web.txt

# Get recent news
python scripts/search.py "machine learning" --type news --date-filter month --output ml_news.txt

# Find tutorial videos
python scripts/search.py "machine learning tutorial" --type videos --max-results 10 --output ml_videos.txt

Current Events Monitoring

Track news on specific topics:

python scripts/search.py "climate summit" --type news --date-filter day --format markdown --output daily_climate_news.md

Finding Visual Resources

Search for images with specific criteria:

python scripts/search.py "data visualization examples" --type images --max-results 20 --output viz_images.txt

Fact-Checking

Verify information with recent sources:

python scripts/search.py "specific claim to verify" --date-filter week --max-results 20

Academic Research

Find resources on scholarly topics:

python scripts/search.py "quantum entanglement research" --date-filter year --max-results 20 --output quantum_research.txt

Market Research

Gather information about products or companies:

python scripts/search.py "electric vehicle market 2025" --max-results 20 --format markdown --output ev_market.md
python scripts/search.py "EV news" --type news --date-filter month --output ev_news.txt

Implementation Approach

When users request web searches:

  1. Identify search intent:

    • What type of content (web, news, images, videos)?
    • How recent should results be?
    • How many results are needed?
    • Any filtering requirements?
  2. Configure search parameters:

    • Choose appropriate search type (--type)
    • Set a date filter if currency matters (--date-filter)
    • Adjust result count (--max-results)
    • Apply filters (image size, video duration, etc.)
  3. Select output format:

    • Text for quick reading
    • Markdown for documentation
    • JSON for further processing
  4. Execute search:

    • Run the search command
    • Save to file if results need to be preserved
    • Print to stdout for immediate review
  5. Process results:

    • Read saved files if needed
    • Extract URLs or specific information
    • Combine results from multiple searches

Quick Reference

Command structure:

python scripts/search.py "<query>" [options]

Essential options:

  • -t, --type - Search type (text, news, images, videos, books)
  • -n, --max-results - Maximum results (default: 10)
  • --date-filter - Time filter (day, week, month, year)
  • --page - 1-based page number
  • --offset - Continue from a later result offset
  • --backend - Backend or auto fallback order
  • -f, --format - Output format (text, markdown, json)
  • -o, --output - Save to file

Get full help:

python scripts/search.py --help

Best Practices

  1. Be specific - Use clear, specific search queries for better results
  2. Use time filters - Apply --date-filter for current information
  3. Adjust result count - Start with 10-20 results, increase if needed
  4. Save important searches - Use --output to preserve results
  5. Choose appropriate type - Use news search for current events, web for general info
  6. Use JSON for automation - JSON format is easiest to parse programmatically
  7. Respect usage - Don't hammer the API with rapid repeated searches

Troubleshooting

Common issues:

  • "Missing required dependency": Run pip install ddgs requests beautifulsoup4 lxml markdownify
  • No results found: Try broader search terms, remove time filters, or use text search. News searches already fall back to text automatically.
  • Timeout errors: The search service may be temporarily unavailable; retry after a moment
  • Rate limiting: Space out searches if making many requests
  • Unexpected results: DuckDuckGo's results may differ from Google; try refining the query

Limitations:

  • Results quality depends on DuckDuckGo's index and algorithms
  • Search operators supported by the selected backend, such as site:, quotes, and OR, are passed through to ddgs; exact support varies by backend.
  • Image and video searches may have fewer results than web search
  • No control over result ranking or relevance scoring
  • Some specialized searches may work better on dedicated search engines

Advanced Use Cases

Combining Multiple Searches

Gather comprehensive information by combining search types:

# Web overview
python scripts/search.py "topic" --max-results 15 --output topic_web.txt

# Recent news
python scripts/search.py "topic" --type news --date-filter week --output topic_news.txt

# Images
python scripts/search.py "topic" --type images --max-results 20 --output topic_images.txt

Programmatic Processing

Use JSON output for automated processing:

python scripts/search.py "research topic" --format json --output results.json
# Then process with another script
python analyze_results.py results.json

Building a Knowledge Base

Create searchable documentation from web results:

# Search multiple related topics
python scripts/search.py "topic1" --format markdown --output kb/topic1.md
python scripts/search.py "topic2" --format markdown --output kb/topic2.md
python scripts/search.py "topic3" --format markdown --output kb/topic3.md

Resources

scripts/search.py

The main search tool implementing DuckDuckGo search functionality. Key features:

  • Multiple search types - Text, news, images, videos, and books
  • Flexible filtering - Date filter, backend selection, and offset pagination
  • Multiple output formats - Text, Markdown, and JSON
  • File output - Save results for later processing
  • Clean formatting - Human-readable output with all essential information
  • Error handling - Graceful handling of network errors and empty results

The script can be executed directly and includes comprehensive command-line help via --help.

scripts/fetch.py

Fetches a URL found through search and returns clean Markdown, raw HTML, or links. Use --max-chars and --offset for pagination on long pages.