Overview
Search the web using ddgs to find information across web pages, news articles, images, videos, and books. Use scripts/fetch.py after search when the answer requires reading a page rather than only listing results. The bundled scripts mirror the Nymbo-Tools Web_Search and Web_Fetch MCP tools: backend fallbacks, page/offset pagination, smart news fallback, rich result formatting, clean Markdown extraction, raw HTML, link scraping, and best-effort in-process rate limiting.
Prerequisites
Install dependencies if they are missing:
pip install ddgs requests beautifulsoup4 lxml markdownify
This library provides a simple Python interface to DuckDuckGo's search API without requiring API keys or authentication.
Core Capabilities
1. Basic Web Search
Search for web pages and information:
python scripts/search.py "<query>"
Example:
python scripts/search.py "python asyncio tutorial"
Returns the top 10 web results with titles, URLs, and descriptions in a clean text format.
2. Limiting Results
Control the number of results returned:
python scripts/search.py "<query>" --max-results <N>
Example:
python scripts/search.py "machine learning frameworks" --max-results 20
Useful for:
- Getting more comprehensive results (increase limit)
- Quick lookups with fewer results (decrease limit)
- Balancing detail vs. processing time
3. Time Range Filtering
Filter results by recency:
python scripts/search.py "<query>" --date-filter <day|week|month|year>
Time range options:
d- Past dayw- Past weekm- Past monthy- Past year
Example:
python scripts/search.py "artificial intelligence news" --date-filter week
Great for:
- Finding recent news or updates
- Filtering out outdated content
- Tracking recent developments
4. News Search
Search specifically for news articles:
python scripts/search.py "<query>" --type news
Example:
python scripts/search.py "climate change" --type news --date-filter week --max-results 15
News results include:
- Article title
- Source publication
- Publication date
- URL
- Article summary/description
If news search returns no results, the script automatically tries general web search and labels the output as a smart fallback.
5. Image Search
Search for images:
python scripts/search.py "<query>" --type images
Example:
python scripts/search.py "sunset over mountains" --type images --max-results 20
Image results include:
- Image title
- Image URL (direct link to image)
- Thumbnail URL
- Source website
- Dimensions (width x height)
6. Video Search
Search for videos:
python scripts/search.py "<query>" --type videos
Example:
python scripts/search.py "python tutorial" --type videos --max-results 15
Video results include:
- Video title
- Publisher/channel
- Duration
- Publication date
- Video URL
- Description
7. Fetch a Result Page
After search identifies a useful URL, fetch readable page content:
python scripts/fetch.py "https://example.com/article" --mode markdown --max-chars 5000
python scripts/fetch.py "https://example.com/article" --mode links
python scripts/fetch.py "https://example.com/article" --mode html --max-chars 0
If output is truncated, use the printed Next cursor with --offset.
8. Pagination
Use page-based or exact-offset pagination:
python scripts/search.py "OpenAI news" --type news --page 2 --max-results 5
python scripts/search.py "OpenAI news" --type news --offset 5 --max-results 5
--offset overrides --page when it is greater than 0. Search output includes the next page and exact next offset when more results are available.
9. Output Formats
Choose how results are formatted:
Text format (default):
python scripts/search.py "quantum computing"
Clean, readable plain text with numbered results.
Markdown format:
python scripts/search.py "quantum computing" --format markdown
Formatted markdown with headers, bold text, and links.
JSON format:
python scripts/search.py "quantum computing" --format json
Structured JSON data for programmatic processing.
10. Saving Results to File
Save search results to a file:
python scripts/search.py "<query>" --output <file-path>
Example:
python scripts/search.py "artificial intelligence" --output ai_results.txt
python scripts/search.py "AI news" --type news --format markdown --output ai_news.md
python scripts/search.py "AI research" --format json --output ai_data.json
The file format is determined by the --format flag, not the file extension.
Output Format Examples
Text Format
1. Page Title Here
URL: https://example.com/page
Brief description of the page content...
2. Another Result
URL: https://example.com/another
Another description...
Markdown Format
## 1. Page Title Here
**URL:** https://example.com/page
Brief description of the page content...
## 2. Another Result
**URL:** https://example.com/another
Another description...
JSON Format
[
{
"title": "Page Title Here",
"href": "https://example.com/page",
"body": "Brief description of the page content..."
},
{
"title": "Another Result",
"href": "https://example.com/another",
"body": "Another description..."
}
]
Common Usage Patterns
Research on a Topic
Gather comprehensive information about a subject:
# Get overview from web
python scripts/search.py "machine learning basics" --max-results 15 --output ml_web.txt
# Get recent news
python scripts/search.py "machine learning" --type news --date-filter month --output ml_news.txt
# Find tutorial videos
python scripts/search.py "machine learning tutorial" --type videos --max-results 10 --output ml_videos.txt
Current Events Monitoring
Track news on specific topics:
python scripts/search.py "climate summit" --type news --date-filter day --format markdown --output daily_climate_news.md
Finding Visual Resources
Search for images with specific criteria:
python scripts/search.py "data visualization examples" --type images --max-results 20 --output viz_images.txt
Fact-Checking
Verify information with recent sources:
python scripts/search.py "specific claim to verify" --date-filter week --max-results 20
Academic Research
Find resources on scholarly topics:
python scripts/search.py "quantum entanglement research" --date-filter year --max-results 20 --output quantum_research.txt
Market Research
Gather information about products or companies:
python scripts/search.py "electric vehicle market 2025" --max-results 20 --format markdown --output ev_market.md
python scripts/search.py "EV news" --type news --date-filter month --output ev_news.txt
Implementation Approach
When users request web searches:
-
Identify search intent:
- What type of content (web, news, images, videos)?
- How recent should results be?
- How many results are needed?
- Any filtering requirements?
-
Configure search parameters:
- Choose appropriate search type (
--type) - Set a date filter if currency matters (
--date-filter) - Adjust result count (
--max-results) - Apply filters (image size, video duration, etc.)
- Choose appropriate search type (
-
Select output format:
- Text for quick reading
- Markdown for documentation
- JSON for further processing
-
Execute search:
- Run the search command
- Save to file if results need to be preserved
- Print to stdout for immediate review
-
Process results:
- Read saved files if needed
- Extract URLs or specific information
- Combine results from multiple searches
Quick Reference
Command structure:
python scripts/search.py "<query>" [options]
Essential options:
-t, --type- Search type (text, news, images, videos, books)-n, --max-results- Maximum results (default: 10)--date-filter- Time filter (day, week, month, year)--page- 1-based page number--offset- Continue from a later result offset--backend- Backend orautofallback order-f, --format- Output format (text, markdown, json)-o, --output- Save to file
Get full help:
python scripts/search.py --help
Best Practices
- Be specific - Use clear, specific search queries for better results
- Use time filters - Apply
--date-filterfor current information - Adjust result count - Start with 10-20 results, increase if needed
- Save important searches - Use
--outputto preserve results - Choose appropriate type - Use news search for current events, web for general info
- Use JSON for automation - JSON format is easiest to parse programmatically
- Respect usage - Don't hammer the API with rapid repeated searches
Troubleshooting
Common issues:
- "Missing required dependency": Run
pip install ddgs requests beautifulsoup4 lxml markdownify - No results found: Try broader search terms, remove time filters, or use text search. News searches already fall back to text automatically.
- Timeout errors: The search service may be temporarily unavailable; retry after a moment
- Rate limiting: Space out searches if making many requests
- Unexpected results: DuckDuckGo's results may differ from Google; try refining the query
Limitations:
- Results quality depends on DuckDuckGo's index and algorithms
- Search operators supported by the selected backend, such as
site:, quotes, andOR, are passed through toddgs; exact support varies by backend. - Image and video searches may have fewer results than web search
- No control over result ranking or relevance scoring
- Some specialized searches may work better on dedicated search engines
Advanced Use Cases
Combining Multiple Searches
Gather comprehensive information by combining search types:
# Web overview
python scripts/search.py "topic" --max-results 15 --output topic_web.txt
# Recent news
python scripts/search.py "topic" --type news --date-filter week --output topic_news.txt
# Images
python scripts/search.py "topic" --type images --max-results 20 --output topic_images.txt
Programmatic Processing
Use JSON output for automated processing:
python scripts/search.py "research topic" --format json --output results.json
# Then process with another script
python analyze_results.py results.json
Building a Knowledge Base
Create searchable documentation from web results:
# Search multiple related topics
python scripts/search.py "topic1" --format markdown --output kb/topic1.md
python scripts/search.py "topic2" --format markdown --output kb/topic2.md
python scripts/search.py "topic3" --format markdown --output kb/topic3.md
Resources
scripts/search.py
The main search tool implementing DuckDuckGo search functionality. Key features:
- Multiple search types - Text, news, images, videos, and books
- Flexible filtering - Date filter, backend selection, and offset pagination
- Multiple output formats - Text, Markdown, and JSON
- File output - Save results for later processing
- Clean formatting - Human-readable output with all essential information
- Error handling - Graceful handling of network errors and empty results
The script can be executed directly and includes comprehensive command-line help via --help.
scripts/fetch.py
Fetches a URL found through search and returns clean Markdown, raw HTML, or links. Use --max-chars and --offset for pagination on long pages.