High-performance Bing search engine, implementing deep web information extraction based on Playwright.
Core Features
1. Bing Search
Execute a Bing search and return structured results:
- Title, URL, summary, source website
- Automatically filter ads and irrelevant content
- Supports Chinese and English searches
2. Deep Page Analysis
Perform a deep scan of the specified URL and extract:
- All links: text, href, type
- Form information: action, method, input fields
- Button elements: text, type, action
- External scripts: list of JS file URLs
- Page metadata: meta tags, Open Graph, etc.
Usage
Basic Search
python scripts/bing_search.py "OpenClaw AI Agent"
Deep Page Analysis
python scripts/bing_search.py "placeholder" --deep https://example.com
Python API
from bing_search import BingSearchAgent, SearchResult
async with BingSearchAgent(headless=True) as agent:
# 搜索
results = await agent.search("关键词", num_results=10)
# 深度分析
elements = await agent.extract_page_elements("https://example.com")
Dependency Requirements
- Python 3.8+
- playwright (
pip install playwright) - Chrome/Chromium browser
Configuration Instructions
The script searches for the following Chrome paths by default:
~/.local/bin/chrome-for-testing-dir/chrome/usr/bin/google-chrome/usr/bin/chromium
You can customize the paths by modifying the CHROME_PATHS list in the script.
Anti-Detection Features
- Disable automation control flags (
--disable-blink-features=AutomationControlled) - Simulate a real user agent
- Set a reasonable viewport size
- Randomize some behavior patterns