web-scraping-automation

Website Scraping and API Automation

Feature Description

This skill is specifically for automating website data scraping and API interface calls, including:

• Analyze and scrape website structures
• Call and test REST/GraphQL APIs
• Create automated crawler scripts
• Data parsing and cleaning
• Handle anti-scraping mechanisms
• Scheduled tasks and data storage

Use Cases

• "Scrape product information from this website"
• "Help me call this API and parse the returned data"
• "Create a script to regularly scrape news"
• "Analyze this website's API interface documentation"
• "Bypass this website's anti-scraping restrictions"

Technology Stack

⚠️ Resource Cleanup Principle (Mandatory)

After all scraping tasks involving browsers are completed, the Chrome/Selenium process must be automatically closed!

# Playwright 示例
from playwright.sync_api import sync_playwright

def scrape_website():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        # ... 爬取逻辑 ...
        browser.close()

    # ⚠️ 强制清理残留进程
    import subprocess
    subprocess.run(['pkill', '-f', 'chrome'], capture_output=True)

# Selenium 示例
from selenium import webdriver

driver = webdriver.Chrome()
try:
    # ... 爬取逻辑 ...
    pass
finally:
    driver.quit()
    # ⚠️ 确保清理
    import subprocess
    subprocess.run(['pkill', '-f', 'chrome'], capture_output=True)

Reason: Avoid memory leaks and resource usage, and prevent Gateway CPU from overloading at 100%

Python Crawlers

• requests: HTTP request library
• BeautifulSoup4: HTML parsing
• Scrapy: Professional crawler framework
• Selenium: Browser automation
• Playwright: Modern browser automation

JavaScript Crawlers

• axios: HTTP client
• cheerio: Server-side jQuery
• puppeteer: Chrome automation
• node-fetch：Fetch API

Workflow

1. Target Analysis：

   • Inspect website structure and data locations
   • Analyze API interfaces and authentication methods
   • Assess anti-scraping mechanisms

2. Solution Design：

   • Choose an appropriate technology stack
   • Design a data extraction strategy
   • Plan error handling and retry mechanisms

3. Script Development：

   • Write crawler code
   • Implement data parsing logic
   • Add logging and monitoring

4. Testing and Optimization：

   • Verify data accuracy
   • Optimize performance and stability
   • Handle edge cases

Best Practices

• Comply with robots.txt rules
• Set reasonable request intervals
• Use User-Agent and request headers
• Implement error retry mechanisms
• Data deduplication and validation
• Use proxy pools (if needed)
• Save raw data and logs

Common Scenario Examples

1. Simple Web Page Scraping

import requests
from bs4 import BeautifulSoup

def scrape_website(url):
    headers = {'User-Agent': 'Mozilla/5.0'}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')

    # 提取数据
    data = []
    for item in soup.select('.product'):
        data.append({
            'title': item.select_one('.title').text,
            'price': item.select_one('.price').text
        })
    return data

2. API Calls

import requests

def call_api(endpoint, params=None):
    headers = {
        'Authorization': 'Bearer YOUR_TOKEN',
        'Content-Type': 'application/json'
    }
    response = requests.get(endpoint, headers=headers, params=params)
    return response.json()

3. Dynamic Web Page Scraping

from selenium import webdriver
from selenium.webdriver.common.by import By

def scrape_dynamic_page(url):
    driver = webdriver.Chrome()
    driver.get(url)

    # 等待页面加载
    driver.implicitly_wait(10)

    # 提取数据
    elements = driver.find_elements(By.CLASS_NAME, 'item')
    data = [elem.text for elem in elements]

    driver.quit()
    return data

Anti-Scraping Response Strategies

• Request header disguise: Simulate a real browser
• Proxy rotation: Use proxy pools
• CAPTCHA handling: OCR or third-party services
• Cookie management: Maintain session state
• Request rate control: Avoid triggering restrictions
• JavaScript rendering: Use Selenium/Playwright

Data Storage Solutions

• CSV/Excel: Simple data export
• JSON: Structured data storage
• Database：MySQL、PostgreSQL、MongoDB
• Cloud storage：S3、OSS
• Data warehouse: for large-scale data analysis