web-scraping-automation

Website Scraping and API Automation

Feature Description

This skill is specifically for automating website data scraping and API interface calls, including:

• Analyze and scrape website structures
• Call and test REST/GraphQL APIs
• Create automated crawler scripts
• Data parsing and cleaning
• Handle anti-scraping mechanisms
• Scheduled tasks and data storage

Use Cases

• "Scrape product information from this website"
• "Help me call this API and parse the returned data"
• "Create a script to scrape news on a schedule"
• "Analyze this website's API documentation"
• "Bypass this website's anti-scraping restrictions"

Tech Stack

Python Crawlers

• requests: HTTP request library
• BeautifulSoup4: HTML parsing
• Scrapy: Professional crawler framework
• Selenium: Browser automation
• Playwright: Modern browser automation

JavaScript Crawlers

• axios: HTTP client
• cheerio: Server-side jQuery
• puppeteer: Chrome automation
• node-fetch：Fetch API

Workflow

1. Target Analysis：

   • Inspect website structure and data locations
   • Analyze API interfaces and authentication methods
   • Assess anti-scraping mechanisms

2. Solution Design：

   • Choose an appropriate tech stack
   • Design data extraction strategy
   • Plan error handling and retry mechanisms

3. Script Development：

   • Write crawler code
   • Implement data parsing logic
   • Add logging and monitoring

4. Testing and Optimization：

   • Verify data accuracy
   • Optimize performance and stability
   • Handle edge cases

Best Practices

• Follow robots.txt rules
• Set reasonable request intervals
• Use User-Agent and request headers
• Implement error retry mechanisms
• Data deduplication and validation
• Use proxy pools (if needed)
• Save raw data and logs

Common Scenario Examples

1. Simple Web Scraping

import requests
from bs4 import BeautifulSoup

def scrape_website(url):
    headers = {'User-Agent': 'Mozilla/5.0'}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')

    # 提取数据
    data = []
    for item in soup.select('.product'):
        data.append({
            'title': item.select_one('.title').text,
            'price': item.select_one('.price').text
        })
    return data

2. API Calls

import requests

def call_api(endpoint, params=None):
    headers = {
        'Authorization': 'Bearer YOUR_TOKEN',
        'Content-Type': 'application/json'
    }
    response = requests.get(endpoint, headers=headers, params=params)
    return response.json()

3. Dynamic Web Scraping

from selenium import webdriver
from selenium.webdriver.common.by import By

def scrape_dynamic_page(url):
    driver = webdriver.Chrome()
    driver.get(url)

    # 等待页面加载
    driver.implicitly_wait(10)

    # 提取数据
    elements = driver.find_elements(By.CLASS_NAME, 'item')
    data = [elem.text for elem in elements]

    driver.quit()
    return data

Anti-Scraping Countermeasures

• Request header spoofing: Simulate a real browser
• Proxy rotation: Use proxy pools
• CAPTCHA handling: OCR or third-party services
• Cookie management: Maintain session state
• Request rate control: Avoid triggering limits
• JavaScript rendering: Use Selenium/Playwright

Data Storage Options

• CSV/Excel: Simple data export
• JSON: Structured data storage
• Database：MySQL、PostgreSQL、MongoDB
• Cloud storage：S3、OSS
• Data warehouse: For large-scale data analysis