playwright-automation

Playwright Browser Automation

Overview

Playwright is a powerful browser automation tool that can simulate real user operations and supports:

• Headless browser mode (runs in the background)
• Data collection and crawling
• Automatic form filling
• UI automated testing
• Screenshot and PDF generation

Why Playwright Is Needed

⚠️ Resource Cleanup Principle (Mandatory)

After all cron tasks involving the browser are completed, the Chrome/Chromium process must be automatically closed!

# 任务结束时必须执行
import subprocess

# 方式1: 显式关闭浏览器
await browser.close()
await context.close()

# 方式2: 强制清理残留进程（推荐在脚本结束时调用）
subprocess.run(['pkill', '-f', 'chrome'], capture_output=True)
subprocess.run(['pkill', '-f', 'chromium'], capture_output=True)

Reason: To avoid memory leaks and resource usage, and prevent Gateway CPU 100% overload

Difference from browser tool

Feature	browser tool	Playwright
Requires user participation	✅ Requires manually opening the browser	❌ Fully automatic
Suitable for scheduled tasks	❌	✅
Runs in the background	❌	✅
Debug-friendly	✅ Visual operation	⚠️ Requires logs
No installation required	✅ Already integrated	❌ Requires installation

Use Cases

Use Playwright:

• ✅ Scheduled monitoring (cron tasks)
• ✅ Large-scale data collection
• ✅ Unattended operation
• ✅ Production environment deployment

Use browser tool:

• ✅ Interactive debugging
• ✅ Operations requiring manual decision-making
• ✅ One-time tasks
• ✅ Bypassing complex CAPTCHAs

Quick Start

1. Install Playwright

# 安装 Python 包
pip install playwright

# 安装浏览器（Chromium）
playwright install chromium

# 验证安装
python3 -c "from playwright.sync_api import sync_playwright; print('✅ 安装成功')"

2. Basic Usage

Synchronous API (simple tasks)

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)  # headless=False 显示浏览器
    page = browser.new_page()
    page.goto('https://example.com')
    print(page.title())
    browser.close()

Asynchronous API (recommended, better performance)

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto('https://example.com')
        title = await page.title()
        print(title)
        await browser.close()

asyncio.run(main())

Common Features

1. Page Navigation

# 等待加载完成
await page.goto('https://example.com', wait_until='domcontentloaded')

# 等待选项：
# - 'load' - 页面完全加载
# - 'domcontentloaded' - DOM 加载完成（更快）
# - 'networkidle' - 网络空闲（最慢但最稳）

2. Element Location

# CSS 选择器
await page.click('button.submit')
await page.fill('input[name="username"]', 'myuser')

# XPath
await page.click('xpath=//button[@type="submit"]')

# 文本选择器
await page.click('text=登录')

# 组合选择器
await page.click('div.login-form >> text=登录')

3. Extract Data

# 获取文本
text = await page.text_content('h1.title')

# 获取属性
href = await page.get_attribute('a.link', 'href')

# 获取多个元素
items = await page.query_selector_all('div.item')
for item in items:
    text = await item.text_content()
    print(text)

# 执行 JavaScript
result = await page.evaluate('() => document.title')

# 获取整个 HTML
html = await page.content()

4. Form Operations

# 填写表单
await page.fill('input[name="username"]', 'myuser')
await page.fill('input[name="password"]', 'mypass')
await page.click('button[type="submit"]')

# 下拉选择
await page.select_option('select#country', 'China')

# 复选框
await page.check('input#agree')

# 上传文件
await page.set_input_files('input[type="file"]', 'path/to/file.pdf')

5. Waiting Strategies

# 等待元素出现
await page.wait_for_selector('div.result', timeout=5000)

# 等待导航
async with page.expect_navigation():
    await page.click('a.link')

# 等待特定条件
await page.wait_for_function('() => document.title.includes("加载完成")')

# 固定延迟
import asyncio
await asyncio.sleep(2)  # 等待 2 秒

6. Scrolling and Interaction

# 滚动到页面底部
await page.evaluate('window.scrollTo(0, document.body.scrollHeight)')

# 滚动到元素
await page.locator('div.footer').scroll_into_view_if_needed()

# 鼠标悬停
await page.hover('div.menu')

# 拖拽
await page.drag_and_drop('div.draggable', 'div.dropzone')

7. Screenshots and PDF

# 截图
await page.screenshot(path='screenshot.png')

# 全页截图
await page.screenshot(path='full.png', full_page=True)

# 元素截图
await page.locator('div.content').screenshot(path='element.png')

# 生成 PDF
await page.pdf(path='page.pdf', format='A4')

8. Handling Popups

# 接受 alert
async with page.expect_event('dialog') as dialog_info:
    await page.click('button')
dialog = await dialog_info.value
await dialog.accept()

# 输入 prompt
async with page.expect_event('dialog') as dialog_info:
    await page.click('button')
dialog = await dialog_info.value
await dialog.accept('my input')

Anti-Crawling Strategies

1. User-Agent Rotation

user_agents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36'
]

browser = await p.chromium.launch(
    user_agent=random.choice(user_agents)
)

2. Random Delays

import random
import asyncio

# 在操作之间添加随机延迟
await asyncio.sleep(random.uniform(2, 5))

3. Cookie Saving and Loading

# 第一次登录后保存 Cookie
context = await browser.new_context()
await page.goto('https://example.com/login')
# ... 登录操作 ...
await context.storage_state(path='cookies.json')

# 后续使用保存的 Cookie
context = await browser.new_context(storage_state='cookies.json')

4. Proxy Settings

browser = await p.chromium.launch(
    proxy={
        'server': 'http://proxy.example.com:8080',
        'username': 'user',
        'password': 'pass'
    }
)

5. Browser Context Isolation

# 创建独立的上下文（相当于无痕模式）
context = await browser.new_context(
    viewport={'width': 1920, 'height': 1080},
    user_agent='Custom UA',
    locale='zh-CN'
)
page = await context.new_page()

Debugging Tips

1. Display the Browser

# 开启有头模式，可以看到操作过程
browser = await p.chromium.launch(headless=False, slow_mo=1000)
# slow_mo=1000 会在每个操作间延迟 1 秒

2. Screenshot Debugging

# 在关键步骤截图
await page.goto('https://example.com')
await page.screenshot(path='step1.png')

await page.click('button')
await page.screenshot(path='step2.png')

3. View Logs

# 监听控制台消息
page.on('console', lambda msg: print(f'Console: {msg.text}'))

# 监听网络请求
page.on('request', lambda request: print(f'Request: {request.url}'))
page.on('response', lambda response: print(f'Response: {response.status}'))

4. Playwright Inspector

# 启动 Inspector 模式
PWDEBUG=1 python3 your_script.py

Common Questions

Q: How to handle CAPTCHAs?

A: Several options

1. Use CAPTCHA-solving platforms (Chaojiying, Ruokuai Dama)
2. Manual handling: pause and wait for user input
3. Fallback: use browser tool to let the user operate manually

# 方案 2: 手动处理
input("遇到验证码，请在浏览器中完成，然后按回车继续...")
await asyncio.sleep(2)  # 等待验证通过

Q: What to do if an element cannot be found?

A: Check the following points

1. Whether it is inside an iframe (needs switching)
2. Whether it is dynamically loaded (needs waiting)
3. Whether the selector is correct

# 切换到 iframe
frame = page.frame('iframe-id')
await frame.click('button')

# 等待动态加载
await page.wait_for_selector('div.dynamic-content')

Q: How to improve performance?

A:

1. Use the asynchronous API
2. Concurrently multiple pages
3. Reduce unnecessary waiting

# 并发多个页面
async def fetch(url):
    page = await browser.new_page()
    await page.goto(url)
    # ...

tasks = [fetch(url) for url in urls]
await asyncio.gather(*tasks)

Integration with OpenClaw

Usage in healthcare-monitor

# scraper_free.py
from playwright.async_api import async_playwright

async with async_playwright() as p:
    browser = await p.chromium.launch(headless=True)
    context = await browser.new_context(
        user_agent="Mozilla/5.0 ..."
    )
    page = await context.new_page()
    await page.goto(url)
    # ... 采集数据 ...
    await browser.close()

Working with the browser tool

1. Playwright → Daily automated monitoring
2. browser tool → Debugging and exception handling

Resources

• Official documentation: https://playwright.dev/python/
• API reference: https://playwright.dev/python/docs/api/class-playwright
• Example code: ~/clawd/skills/playwright-automation/examples/

Quick commands

# 安装
pip install playwright && playwright install chromium

# 运行脚本
python3 script.py

# 调试模式
PWDEBUG=1 python3 script.py

# 查看版本
playwright --version

---

Remember: Playwright makes your automation tasks completely unattended! 🚀