browser-xterm-interaction

Playwright Browser MCP × Web Terminal Interaction

Web terminals (xterm.js, hterm, etc.) render the terminal interface in the browser, but their output is usually rendered with canvas or custom DOM, and the standard browser_snapshot() can only see the last line of the prompt in the accessibility tree. This is the biggest pain point when interacting with Web terminals—you need specific JS methods to reliably read terminal output.

Core Principles

1. Set it once — Once you find an effective reading method, stick with it; don't switch methods every time, because switching back and forth wastes a large number of turns
2. Rule of 3 — Try a method at most 3 times before switching to the next one, to avoid consuming turns on dead ends
3. screenshot fallback — When all JS methods fail, screenshot + visual analysis is the 100% reliable final resort
4. marker wrapping — Use echo "===START==="; cmd; echo "===END===" to wrap command output for precise extraction

---

Phase 1: Identify terminal type

First use one browser_snapshot() or browser_evaluate to probe the terminal type:

// 探测终端类型
() => {
  const xterm = document.querySelector('.xterm');
  const jquery_term = window.jQuery && window.jQuery.fn.terminal;
  const hterm = document.querySelector('[id*="hterm"]');
  return {
    xterm: !!xterm,
    jquery_terminal: !!jquery_term,
    hterm: !!hterm,
    terminal_classes: xterm ? xterm.className : 'not found'
  };
}

---

Phase 2: Command Execution (Input)

Method A: Input via textbox ref (recommended)

browser_type(ref='<terminal_input_ref>', text='your command', submit=True)

Method B: Input directly via keyboard

browser_click(element='terminal area', ref='<terminal_ref>')
browser_press_key(key='your command text')  // 逐字符
browser_press_key(key='Enter')

Method C: Write via evaluate

() => {
  const textarea = document.querySelector('.xterm-helper-textarea');
  if (textarea) {
    textarea.focus();
    // 通过 InputEvent 模拟输入
  }
}

⚠️ Important: After executing a command, you must wait long enough for the output to complete:

• Fast commands (echo, cat): browser_wait_for(time=2)
• Medium-speed commands (nslookup, curl): browser_wait_for(time=5)
• Slow commands (scanning, compiling): browser_wait_for(time=15-30)

---

Phase 2.5: Output Completeness Verification

After reading terminal output,you must check whether the output is complete：

Signals of incompleteness

• The config file shows only the last 1-2 lines (e.g. resolv.conf only has the options line, missing nameserver/search)
• The output only has a prompt, with no command results at all
• The output looks like a middle fragment (no starting line)

Fix method

When you find the output may be incomplete,immediately retry with the redirection method：

command > /tmp/verify.txt 2>&1 && cat /tmp/verify.txt

Do not continue making decisions based on incomplete output.

---

Phase 3: Output Reading (Core Difficulty)

⭐ Method 1: xterm-rows innerText (most reliable)

() => {
  const rows = document.querySelectorAll('.xterm-rows > div');
  let lines = [];
  for (let row of rows) {
    const text = row.innerText || row.textContent || '';
    if (text.trim()) lines.push(text);
  }
  return lines.join('\n');
}

Method 2: xterm-screen innerText

() => {
  const screen = document.querySelector('.xterm-screen');
  return screen ? screen.innerText : 'not found';
}

Method 3: Via Terminal API (if exposed)

() => {
  // 尝试通过 xterm.js Terminal 实例的 buffer 读取
  const term = document.querySelector('.xterm');
  if (term && term._core) {
    const buffer = term._core.buffer.active;
    let lines = [];
    for (let i = 0; i < buffer.length; i++) {
      const line = buffer.getLine(i);
      if (line) lines.push(line.translateToString(true));
    }
    return lines.join('\n');
  }
  return 'Terminal API not accessible';
}

Method 4: Marker wrapping method (precise command output extraction)

When executing commands, wrap them with markers for precise extraction:

echo "===START==="; your_command_here; echo "===END==="

Then extract in JS the content between ===START=== and ===END=== .

Method 5: Redirect to file + cat

your_command > /tmp/out.txt 2>&1; cat /tmp/out.txt

Suitable for cases where output is very long or content is lost due to scrolling.

---

Phase 4: Fallback Strategy (when JS methods fail)

⭐ Screenshot + Visual Analysis (final fallback)

When JS methods cannot reliably read terminal output:

browser_take_screenshot(type='png', filename='term_output.png')
Read(file_path='<output_dir>/term_output.png')

Then visually analyze the terminal output through the image content.

⚠️ Do not use fullPage=True — Full-page screenshots are extremely large (>1MB), which causes the SDK JSON buffer to overflow and crash. Only use the default viewport screenshot.

Pros: 100% reliable, does not depend on DOM structure Cons: consumes more tokens, can only see the visible area

Tips：

• Before taking a screenshot, scroll the terminal to the bottom first
• Take multiple screenshots for long output
• Use fullPage=True try a full-page screenshot
• Use the marker method to locate key output areas

---

Common issues and solutions

Q: browser_snapshot() only returns the last line prompt

xterm.js renders with a canvas or DOM renderer, and snapshot can only see the accessibility tree. Solution: use browser_evaluate + the JS method from Phase 3.

Q: browser_evaluate returns an empty string

The terminal may be using a canvas renderer, and there are no text nodes in the DOM. Solution: use screenshot as a fallback.

Q: Terminal output is too long and gets truncated

xterm.js has a scrollback buffer limit (1000 lines by default). Solution: use the redirect-to-file method, or view it in segments.

Q: No output change is visible after running a command

The wait time may not be enough, or the terminal did not refresh. Solution: increase browser_wait_for time, or press Enter to trigger a refresh.

---

Decision flowchart

开始
 ↓
识别终端类型 (Phase 1)
 ↓
执行命令 (Phase 2, 方法 A 优先)
 ↓
等待输出 (browser_wait_for)
 ↓
尝试 JS 读取 (Phase 3, 方法 1)
 ├── 成功 → 锁定此方法，后续复用
 ├── 失败 → 尝试方法 2
 │    ├── 成功 → 锁定
 │    ├── 失败 → 尝试方法 3
 │    │    ├── 成功 → 锁定
 │    │    └── 失败 → Screenshot 兜底 (Phase 4)
 └── 3 次后仍不稳定 → Screenshot 兜底

⚠️ Prohibited: Switching back and forth between JS reading methods more than 3 times. Once you decide on a method, stick with it.

---

CTF / Cloud Shell section

Best practices for writing multi-line scripts

The input box of a Web terminal is usuallysingle-line, and directly pasting multi-line Python/Bash scripts will cause formatting errors.

Method A: heredoc to write a file (recommended)

// 逐行写入脚本文件
browser_type(ref, "cat > /tmp/exploit.py << 'PYEOF'", submit=True)
browser_wait_for(time=1)
browser_type(ref, "import boto3, json", submit=True)
browser_type(ref, "s3 = boto3.client('s3')", submit=True)
browser_type(ref, "print(s3.list_buckets())", submit=True)
browser_type(ref, "PYEOF", submit=True)
browser_wait_for(time=1)
browser_type(ref, "python3 /tmp/exploit.py", submit=True)

Method B: base64 encoding (most reliable)

// 先在本地构造脚本，base64 编码后一行写入
// 本地 bash:
echo 'import boto3; print(boto3.client("s3").list_buckets())' | base64
// → aW1wb3J0IGJvdG8z...

// 在 Web 终端:
browser_type(ref, "echo 'aW1wb3J0IGJvdG8z...' | base64 -d > /tmp/s.py && python3 /tmp/s.py", submit=True)

Method C: echo append (simple scripts)

browser_type(ref, "echo 'import boto3' > /tmp/s.py", submit=True)
browser_type(ref, "echo 's3=boto3.client(\"s3\")' >> /tmp/s.py", submit=True)
browser_type(ref, "echo 'print(s3.list_buckets())' >> /tmp/s.py", submit=True)
browser_type(ref, "python3 /tmp/s.py", submit=True)

⚠️ Prohibited: in browser_type 's text parameter, include the \n newline character — the Web terminal input box does not support multi-line input.

Environment variable setup

Web terminals may restrict writing ~/.aws/credentials（Permission denied）：

// 方法 A: export 在同一行（仅当前命令有效）
browser_type(ref, "AWS_ACCESS_KEY_ID=AKIAXXXX AWS_SECRET_ACCESS_KEY=YYYY aws s3 ls", submit=True)

// 方法 B: 在 Python 脚本内设置
browser_type(ref, "cat > /tmp/s.py << 'EOF'", submit=True)
browser_type(ref, "import os, boto3", submit=True)
browser_type(ref, "os.environ['AWS_ACCESS_KEY_ID']='AKIAXXXX'", submit=True)
browser_type(ref, "os.environ['AWS_SECRET_ACCESS_KEY']='YYYY'", submit=True)
browser_type(ref, "s3=boto3.client('s3',region_name='us-east-1')", submit=True)
browser_type(ref, "print(s3.list_buckets())", submit=True)
browser_type(ref, "EOF", submit=True)
browser_type(ref, "python3 /tmp/s.py", submit=True)

Capturing long output

The Web terminal output buffer is limited, and long output will lose the beginning part:

// 使用 marker 包裹 + tail 截取
browser_type(ref, "echo '===START==='; aws s3 ls 2>&1 | tail -50; echo '===END==='", submit=True)

// 输出写文件再分段读取
browser_type(ref, "python3 /tmp/exploit.py > /tmp/out.txt 2>&1", submit=True)
browser_type(ref, "head -20 /tmp/out.txt", submit=True)
browser_type(ref, "tail -20 /tmp/out.txt", submit=True)