SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsagent-device-mobile-automation

agent-device-mobile-automation

Testing & quality

View repository

Updated 10-16 07:26

About this skill

agent-device Mobile Automation

Skill by ara.so — AI Agent Skills collection.

agent-device is a CLI tool designed for AI agents to control and verify mobile apps on iOS, Android, TV, web, and desktop platforms. It provides token-efficient snapshots, semantic element references, and evidence capture for AI-driven testing workflows.

What agent-device Does

  • Inspect app UI through structured accessibility snapshots with interactive refs (@e1, @e2, etc.)
  • Interact with apps: open, tap, type, scroll, gestures, wait, assert, handle alerts
  • Capture evidence: screenshots, videos, logs, traces, network traffic, performance samples
  • Replay workflows by recording .ad scripts for CI and repeatable e2e checks
  • Cross-platform support for iOS Simulator, Android Emulator, physical devices, tvOS, macOS, Linux

Installation

Global CLI Installation

npm install -g agent-device@latest

Verify Installation

agent-device doctor
agent-device --version
agent-device help workflow

Prerequisites by Platform

  • All platforms: Node.js 22+
  • iOS/tvOS/macOS: Xcode with command-line tools, macOS Accessibility permissions
  • Android: Android SDK, ADB configured
  • Web: Node.js 24+
  • Linux desktop: AT-SPI accessibility support

Check setup:

agent-device doctor --platform ios
agent-device doctor --platform android

Key Commands Reference

Discovery & Session Management

# List available apps
agent-device apps --platform ios
agent-device apps --platform android

# Start session with an app
agent-device open "AppName" --platform ios
agent-device open com.example.app --platform android

# Close current session
agent-device close

Inspection

# Full accessibility snapshot
agent-device snapshot

# Interactive elements only (recommended for agents)
agent-device snapshot -i

# With component tree (React Native apps)
agent-device snapshot --components

# With selectors visible
agent-device snapshot --selectors

Snapshot Output Example:

@e1 [heading] "Settings"
@e2 [button] "Sign In"
@e3 [text-field] "Email"
@e4 [button] "Submit"

Interaction Commands

# Tap element by ref
agent-device tap @e2

# Tap by selector
agent-device tap 'button[label="Sign In"]'

# Fill text field
agent-device fill @e3 "user@example.com"

# Type text (supports keyboard input)
agent-device type "Hello World"

# Scroll
agent-device scroll down
agent-device scroll up --amount 0.5

# Swipe gesture
agent-device swipe left
agent-device swipe right --from @e5

# Wait for element
agent-device wait @e6 --timeout 5000

# Wait for time
agent-device wait 2000

# Assert element exists
agent-device assert @e7 --exists

# Assert element has text
agent-device assert @e8 --text "Welcome"

# Handle alerts
agent-device alert accept
agent-device alert dismiss

Evidence Capture

# Take screenshot
agent-device screenshot ./artifacts/screen.png

# Record video (start/stop)
agent-device video start
# ... perform actions ...
agent-device video stop ./artifacts/recording.mp4

# Capture logs
agent-device logs ./artifacts/app.log

# Capture network traffic
agent-device network start
# ... perform actions ...
agent-device network stop ./artifacts/network.har

# Performance profile
agent-device profile start
# ... perform actions ...
agent-device profile stop ./artifacts/perf.json

# React Native profile
agent-device react-profile start
# ... interact with app ...
agent-device react-profile stop ./artifacts/react-profile.json

Replay & Recording

# Record commands to .ad script
agent-device record start ./scripts/login-flow.ad

# Execute commands...
agent-device open MyApp --platform ios
agent-device tap @e1
agent-device fill @e2 "test@example.com"

# Stop recording
agent-device record stop

# Replay script
agent-device replay ./scripts/login-flow.ad

# Export to Maestro YAML
agent-device export maestro ./scripts/login-flow.ad --output ./maestro/login.yaml

Configuration

Session Options

# Specify device
agent-device open MyApp --platform ios --device "iPhone 15 Pro"

# Launch with arguments
agent-device open MyApp --platform ios --args "--env=staging"

# Use specific bundle ID
agent-device open com.company.myapp --platform android

Global Flags

# Verbose output for debugging
agent-device snapshot -i --verbose

# JSON output for programmatic use
agent-device snapshot --format json

# Specify timeout
agent-device wait @e1 --timeout 10000

Environment Variables

# Platform preference
export AGENT_DEVICE_PLATFORM=ios

# Device preference
export AGENT_DEVICE_DEVICE="iPhone 15 Pro"

# Artifacts directory
export AGENT_DEVICE_ARTIFACTS=./test-artifacts

Real Code Examples

Example 1: Login Flow Verification

#!/bin/bash
# Verify login flow on iOS

# Start session
agent-device open "MyApp" --platform ios

# Take initial screenshot
agent-device screenshot ./artifacts/01-launch.png

# Get interactive elements
agent-device snapshot -i

# Assuming snapshot shows:
# @e1 [text-field] "Email"
# @e2 [text-field] "Password"
# @e3 [button] "Log In"

# Fill credentials
agent-device fill @e1 "$TEST_EMAIL"
agent-device fill @e2 "$TEST_PASSWORD"

# Screenshot before submit
agent-device screenshot ./artifacts/02-filled.png

# Submit
agent-device tap @e3

# Wait for home screen
agent-device wait 'heading[label="Welcome"]' --timeout 5000

# Verify success
agent-device snapshot -i
agent-device screenshot ./artifacts/03-logged-in.png

# Close session
agent-device close

Example 2: Android E2E Test with Evidence

#!/bin/bash
# Android checkout flow with full evidence capture

agent-device open com.example.shop --platform android

# Start evidence capture
agent-device video start
agent-device network start
agent-device logs start ./artifacts/app.log

# Navigate to product
agent-device snapshot -i
agent-device tap 'button[text="Shop"]'
agent-device wait 2000

agent-device snapshot -i
agent-device tap '@e5' # Assuming @e5 is a product

# Add to cart
agent-device wait 'button[text="Add to Cart"]'
agent-device tap 'button[text="Add to Cart"]'
agent-device screenshot ./artifacts/added-to-cart.png

# Go to cart
agent-device tap 'button[content-desc="Cart"]'
agent-device wait 2000
agent-device snapshot -i

# Checkout
agent-device tap 'button[text="Checkout"]'
agent-device wait 3000

# Stop evidence capture
agent-device video stop ./artifacts/checkout-flow.mp4
agent-device network stop ./artifacts/network.har

agent-device close

Example 3: React Native Component Inspection

// From Node.js/TypeScript context
import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);

async function inspectReactNativeApp() {
  // Open app
  await execAsync('agent-device open MyRNApp --platform ios');
  
  // Get component tree
  const { stdout } = await execAsync('agent-device snapshot --components --format json');
  const snapshot = JSON.parse(stdout);
  
  // Find specific component
  const loginButton = snapshot.components.find(
    (c: any) => c.type === 'Button' && c.props?.title === 'Log In'
  );
  
  if (loginButton) {
    console.log('Found login button:', loginButton.ref);
    await execAsync(`agent-device tap ${loginButton.ref}`);
  }
  
  // Take screenshot
  await execAsync('agent-device screenshot ./artifacts/after-tap.png');
  
  // Close
  await execAsync('agent-device close');
}

Example 4: Replay Script (.ad format)

Create login-test.ad:

# Login flow replay script
open MyApp --platform ios
wait 2000
snapshot -i
fill @e1 "test@example.com"
fill @e2 "password123"
screenshot ./artifacts/login-filled.png
tap @e3
wait heading[label="Dashboard"] --timeout 5000
assert heading[label="Dashboard"] --exists
screenshot ./artifacts/dashboard.png
close

Execute:

agent-device replay ./login-test.ad

Example 5: Performance Profiling

#!/bin/bash
# Profile app launch and navigation

agent-device open MyApp --platform ios

# Start performance profiling
agent-device profile start

# Navigate through screens
agent-device tap @e1
agent-device wait 1000
agent-device scroll down
agent-device wait 1000
agent-device tap @e5
agent-device wait 2000

# Stop and save profile
agent-device profile stop ./artifacts/navigation-perf.json

# React Native specific profiling
agent-device react-profile start
agent-device tap 'button[label="Load Data"]'
agent-device wait 3000
agent-device react-profile stop ./artifacts/react-render.json

agent-device close

Common Patterns

Pattern 1: Conditional Flow Based on Snapshot

# Take snapshot and parse
SNAPSHOT=$(agent-device snapshot -i)

if echo "$SNAPSHOT" | grep -q "Sign In"; then
  echo "User not logged in, performing login..."
  agent-device tap 'button[label="Sign In"]'
  agent-device fill @e1 "$USER_EMAIL"
  agent-device fill @e2 "$USER_PASSWORD"
  agent-device tap 'button[label="Submit"]'
  agent-device wait 3000
else
  echo "User already logged in"
fi

Pattern 2: Retry with Evidence

retry_with_screenshot() {
  local max_attempts=3
  local attempt=1
  
  while [ $attempt -le $max_attempts ]; do
    echo "Attempt $attempt..."
    
    if agent-device tap "$1" 2>/dev/null; then
      echo "Success"
      return 0
    fi
    
    agent-device screenshot "./artifacts/retry-$attempt.png"
    agent-device snapshot -i > "./artifacts/snapshot-$attempt.txt"
    
    attempt=$((attempt + 1))
    sleep 2
  done
  
  return 1
}

retry_with_screenshot '@e5'

Pattern 3: Multi-Platform Test

#!/bin/bash
test_on_platform() {
  local platform=$1
  
  echo "Testing on $platform..."
  agent-device open MyApp --platform "$platform"
  
  agent-device snapshot -i
  agent-device tap @e1
  agent-device wait 2000
  agent-device screenshot "./artifacts/${platform}-result.png"
  
  agent-device close
}

test_on_platform ios
test_on_platform android

Pattern 4: Selector-Based Interaction (More Reliable Than Refs)

# Refs (@e1, @e2) are session-specific and change after scrolling
# Use selectors for stable references

# By label/text
agent-device tap 'button[label="Continue"]'
agent-device fill 'text-field[label="Username"]' "myuser"

# By test ID (best practice)
agent-device tap '[testID="submit-button"]'
agent-device assert '[testID="success-message"]' --exists

# By role + text
agent-device tap 'button[text="Sign Up"]'

# Complex selector
agent-device tap 'button[label="Submit"][enabled="true"]'

Troubleshooting

Issue: "No devices found"

# Check available devices
xcrun simctl list devices # iOS
adb devices # Android

# Boot simulator if needed
xcrun simctl boot "iPhone 15 Pro"

# Check agent-device can see devices
agent-device apps --platform ios

Issue: "Element not found"

# Always take fresh snapshot after screen changes
agent-device tap @e1
agent-device wait 2000
agent-device snapshot -i  # Get new refs

# Use selectors instead of refs for stability
agent-device tap 'button[label="Next"]'

# Increase wait time
agent-device wait @e5 --timeout 10000

Issue: "Session timeout"

# Increase timeout globally
export AGENT_DEVICE_TIMEOUT=30000

# Or per command
agent-device wait @e1 --timeout 30000

Issue: Accessibility data missing

# Ensure accessibility is enabled on iOS Simulator
# System Settings > Accessibility > VoiceOver

# For Android, ensure TalkBack is available (doesn't need to be on)

# Add accessibility labels in your app code:
# iOS: accessibilityLabel, accessibilityIdentifier
# Android: contentDescription, testTag
# React Native: accessibilityLabel, testID

Issue: Commands not being recorded

# Verify recording is active
agent-device record start ./test.ad

# Check file is being written
ls -lh ./test.ad

# Ensure commands are session commands (not all flags are recorded)
# Recordable: open, tap, fill, wait, snapshot, screenshot
# Not recorded: record, replay, export, doctor

Debugging Tips

# Use verbose mode
agent-device snapshot -i --verbose

# Capture full logs
agent-device logs ./debug.log

# Take frequent screenshots to understand state
agent-device screenshot ./debug-$(date +%s).png

# Use JSON output for parsing
agent-device snapshot --format json | jq '.elements[] | select(.role=="button")'

CI/CD Integration

GitHub Actions Example

name: Mobile QA
on: [pull_request]

jobs:
  test-ios:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '22'
      
      - name: Install agent-device
        run: npm install -g agent-device@latest
      
      - name: Run iOS tests
        run: |
          agent-device doctor --platform ios
          ./scripts/test-ios.sh
      
      - name: Upload artifacts
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-artifacts
          path: ./artifacts/

EAS Workflow (Expo)

# .eas/workflows/agent-qa-mobile.yml
build:
  name: Agent QA
  steps:
    - run:
        name: Install agent-device
        command: npm install -g agent-device@latest
    
    - run:
        name: Run tests
        command: |
          agent-device open MyExpoApp --platform ios
          agent-device replay ./e2e/smoke-test.ad

Additional Resources

Best Practices for AI Agents

  1. Always run agent-device snapshot -i after screen changes to get fresh element refs
  2. Prefer selectors over refs for stable automation: 'button[label="Submit"]' vs @e5
  3. Use testID in React Native for reliable element targeting
  4. Capture evidence before assertions to help debug failures
  5. Use --verbose flag when troubleshooting unclear behavior
  6. Start recordings at workflow beginning to capture full context
  7. Check agent-device doctor before starting automation on new environments
  8. Use agent-device apps to verify app is installed and discoverable
  9. Set reasonable timeouts (default 5s may be too short for slow screens)
  10. Close sessions explicitly with agent-device close to free resources