e2e-testing

E2E Testing

Set up comprehensive end-to-end testing infrastructure for Assistant Skills plugins. Auto-generates test cases from your plugin structure and skills.

Quick Start

# Initialize E2E testing infrastructure
python skills/e2e-testing/scripts/setup_e2e.py /path/to/project

# Auto-generate test cases from plugin structure
python skills/e2e-testing/scripts/generate_test_cases.py /path/to/project

# Run tests
./scripts/run-e2e-tests.sh

# Update project documentation
python skills/e2e-testing/scripts/update_docs.py /path/to/project

Scripts Reference

Script	Purpose
setup_e2e.py	Initialize E2E testing infrastructure
generate_test_cases.py	Auto-generate test cases from plugin
run_tests.py	Execute tests with multiple output formats
update_docs.py	Update project documentation

Authentication

Method	Location	Setup
OAuth	~/.claude.json	claude auth login
API Key	Environment	export ANTHROPIC_API_KEY="sk-ant-..."

Local runs (--local) prefer OAuth credentials over API keys.

Configuration

Variable	Default	Description
E2E_TEST_TIMEOUT	120	Timeout per test (seconds)
E2E_TEST_MODEL	claude-sonnet-4-20250514	Claude model
E2E_MAX_TURNS	5	Max conversation turns

Output Formats

Format	Flag	Description
Console	(default)	Colored terminal output
JSON	--output results.json	Machine-readable results
JUnit	--junit results.xml	CI/CD integration
HTML	--html report.html	Visual report

Cost Estimates

Model	Cost per Test	20 Tests
Haiku	~$0.001	~$0.02
Sonnet	~$0.01	~$0.20
Opus	~$0.05	~$1.00

Customizing Tests

Edit tests/e2e/test_cases.yaml to add custom test cases:

suites:
  custom:
    tests:
      - id: my_test
        prompt: "Do something specific"
        timeout: 180      # Override default
        max_turns: 10     # Override default
        expect:
          output_contains: ["expected"]

Response Logging

Failed test responses logged to test-results/e2e/responses_latest.log.

More Information

• Detailed guide: See tests/e2e/README.md
• Prompt patterns: See docs/test-prompt-best-practices.md
• Troubleshooting: See docs/troubleshooting.md