Flow authoring: This skill covers engine infrastructure and operations only. For building or modifying StreamSets flows, use your path's flow authoring skill.
When to Use This Skill
- Configuring or troubleshooting StreamSets environments and engines
- Engine deployment (Docker/Podman), communication methods (tunneling vs. direct)
- Job lifecycle: execution, monitoring, failover, offset management
- High availability setup and resource management
Engine Characteristics
- Container-based: Runs as Docker/Podman containers in your network
- Data ownership: All processing occurs within your infrastructure
- Continuous processing: Jobs run continuously, processing data as it arrives
- High availability: Multiple engines support failover and load distribution
- Flexible communication: Tunneling (default) or direct connection methods
Key Concepts
- Environments: Configure engines and compute resources for projects
- Engines: Data Collector containers that execute flows
- Jobs: Flow executions that run continuously on engines
- Offsets: Track processing progress for resumable execution
- VPCs: Virtual CPUs allocated to engine containers
Architecture
- Browser ↔ watsonx.data integration ↔ Engine (in your network)
- Engines process data locally, send metrics to watsonx.data integration
- Jobs fail over automatically between engines (6.4+)
- Resource thresholds prevent engine overload
Quick Start
- Create environment → environments.md
- Run engine → engines.md
- Execute job → jobs.md
References
- Environments - Environment creation and configuration
- Engines - Engine deployment, management, and monitoring
- Jobs - Job execution and lifecycle management
- Technical Reference - System requirements, commands, and troubleshooting