expanso-kafka-to-s3

Kafka to S3

Stream data from Kafka topics to S3 buckets with intelligent partitioning, batching, and compression.

Category

data-routing

Quick Start

# Configure environment
export KAFKA_BROKERS=localhost:9092
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export S3_BUCKET=your-bucket

Run the pipeline

./run.sh

## Pipeline

The `pipeline.yaml` streams from Kafka to S3 with:
- Consumer group management
- Time-based partitioning (hourly/daily)
- Gzip compression
- Batching for efficient S3 writes

## Requirements

- Expanso Edge installed (`clawhub install expanso`)
- Kafka broker access
- AWS credentials with S3 write permissions

## Related

- [Expanso Documentation](https://docs.expanso.io)
- [More Examples](https://examples.expanso.io)