data-engineering-storage-lakehouse

Lakehouse Formats

Lakehouse formats add ACID transactions, schema evolution, and time travel to data lakes stored on object storage (S3, GCS, Azure). This skill covers the three major open table formats: Delta Lake, Apache Iceberg, and Apache Hudi.

Quick Comparison

Feature	Delta Lake	Apache Iceberg	Apache Hudi
ACID Transactions	✅	✅	✅
Time Travel	✅	✅	✅
Schema Evolution	✅	Advanced (branching)	✅
Primary Ecosystem	Spark/Databricks	Engine-agnostic	Spark (CDC focus)
Write Optimization	Copy-on-write	CoW, Merge-on-Read	CoW, Merge-on-Read
Python API	`deltalake` (pure), PySpark	`pyiceberg` (pure)	PySpark only
Best For	Spark ecosystems, Databricks	Multi-engine analytics	Change data capture, streaming

When to Use Which?

Delta Lake: You're in the Spark/Databricks ecosystem, need mature tooling with pure-Python deltalake library
Apache Iceberg: You need engine-agnostic tables (Spark, Trino, Flink, DuckDB), advanced schema branching
Apache Hudi: You're building CDC pipelines from Kafka/DB logs, need upsert/delete support

Interoperability

Apache XTable and Delta UniForm (2024+) enable cross-format reads without conversion. Platforms like Databricks Unity Catalog and Snowflake support multiple formats natively, reducing vendor lock-in.

Related Skills

@data-engineering-storage-remote-access/integrations/delta-lake - Delta Lake on S3/GCS/Azure
@data-engineering-storage-remote-access/integrations/iceberg - Iceberg with cloud catalogs
@data-engineering-orchestration/dbt - dbt adapters for Delta/Iceberg
@data-engineering-storage-remote-access - fsspec, PyArrow filesystem for cloud access

Skill Dependencies

This skill assumes familiarity with:

@data-engineering-core - Polars, DuckDB, PyArrow basics
@data-engineering-storage-remote-access - Cloud storage access patterns

Detailed Guides

Delta Lake

See: @data-engineering-storage-lakehouse/delta-lake.md

Pure-Python API (deltalake package)
PySpark integration
Time travel (version AsOf, timestamp AsOf)
Schema evolution (add/drop/rename/upcast)
Vacuum and optimize
S3/GCS/Azure storage integration

Apache Iceberg

See: @data-engineering-storage-lakehouse/iceberg.md

PyIceberg catalog abstraction (Hive, AWS Glue, REST)
Schema evolution with branch support
Partition evolution
Time travel and versioning
Local catalog for development

Apache Hudi

See: @data-engineering-storage-lakehouse/hudi.md

Copy-on-write and Merge-on-Read modes
CDC integration (Debezium, Kafka)
Hoodie tables, indexes, bloom filters
Querying via Spark

Common Patterns

Time Travel Queries

# Delta Lake
from deltalake import DeltaTable
dt = DeltaTable("s3://bucket/delta-table")
dt.load_version(5)  # Load specific version
df = dt.to_pandas()

# Iceberg
table = catalog.load_table("db.table")
df = table.scan(as_of_timestamp="2024-01-01T00:00:00Z").to_pandas()

Schema Evolution

# Delta Lake (auto-evolves on write by default)
dt = DeltaTable("s3://bucket/delta-table")
# When writing with new column, Delta adds it automatically

# Iceberg (explicit)
with table.update_schema() as update:
    update.add_column("new_field", StringType(), required=False)

Incremental Processing

# Read only changes since last checkpoint
delta_table = DeltaTable("s3://bucket/delta-table")
last_version = get_last_processed_version()

# Get changes as Arrow table
changes = (
    delta_table
    .history()  # Get commit history
    .filter(f"version > {last_version}")
    .to_pyarrow_table()
)

Best Practices

Use Partitions: Partition by date/region to enable predicate pushdown
Vacuum Regularly: Clean up old files to avoid storage bloat (Delta: vacuum(), Iceberg: expire_snapshots())
Optimize Layouts: Compaction for small files (Delta: OPTIMIZE, Iceberg: rewrite_data_files())
Catalog Choice: Use AWS Glue for AWS, Hive Metastore for on-prem, REST for SaaS
Transaction Size: Batch writes for throughput but avoid too-large transactions
Monitor Table Metadata: Table metadata grows with operations; archive old versions

About this skill