Description
Enables the agent to model, implement, and orchestrate the Write-Audit-Publish (WAP) pattern natively using Apache Iceberg branches.
Context
In an Enterprise data lakehouse, writing directly to production tables (main branch) during ETL processes poses a massive risk of data corruption. Apache Iceberg solves this natively with Git-like branching. The agent must implement WAP patterns over traditional staging tables.
Instructions
1. WAP Pattern Flow
Instead of creating _stg tables, instruct the user to use Iceberg branches.
Step 1: Create a Branch
Create a branch pointing to the current state of main.
ALTER TABLE prod.sales.transactions CREATE BRANCH etl_job_123;
Step 2: Configure Spark to Write to Branch Spark must be told to write to the specific branch.
# PySpark Example
spark.conf.set("spark.wap.branch", "etl_job_123")
# Now any INSERT or MERGE INTO statement goes to the branch
df.writeTo("prod.sales.transactions").append()
Step 3: Audit / Data Quality Check Read from the branch to verify row counts, nulls, and constraints.
SELECT count(*) FROM prod.sales.transactions VERSION AS OF 'etl_job_123';
-- Trigger Soda or Great Expectations on this branch
Step 4: Publish (Fast-Forward) Once the audit passes, publish the branch back to main.
CALL catalog.system.fast_forward('prod.sales.transactions', 'main', 'etl_job_123');
Step 5: Cleanup
ALTER TABLE prod.sales.transactions DROP BRANCH etl_job_123;
2. Hidden Partitioning
Iceberg uses Hidden Partitioning. If a user tries to partition by year or month by extracting it manually:
- ❌ Bad:
df.withColumn("year", year("ts")).write.partitionBy("year") - ✅ Good:
CREATE TABLE ... PARTITIONED BY (years(ts))
Explain to the user that Iceberg handles the partition extraction automatically without needing explicit columns.
Output Format: Iceberg WAP Implementation Guide
### 🛡️ Write-Audit-Publish (WAP) Plan: `[Table Name]`
#### 1. Branch Creation
Run: `ALTER TABLE ... CREATE BRANCH ...`
#### 2. Spark Write Configuration
Ensure your SparkSession has: `spark.wap.branch=[branch_name]`
#### 3. Audit Queries
- Run: `SELECT ... VERSION AS OF '[branch_name]'`
- Expected: 0 Nulls.
#### 4. Publish Command
Run: `CALL catalog.system.fast_forward(...)`