Overview
Use this skill when the storage platform needs structure before pipelines scale into chaos. It helps agents design clear lake zones, dataset boundaries, ownership, lifecycle rules, and publish-safe storage conventions.
When to Use
- designing a new data lake
- reorganizing raw, staging, refined, or curated zones
- defining object storage layout and lifecycle rules
- separating landing, transformation, and publish responsibilities
- reducing data swamp behavior in shared lake storage
Do not use this to justify creating extra layers with no operational purpose.
Workflow
-
Define the lake purpose and consumers. Clarify:
- source landing needs
- internal producer teams
- publish consumers
- compliance and retention expectations
-
Define the zone model intentionally. Typical zones include:
- raw or landing
- standardized or staging
- refined or modeled
- publish or serving
-
Assign responsibilities to each zone. Decide:
- who writes to it
- who reads from it
- what quality guarantees exist
- whether mutation is allowed
-
Design storage conventions. Include:
- path or catalog naming
- partition strategy
- retention lifecycle
- file-size expectations
- ownership tags and metadata
-
Keep publish rules separate from lake convenience. Not every dataset in the lake is ready for shared consumption.
Cross-Cloud Architecture
Use references/cloud-data-engineering-architecture-patterns.md when the task is not only zone design, but choosing the overall cloud architecture pattern across lake, warehouse, lakehouse, streaming, and hybrid shapes.
Common Rationalizations
| Rationalization | Reality |
|---|---|
| "We can dump everything into one bucket or container and organize later." | That is how data lakes turn into data swamps. |
| "More zones always means better governance." | Extra layers without distinct purpose add complexity and slow teams down. |
| "If the file exists in the lake, it is available for analytics." | Raw landing data rarely has the quality or contract guarantees needed for shared use. |
Red Flags
- zone meanings overlap or are undocumented
- ownership is unclear at the dataset or zone level
- publish and landing data are mixed together
- retention, cleanup, or lifecycle policy is absent
Verification
- The zone model has clear purposes and boundaries
- Ownership, read/write expectations, and quality guarantees are explicit
- Storage conventions and lifecycle rules are documented
- Shared publish datasets are separated from raw landing data