plan-storage

/plan-storage — Storage Plan interview

This skill runs a short, branching interview that converts a planned (or existing-but-undocumented) storage destination into a Storage Plan markdown file at docs/storage-plans/<dataset>.md in the consumer's project. It is an interview, not a template — every plan that lands has been pressure-tested against the Storage kill-switch (a named upstream Ingestion Plan), the storage-system-neutrality rule, the honest-deletion-mechanism test, the residency-inheritance test, the access-control-enforcement test, and the "is this actually an ADR moment?" routing test.

This skill is self-contained: the Storage Plan contract it writes against ships beside it at references/storage-plan.md, and the linter that enforces that contract ships at scripts/lint-storage-plan.sh. Read the contract for the frontmatter fields, enum domains, and body section order before running — do not duplicate that material into the interview; reference it.

Optional deeper reading — not required to run this skill (a per-skill install works without them): the full repo worldview (including the CONTEXT.md entries "Storage Plan" and "Storage kill-switch") in CONTEXT.md and the lifecycle reference in docs/data-engineering-101.md.

A Storage Plan is not an infrastructure-as-code template — not a Terraform module, not a DDL file, not a bucket-policy JSON. It is the operational and contractual model of the storage layer: the human-authored commitments (format, retention, residency, access, tiering, partitioning) that the actual infrastructure should reflect. The IaC that implements those commitments lives in the consumer's infrastructure repo and is out of scope — this skill emits no Terraform, no dbt config, no DDL. The plan is the contract the IaC should be reviewed against, not a substitute for it. It is the structural analogue of the Ingestion Plan one stage upstream, deliberately human-authored and never inferred from a cloud API (no BigQuery INFORMATION_SCHEMA scan, no Snowflake account-usage read, no S3 bucket listing).

Soul of the skill: the Storage kill-switch

The single load-bearing question this skill exists to force is:

Which Ingestion Plan(s) under docs/ingestion-plans/ does this storage destination receive data from?

If the user cannot point to at least one accepted-or-draft Ingestion Plan in related_ingestion_plans, the storage does not deserve a plan. A Storage Plan with no named upstream ingestion is the structural twin of an Ingestion Plan with no named upstream Source Profile, of a Source Profile with no consumer DGQ, and of a DGQ with action_change: nothing — documentation that floats free of the lineage graph, the failure mode this stage exists to refuse. Stage 3 is the stage where floating nodes are most economically damaging: orphan storage is orphan spend. Surface that conclusion explicitly. Recommend the user run /plan-ingestion first; come back to /plan-storage once the upstream ingestion path is planned.

Do not soften the kill-switch. Do not paper over the absence by inventing a plausible-sounding Ingestion Plan path on the user's behalf. The kill-switch (Turn 1) is the one place in this skill where the user must speak first; a recommendation there would coach them past the very gap the skill exists to surface.

The linter enforces the structural contract — an accepted plan with an empty related_ingestion_plans fails scripts/lint-storage-plan.sh. This skill is the upstream half: it refuses to start the interview without one named upstream Ingestion Plan.

Target shape

• Length. 8–12 substantive exchanges per plan is typical — slightly longer than Stage 2, because retention, residency, tiering, and deletion mechanism are independently load-bearing concerns that cannot be folded into a single turn without losing the honesty posture. Keep going until the operational, lifecycle, and access contract is honestly named — don't truncate to hit the cap, and don't pad to fill it. Sharp, not exhausting.
• Output. One markdown file per dataset at docs/storage-plans/<dataset>.md. If one Ingestion Plan lands into several stored representations (raw + curated, or a hot table + a cold archive), write one fully-drilled plan first, the rest queued as status: draft so a later run can pick them up.
• Tone. Interrogative, not adversarial. The goal is to find the honest contract — including "retention is a wiki page nobody enforces" and "we grant SELECT to everyone and police PII by convention" answers — not to make the user feel bad for inheriting an underspecified storage layer.

Resumability check (run this first)

Before starting a fresh interview, scan docs/storage-plans/ in the consumer's project. If one or more Storage Plan files exist, pause and ask:

I see existing Storage Plans in docs/storage-plans/. Do you want to:

1. Continue an existing draft or needs-follow-up plan (resumable)
2. Start a new plan
3. List all plans and their statuses

Treat status: accepted, status: rejected, and status: superseded-by-<path> files as immutable from this skill's perspective — do not offer to edit them. Superseding an old plan is done by writing a new one and updating the old one's status by hand. If the user picks "continue", load the chosen file and resume from the first unfilled section (commonly the section that corresponds to the next unanswered turn). If they pick "list", print one line per file: <path> — <status> — <dataset> → <storage_system>.

If docs/storage-plans/ does not exist, create it on first write — not before.

Steps

Carry out the nine turns below in order. One question per turn, not a wall of multiple questions, except where the spec explicitly says "combined turn" (Turn 8). If the user volunteers material that answers a later turn early, accept it and skip ahead — do not re-ask. Do not deliver multi-paragraph monologues; keep each prompt tight.

Two cross-cutting rules apply to every turn:

• Recommend before you ask. For each turn's question, propose a plausible answer first — derived from the named upstream Ingestion Plan, the dataset name itself, adjacent plans/ADRs, or sensible defaults — then ask the user to accept, reject, or refine it. Reacting to a concrete proposal is faster and sharper than generating one cold. If you genuinely have no basis for a recommendation, say so and ask open. Never invent the kill-switch answer (Turn 1) this way — the named upstream Ingestion Plan is the one place the user must speak first. And never nudge toward a multi-tier tiering posture (Turn 5) or a particular storage_system (Turn 3) just because the system supports it — those defaults are exactly the template-mode bias this skill exists to refuse.
• Explore before you ask. If a turn's question can be answered by looking at the project — existing Ingestion Plans in docs/ingestion-plans/, Source Profiles in docs/sources-of-record/, DGQs in docs/requirements/, ADRs in docs/adr/, adjacent Storage Plans in docs/storage-plans/, glossary entries in CONTEXT.md — look first, then bring what you found into the turn. In particular, always read the named upstream Ingestion Plan(s) once identified in Turn 1, and the Source Profile(s) they chain to; their sensitivity, delivery_mode, residency/governance notes, and operational contract seed Turns 5–8 directly. Do not make the user re-derive what the repo already knows. If a stakeholder term collides with CONTEXT.md's glossary, call it out the moment you notice it.

Turn 1 — Name the upstream Ingestion Plan(s)

Ask, verbatim or close:

Which Ingestion Plan(s) under docs/ingestion-plans/ does this storage destination receive data from? Point me at one or more by filename or by pipeline/destination.

Before asking, scan docs/ingestion-plans/ and list the available Ingestion Plan files with their pipeline/destination line and status — so the user is choosing from a concrete set, not naming a file from memory. If docs/ingestion-plans/ is empty or absent, that is itself the answer: there is no planned upstream ingestion; the storage does not deserve a plan yet.

Three branches:

• One or more named upstream Ingestion Plans. The user points to one or more files with status: accepted or status: draft. Record the paths; they go into related_ingestion_plans. Read each one — and the Source Profile(s) it chains to — before continuing; their sensitivity, delivery_mode, governance notes, and operational contract seed the next several turns. Continue to Turn 2.

• A plan exists but it is rejected or superseded-by-<path>. Push back: "That Ingestion Plan was rejected — what's the live upstream for this storage?" / "That plan was superseded by <path> — should we plan storage against the superseding one instead?" Do not start the plan against a non-live ingestion path; the upstream link must be an ingestion someone has actually accepted as the contract.

• No Ingestion Plan exists for the upstream, or the user cannot name one. Trigger the kill-switch. Do not write a plan. Tell the user:

  There's no named upstream Ingestion Plan for this storage. A Storage Plan without a planned ingestion path is documentation that floats free of the lineage graph — and at Stage 3 that floating node is orphan spend, not just orphan docs. I recommend running /plan-ingestion first to plan the ingestion path, then coming back to /plan-storage once at least one Ingestion Plan under docs/ingestion-plans/ is in place.

  Then exit cleanly. Do not start drafting a plan "just in case." The kill-switch is the differentiator; surface it without apology.

Turn 2 — Identify the dataset

Ask:

What's the dataset — a kebab-case dataset slug — and is this one dataset or several?

Capture one field now:

• dataset — kebab-case slug uniquely naming the stored dataset (raw-stripe-charges, fct-orders, audit-events-eu, customer-pii). Aim for a name that reads as "what is stored," not a code-repo or table-engine name.

Listen for plural datasets hiding inside one ask. A single Ingestion Plan often lands into multiple stored representations: raw-stripe-charges (landed passthrough) plus fct-charges (curated analytical table); or orders-hot (operational table, 90-day window) plus orders-cold-archive (object-store, 7-year retention). Each is a separate Storage Plan — they have genuinely different partition strategies, retention, tiering, access patterns, and cost envelopes.

Branching rule. If more than one dataset surfaces:

1. Ask the user to pick one to drill fully right now.
2. For each of the others, write a stub Storage Plan to docs/storage-plans/<dataset>.md with status: draft, frontmatter populated with the upstream Ingestion Plan(s) from Turn 1 in related_ingestion_plans, the dataset identifying the stub, and all other linter-required fields set to safe placeholders (storage_system: object-store, format: parquet, partition_strategy: not-applicable, retention: tbd, residency: no-constraint, sensitivity: internal, access_pattern: analytical, tiering: single-tier, deletion_mechanism: manual), with a note in ## Open follow-ups that this plan needs resumption. Body sections present but with TBD bullets, so the linter passes.
3. Tell the user the stubs are saved and this session will drill the chosen dataset. Do not abandon them silently.

When a derived destination chains through a durable one (see Turn 3), queue the durable Storage Plan as a stub here even if the user only asked about the derived store — the derived plan should reference a durable plan, not float free.

Turn 3 — Storage system and format

Ask:

Where does this dataset live — warehouse, lakehouse, object-store, operational-store, search-index, or cache — and in what format?

Capture two fields:

• storage_system — one of warehouse | lakehouse | object-store | operational-store | search-index | cache. All six are equally first-class. Do not default to warehouse or lakehouse. The framing must come from the user's answer, not from a built-in lean — storage-system neutrality matters here as much as streaming/batch neutrality does upstream.
• format — one of native | parquet | orc | avro | json | delta | iceberg | hudi | row-store. native is the honest answer for a warehouse's own table storage or an operational store's engine format; the open table formats (delta, iceberg, hudi) are lakehouse answers; parquet/orc/avro/json are object-store / file-layout answers; row-store is the operational-store / cache answer.

Durable-or-derived sub-question. For operational-store, search-index, or cache, surface explicitly: "Is this destination the durable record, or a derived copy of one (an index rebuilt from a table, a cache populated from a warehouse, a read replica)?" A derived destination should chain through a durable Storage Plan — name the durable plan in related_ingestion_plans or queue it as a stub (Turn 2) — rather than skip Stage 3 entirely. A derived store with no durable plan behind it is the Stage 3 analogue of a floating node.

Branch the layout questions (Turn 4) on the storage_system answer. Ask only the relevant sub-questions; don't ask warehouse clustering questions of a search-index, or cache-eviction questions of a lakehouse.

Turn 4 — Layout

Ask, branched on Turn 3's storage_system:

• warehouse: "What's the partitioning / clustering strategy — clustering keys, partition column, materialised views? What's the table-organisation hint Stage 4 should assume?"
• lakehouse: "What are the partition columns, and what's the target file size (to avoid small-file or over-compaction problems)? Compaction cadence?"
• object-store: "What's the key/prefix layout — date-partitioned prefixes, hive-style partitions, a single blob? Object naming convention?"
• operational-store: "What's the primary key, and what secondary indexes exist? Any sharding / partition key?"
• search-index: "What's the shard count and refresh interval? How is the index rebuilt, and from what durable source?"
• cache: "What's the key namespace and the eviction policy (TTL, LRU, LFU)? What's the source of truth this cache is populated from?"

Map the answer to partition_strategy (free-text, OR the literal not-applicable). not-applicable is a valid answer for genuinely unpartitioned single-blob storage (a single file, a small lookup table, a cache with no logical partitioning) — but it must carry a body rationale in ## Layout & format. It is a written-down decision, not a soft-skip; the linter accepts it as deliberate non-empty. Do not record not-applicable to bypass the question.

Capture the rest of the layout shape in the body's ## Layout & format section — the clustering/index/file-size detail above, and the relationship to landed-raw: does this dataset replace the landing layer, promote it (curate raw into a modelled table), or sit alongside it? Stage 4 (Transformation) reads from here; what this section commits is what Stage 4 is allowed to assume about physical layout.

Turn 5 — Lifecycle (the loudest turn)

This is the turn where template-mode regression and aspirational-policy-drift are most likely. The skill is explicit and slow here. Capture three fields: retention, tiering, deletion_mechanism.

Ask, in two passes:

Pass A — retention and tiering:

How long is this data kept, and what justifies that — legal, business, contractual? And is it single-tier, or does it move hot-cold / hot-warm-cold over its life?

Map to:

• retention — free-text: "7 years", "90 days", "indefinite", "until-deletion-request". Record what justifies it in ## Lifecycle.
• tiering — one of single-tier | hot-cold | hot-warm-cold. single-tier is the honest default for most small datasets. Do not nudge toward multi-tier just because the storage system supports it — hot-cold and hot-warm-cold should be deliberate choices justified by scan-frequency analysis. If multi-tier, name in ## Lifecycle when partitions move and by what mechanism.

Pass B — deletion mechanism (the honesty test):

How is retention actually enforced today — not what you want to be true. Is there a lifecycle-policy baked into the storage system, a scheduled-job running on a cadence, a manual cleanup procedure, a retention-lock immutability setting, or is deletion genuinely not-applicable (a permanent record)?

Map to deletion_mechanism: one of lifecycle-policy | scheduled-job | manual | retention-lock | not-applicable. Three rules:

1. Honest, not aspirational. "We have a wiki page saying 7 years" is not lifecycle-policy — if no lifecycle rule or scheduled job actually runs, you have manual (someone deletes by hand) or you have a gap. Record manual honestly and add an ## Open follow-ups line if the gap should be closed. Silent aspiration — writing lifecycle-policy because it sounds right — is the exact failure this turn exists to catch. manual is an honest valid answer and must not be punished by the interview; what should be punished is pretending automation exists.
2. If the user cannot answer, do not paper over it. Set status: needs-follow-up, name the specific outstanding question ("is there a lifecycle policy on this bucket, or does anything actually delete old partitions?"), stop the interview, and prompt the user to go and find out. Retention-by-wiki-page is the Stage 3 failure the kill-switch in spirit guards against, even though the structural kill-switch is upstream-chain.
3. not-applicable is a valid honest answer for genuinely permanent records (an immutable audit trail, a regulatory ledger). If not-applicable, the body must name why deletion does not apply in ## Lifecycle — not just leave it implicit. The linter accepts it as deliberate non-empty.

Record retention justification, tiering mechanism, and the deletion-mechanism reality (including any wiki-page-vs-enforcement gap) in the body's ## Lifecycle section. This is the Stage-3-distinctive section; do not let it be one line.

Turn 6 — Residency and sensitivity

Capture residency and sensitivity. Before asking, read the named upstream Ingestion Plan(s) and the Source Profile(s) they chain to — their sensitivity and any residency/governance notes seed this turn directly.

Ask:

The upstream chain records sensitivity: <upstream sensitivity> and <residency note, if any>. Where does this storage physically live (the residency commitment), and what's the sensitivity of the data at rest here?

Map to:

• residency — free-text but required: "eu-west", "us-only", "multi-region", "no-constraint". Real pipelines often inherit a residency obligation transitively from the upstream Source Profile and Ingestion Plan, but the storage layer gets provisioned in a region chosen for convenience. Surface the inheritance explicitly: "the upstream is sourced from <region> — is the residency commitment inherited, or has it been deliberately broken?" no-constraint is valid only when the upstream chain is genuinely unconstrained — not as a default for "we didn't think about it." Record where residency is enforced (storage location, replication policy, third-party tool boundaries) in ## Lifecycle.
• sensitivity — one of public | internal | confidential | regulated. This enum is deliberately identical to the Source Profile's and Ingestion Plan's; cross-stage enum drift would break the lineage-graph reasoning the kill-switch chain enables.

Sensitivity step-down gap. If this Storage Plan's sensitivity is lower than the upstream Ingestion Plan's (a regulated ingestion landing into an internal storage), surface the comparison explicitly: "the upstream ingestion is regulated; this storage is internal — is there a de-identification step, and where is it enforced?" This is either a deliberate de-identification step (worth recording, with a pointer to the masking/pseudonymisation mechanism) or a smell worth interrogating. Disagreement is allowed but must be recorded in the body's ## Access pattern section — never silently accepted.

Turn 7 — Access pattern

Capture access_pattern. Ask:

Who reads this dataset and how — analytical scans, operational point-reads, a hybrid of both, or archival retrieval? And how are row/column-level controls, masking, and pseudonymisation enforced — and where?

Map to access_pattern: one of analytical | operational | hybrid | archival.

Then drill the controls, because this is where Stage 3 most often degrades into a wiki page. For each control the user names, ask where it is enforced — a database view, a policy tag / column-masking policy, a separate table, or the application layer. The honest answer "we grant SELECT on the table and police PII reads by convention" is recorded as such — it is Stage 3's analogue of Stage 1's "us, by accident" posture. "By convention" is an honest answer that should be written down, not papered over. Record the access pattern, the controls, where each is enforced, and any sensitivity-step-down disagreement from Turn 6 in the body's ## Access pattern section.

Turn 8 — Operational contract

Ask, in one combined turn:

When this storage misbehaves: what's the cost envelope class, who owns the spend, who's paged when lifecycle automation fails, where's the runbook, and what does this plan assume about the upstream Ingestion Plan's landing contract?

Five facts to capture in the body's ## Operational contract section:

• Cost envelope class. Rough scale: small (a lookup table, a handful of GB), medium (a recurring analytical table, tens to hundreds of GB), large (a multi-terabyte archive, a continuously-scanned hot table). Storage + scan + lifecycle cost together. Recorded as prose; the linter does not police it.
• Spend ownership. Who owns the storage spend — platform, data team, business unit, chargeback. "Nobody — it's on the platform bill and nobody reads it" is an honest answer worth recording.
• Pager for lifecycle-automation failure. Who is paged when a partition migration stalls, a deletion job fails, or a residency rule is violated. If the honest answer is "nobody — we find out when the storage bill spikes", record that.
• Runbook location. A path, a Confluence URL, or "none yet." If "none yet," that goes to ## Open follow-ups.
• Upstream-dependency expectation. Does this plan assume the upstream Ingestion Plan's landing contract holds? What happens when it doesn't — does the storage degrade, drop writes, accumulate orphan partitions? Name the assumption and the failure mode plainly.

Turn 9 — Write the plan

Pick a filename: <dataset>.md under docs/storage-plans/, the slug kebab-case throughout (lowercase letters, digits, hyphens; single-word slugs are technically valid but at least one hyphen is recommended for readability).

Write the file shaped exactly as the contract in references/storage-plan.md specifies — the fourteen frontmatter fields and the six body sections in order. The earlier turns already gathered the content each field and section needs: the upstream Ingestion Plan(s) from Turn 1, dataset from Turn 2, storage_system and format from Turn 3, partition_strategy and the layout detail from Turn 4, retention, tiering, and deletion_mechanism from Turn 5, residency and sensitivity from Turn 6, access_pattern and the access-controls detail from Turn 7, and the operational contract from Turn 8. Read the contract for the exact field list, enum domains, and section order rather than reproducing them here.

related_ingestion_plans carries the Ingestion Plan path(s) from Turn 1; the linter requires at least one entry for status: accepted. related_dgqs and related_adrs stay [] at first write — populate them later only if cross-links accrue (the upstream chain names a consumer DGQ worth recording transitively, or an ADR-moment surfaces and the user agrees to run /write-adr). Do not invent paths.

After writing, print a one-line confirmation:

Wrote docs/storage-plans/<dataset>.md
  status: <status>

No body summary echoed back — the user just answered the questions; reading it back is noise. Then surface the next queued plan (if Turn 2 created stubs): "You have N draft plans queued from this session — want to drill the next one now?" If the user declines, exit cleanly.

Run the bundled structural linter scripts/lint-storage-plan.sh — it sits beside this SKILL.md, in the skill folder's own scripts/ — as a cheap sanity check after writing. Resolve it relative to this skill folder and invoke it from the consumer project root so it picks up docs/storage-plans/ (or point it elsewhere with STORAGE_PLANS_ROOT=<dir>). It is best-effort: if the script is present in this install, run it; if it is absent — a partial install that dropped the skill's scripts/, say — skip it silently and rely on the consumer's pre-commit hook. It is not a CI gate — just a fast structural check that frontmatter, enums, body sections, and the kill-switch are intact. If the schema enforced by the linter drifts from this prompt, the linter wins — record a fix-up issue and ship the plan that passes the linter.

ADR-moment detection

At any point in the interview, if the decision being recorded is a cross-cutting platform commitment rather than a per-dataset plan, pause and route. Signals that you are looking at an ADR-moment:

• The user says something that applies across datasets, not just this one. Example phrasings to listen for:

  "All PII columns go into a separately governed schema — that's our standard." "All analytical storage is Iceberg-on-S3; we don't do warehouse-native tables any more." "Every dataset gets a 7-year retention floor unless legal grants an exception." "Cold tier is always Glacier after 90 days, full stop."

  These are platform commitments, not per-dataset facts.

• The decision is about a default storage system across a class of datasets ("lakehouse for analytical, operational-store for serving").

• The decision is about a default retention or tiering posture across datasets ("hot-warm-cold for everything over a terabyte").

• The decision is about a default access-control model across datasets ("every regulated table is column-masked by policy tag, no exceptions").

• The decision applies across many datasets, teams, or storage targets rather than to this one dataset.

When you spot one of these, stop the plan interview and say something like:

This isn't a per-dataset plan — it's a cross-cutting platform commitment. The right artefact is an ADR, not a Storage Plan. I recommend running /write-adr separately to capture this decision in docs/adr/. After the ADR lands, you can come back and link it from any related Storage Plans via the related_adrs frontmatter, and the per-dataset plan will just reference the ADR rather than restate it.

Do not inline the ADR scaffold here. One skill, one artefact — the ADR belongs to /write-adr. If the user insists on continuing the plan, write it but note in ## Open follow-ups that the recorded pattern likely should be promoted to an ADR.

After writing

• Do not summarise the body of the Storage Plan back to the user — they just answered the questions; reading it back is noise. The one-line confirmation is sufficient.
• If the conversation surfaced a cross-cutting commitment that the user agreed to record separately, remind them once at the end: "Run /write-adr for <topic> when you're ready — it's not part of this plan's scope."
• If a plan ended needs-follow-up, name the specific question the user needs to take to platform, security, legal, or finance. Vague follow-ups rot; named ones get answered. The most common one here: "confirm whether a lifecycle policy actually runs on this storage, or whether retention is manual."
• If a plan ended rejected — the storage was planned defensively and the user decided the upstream ingestion chain doesn't justify it after all — leave it in place with the rejection rationale in ## Open follow-ups. The rejection is a durable record of the deliberate decision not to build, not a TODO to revisit.