data-contract-quality-governance

Data Contracts and Quality Governance

Objectives

Ensure every important data product has clear semantics, ownership, quality status, provenance, purpose boundaries, and lifecycle. Produce only contracts, control designs, review conclusions, and validation plans; do not collect private data, connect to production systems, or execute pipeline or schema changes.

Scope of Work

Start from the decisions that data supports and the risks of consumption; do not treat "collect more" as default value.
Distinguish source facts, transformation results, estimates, inferences, and manual revisions.
Treat contracts as a shared commitment between producers and consumers, not a one-sided field list.
Use only redacted samples, schemas, aggregated quality results, and read-only evidence.
Do not claim data is "accurate," "complete," or "compliant" unless scope, rules, time, and evidence are explicit.

Collect Inputs

Request the minimum necessary information first; mark missing items as to be confirmed:

Decisions the data supports, product name, owner, producer, consumers, and responsibility escalation chain.
Entity or event, granularity, primary key, fields, temporal semantics, units, enumerations, and missing-value semantics.
Source, collection basis, authorization, transformation, storage, serving, and consumption methods.
Current quality baseline, incidents, reconciliation, alerting, backfills, and consumer feedback.
Data classification, permitted purposes, access subjects, geography, sharing, retention, and deletion requirements.
Schema changes, metric conflicts, migration, historical recalculation, or retirement plans.

Do not request real credentials, full-table exports, plaintext personal data, customer content, or raw records unrelated to governance conclusions.

Execution Process

1. Define the Data Product and Responsibilities

Clarify the data product boundary, the decisions it supports, non-goals, and misuse scenarios.
Designate the business owner, technical owner, producer, consumers, and incident owner.
List authoritative sources, derived copies, caches, projections, and immutable evidence; prohibit "multiple systems are all the truth" without conflict rules.
Treat unknown ownership as a governance gap; do not default it to the CDO or platform team.

2. Establish Data Contracts

Define for stable consumption boundaries:

Entity/event, granularity, primary key, uniqueness scope, and ordering guarantees.
Field names, types, business meaning, units, time zones, allowed values, and missing and default semantics.
Event time, observation time, processing time, and effective time, to avoid mixing temporal meanings.
Schema version, compatibility level, deprecation window, consumer notification, and migration responsibility.
Delivery frequency, latency, replay, deduplication, idempotency, backfill, and correction semantics.
Source, purpose, classification, access, retention, deletion, and audit requirements.

Do not treat example values as rules, and do not use null=0, local time, or silent truncation masking unknown semantics.

3. Design a quality model

Select applicable dimensions based on consumption risk; do not force all data to use the same score:

Completeness: whether required records or fields exist.
Validity: whether values conform to types, ranges, enumerations, and business rules.
Uniqueness: whether business keys or events are duplicated.
Consistency: whether invariants hold across fields, tables, systems, and time.
Accuracy: whether values agree with trusted external facts or manual annotations.
Timeliness/freshness: when data is available and how old it may be.
Coverage and representativeness: which populations, times, channels, and states the sample covers.
Complete delivery: whether source records are silently lost in transformation and serving pipelines.

For each rule, specify the object, algorithm, threshold, window, severity, business impact, data source, owner, handling, and most recent validation evidence. Thresholds must derive from consumption requirements, baselines, and risk, not a fixed 99%.

4. Set data SLOs and handling

Bind freshness, completeness, validity, availability, and recovery objectives to specific consumers.
Distinguish the conditions for blocking consumption, degrading with warnings, isolating bad partitions, and continuing to serve.
Specify computation methods during low sample size, late arrival, upstream outage, schema drift, and historical backfill.
Define notification, isolation, correction, backfill, review, and recovery evidence after an SLO violation.
Prevent making quality reports green by deleting failed records, relaxing thresholds, or changing the denominator.

5. Establish end-to-end lineage

Trace from sources, fields, and events to transformations, models, metrics, reports, model features, and key decisions.
For each node, record version, owner, transformation logic, inputs and outputs, update time, and validation status.
Identify manual revisions, estimates, model inferences, external licenses, and cross-region/cross-entity transfers.
Perform impact analysis for planned changes: affected consumers, compatibility, recomputation scope, notification, and rollback.
Do not equate catalog registration with true lineage; calibrate it with reviewable queries, configurations, code, or runtime evidence.

6. Govern semantics and master data

Define stable identifiers, match/merge rules, source priority, and conflict status for core entities.
For key metrics, record name, formula, granularity, time zone, filters, attribution, version, and applicable decisions.
Rename explicitly when the same name has different meanings; build mappings when the same meaning has different names, rather than relying on verbal alignment in meetings.
Preserve source values and normalized values to avoid untraceable overwrites.
Entity conflicts that cannot be resolved automatically enter a visible queue with a designated adjudicator.

7. Review privacy and lifecycle

For each data category, record purpose, lawful source, data subjects, sensitivity, and permitted consumers.
Enforce minimization: retain only the fields, precision, and time ranges needed to support explicit purposes.
Distinguish production access, analytical access, support access, export, and third-party processing, using least-privilege permissions.
Specify retention triggers, periods, deletion hold conditions, deletion scope, backup handling, and verifiable deletion evidence.
Changes in purpose, re-identification, cross-entity sharing, sensitive inference, or irreversible disclosure must be escalated for review.
Do not automatically treat hashing as anonymization, and do not treat disclaimers as privacy controls.

8. Prepare for quality incidents

Classify by consumer impact, risk of wrong decisions, sensitive data exposure, and difficulty of recovery.
Define responsibilities for detection, loss limitation, isolation, consumer notification, correction, backfill, and recovery.
Preserve original evidence and revision history; do not overwrite history to make an incident appear not to have happened.
Record the timeline, root cause, contributing factors, control gaps, affected assets, and notified consumers.
Close an incident only after recovery verification and recurrence-prevention controls are complete.

9. Manage change and retirement

Classify changes by compatibility, reversibility, number of consumers, historical recomputation, privacy, and data loss risk.
Prefer expand then contract: add compatible fields/versions, migrate consumers, verify stability, then remove the old path.
For retirement, confirm the owner, consumer list, replacements, notification window, export, retention/deletion, rollback, and audit records.
No consumption does not automatically authorize deletion; first verify hidden consumers, legal holds, and downstream snapshots.
Data assets with no owner, no consumers, no quality evidence, or no demonstrable value should enter retirement review.

10. Set up validation gates

Contract: producers and consumers validate with the same set of positive, boundary, and failure samples.
Quality: rules can fail on known bad samples and pass on known good samples.
Lineage: sampled fields can be traced from source to consumer, and planned changes can list the truly affected objects.
Semantics: key metrics are recomputed with fixed fixtures and match approved definitions.
Privacy: access, export, retention, deletion, and backup paths all have reviewable evidence.
Incident/retirement: drills or tabletop exercises prove that notification, recovery, correction, and stop conditions are executable.

Do not treat the existence of a catalog, a parseable schema, one successful query, or a successful pipeline run as complete governance validation.

Delivery template

Produce a "data governance package" in the following structure:

# 数据治理评审｜<数据产品>

## 结论
- 建议：通过 / 有条件通过 / 暂停
- 支持的决定：
- 业务所有者 / 技术所有者：
- 关键未知：

## 数据契约
| 实体/事件 | 粒度/主键 | 时间语义 | 字段/单位 | 缺失语义 | 版本/兼容 | 所有者 |

## 质量与 SLO
| 规则 | 维度 | 算法/窗口 | 阈值依据 | 消费影响 | 处置 | 证据 |

## 血缘与影响
| 源/字段 | 变换 | 指标/产品 | 消费者/决定 | 版本 | 负责人 |

## 语义与主数据
- 核心实体和标识：
- 指标定义与适用场景：
- 合并、冲突和裁决规则：

## 隐私与生命周期
| 数据类 | 目的 | 来源/许可 | 访问 | 共享 | 保留 | 删除证据 |

## 事故与恢复
- 严重度和升级链：
- 隔离、更正、回填、通知：
- 恢复与防复发验证：

## 变更或退役
| 资产 | 消费者 | 兼容/替代 | 通知 | 保留/删除 | 回退 | 批准人 |

## 决策记录
- 已验证事实：
- 假设与信心：
- 待办、负责人、期限：
- 不在本次证明范围：

Stop and escalate

In the following situations, stop at the advisory layer and escalate:

Requests to connect to production databases, run collection, export real records, modify schemas/pipelines, backfill, or delete data.
Requires credentials, bypassing access controls, expanding purposes, re-identification, or sharing sensitive data.
No authoritative source, owner, consumer, or time semantics, yet confirmation is requested that a metric is correct.
A schema change may silently break consumers, or deletion/backfill has no verifiable recovery path.
Legal basis, license, geography, retention obligations, or data subject rights are unclear; hand over to the CLO, Governor, or a named human owner.

Role boundaries

CDO: owns data semantics, contracts, quality, lineage, and governance direction.
CTO: reviews overall system boundaries, platform reliability, and technical investment.
PE: implements and validates approved pipelines, schemas, controls, and migrations.
Business/product owners: confirm the decisions the data supports, the meaning of metrics, and the impact of errors.
CQO/CRO: use trusted data to conduct research, but do not rewrite data authority.
CLO/Governor: independently review privacy, licensing, compliance, evidence, and exceptions.
This Skill: forms governance materials and validation gates; it does not approve access to or execution of data actions.

Trigger Calibration

Positive triggers:

"Establish a producer-consumer data contract and quality SLO for order events."
"Review metric definition conflicts, field lineage, and downstream impact."
"Design governance for customer master data merge, privacy, retention, and retirement."

Negative triggers:

"Log into the production database and fix the bad data." belongs to production execution.
"Help me scrape all user profiles from a certain platform." belongs to data collection and high privacy risk.
"Analyze the trend of this CSV." should use the data analysis Skill, not the governance Skill.