finance-pipeline-dev

Financial investment research data pipeline development

Serves analysts, researchers, and the people who supply data to them at brokerages and funds. Default assumptions:The pipeline has no documentation or the documentation is outdated, the scheduling configuration does not match the code, and there is no ready-made reconciliation mechanism. Task order: first understand the update policy and current state, then decide on the plan. Understand current state → define update policy → define incremental → define idempotency → define backfill → define rerun recovery → deliver.

What this skill delivers

The user gives a pipeline design or troubleshooting requirement, and you produce four things:

1. Pipeline design pipeline.md: source/target, incremental strategy, idempotency semantics, backfill plan, rerun and recovery, scheduling dependencies, and SLA for each pipeline segment.
2. Executable scripts/configurations pipelines/: in the native format of the engine and scheduling system (.sql/.py/DAG configuration), parameterized window,by default only produces output, does not directly run production tasks。
3. Pipeline contract pipeline.yaml: source/target, update policy, frequency, SLA, dependencies, idempotency declaration—references the DDL-produced contract.yaml, and serves as the finance-data-qc timeliness check's CONFIRMED precondition.
4. Troubleshooting report(troubleshooting scenario): root cause localization, evidence level, remediation recommendations.

Pipeline design and contract are the deliverables; the exploration process must be traceable and rerunnable.

How to use this skill (routing overview)

SKILL.md only gives the process skeleton and dispatch rules. The specific methods are under references/ ,loaded on demand according to trigger conditions, do not read them all at once。

Stage	Action	When to load which reference
0 Boundary	Confirm source/target tables, update policy, SLA, deliverables (design/scripts/troubleshooting), execution permissions	No reference needed
1 Understand current state	Read the contract.yaml produced by DDL, existing pipeline definitions, target table structure (partitions/keys/temporal columns)	No reference needed; for unknown specifications go back to the DDL skill
2 Define update policy	Determine real-time/T+1 fixed-time/disclosure-driven/periodic release	Required reading references/incremental.md Section 1
3 Define incremental	Choose watermark/CDC/partition overwrite/full load based on policy and source capability	references/incremental.md
4 Define idempotency	Target key + atomic write + deduplication, consistent results when rerunning after failure	Required reading references/idempotency.md
5 Define backfill	Window parameterization, concurrency rate limiting, idempotent replay	references/backfill.md
6 Define rerun recovery	checkpoint, retry, dead letter, reconciliation, SLA alerting	references/rerun-recovery.md
7 Deliver	Produce pipeline.md + pipelines/ + pipeline.yaml	references/pipeline-contract.md

The update policy and target table keys and partitions come from the DDL contract; when the contract is missing, first go back to using-data-dev and follow the DDL responsibility to fill in the contract, or treat it as pending confirmation.

Hard specifications

1. Review is read-only. Use read-only access only during the investigation and design phase; do not directly modify production schedules or directly run production tasks unless the user explicitly approves the specific operation.
2. Budget first. Review query scope and upper limits; unbounded scans are prohibited.
3. Idempotency is the default requirement. Delivered pipeline scripts must be repeatable with consistent results; if this cannot be achieved, it must be explicitly declared and the risks explained.
4. Parameterize windows. Backfill and rerun scripts must not hard-code one-off dates; always use --from/--to or equivalent parameters.
5. Evidence grading. Mark exploratory observations as OBSERVED, semantic inferences as INFERRED; updates to policy and SLA may be marked CONFIRMED and written into the contract only with user confirmation or support from a versioned specification.
6. Destructive operations require approval. Clearing partitions, overwriting production data, dropping tables, changing schedules: first provide the script and impact scope, then execute after approval.
7. Native format delivery. Save scripts in a format natively executable by the target engine and scheduling system; do not disguise non-SQL as .sql。

Execution flow

1. Confirm boundaries. Record source/target tables, update policy (or pending confirmation), SLA, deliverables, and execution permissions.
2. Assess the current state. Read contract.yaml (keys/partitions/temporal columns), existing pipeline definitions (scheduling DAG/cron/scripts), and target table structures; put unconfirmed items on the pending confirmation list.
3. Define the update policy. Use incremental.md Section 1 to align with QC's four policies (real-time/T+1 fixed-time/disclosure-driven/periodic release); do not write an SLA if the policy is unconfirmed.
4. Define incremental processing. Choose a strategy based on the policy and source capabilities: time watermark, CDC, partition overwrite, full snapshot; specify how late arrivals and revisions are handled.
5. Define idempotency. Define atomic replacement/deduplication/upsert based on the target key and partition; specify how intermediate failure states are cleaned up.
6. Define backfill. Parameterize windows, parallelize by partition, apply rate limits, and check upstream readiness.
7. Define rerun recovery. Checkpoints and recovery points, retry strategy, dead letters, reconciliation, SLA alerts.
8. Delivery. Produce pipeline.md、pipelines/ scripts,pipeline.yaml; self-check against mandatory specifications.
9. Integrate with QC. Write the update policy, expected-but-missing windows, and pipeline timestamps into pipeline.yamlfor consumption by finance-data-qc 's freshness check.

See examples/few-shots.md。

Deliverable requirements

• Pipeline design pipeline.md: source/target for each pipeline segment, incremental method, idempotency semantics, backfill strategy, rerun recovery, SLA, dependencies, and pending confirmation list.
• Script pipelines/: natively executable, window-parameterized, no hard-coded one-off snapshots; with stable pipeline_id and version number.
• Pipeline contract pipeline.yaml: for structure see references/pipeline-contract.md; explicitly mark unknown items as null, and do not apply other markets' policies by default.
• Troubleshooting report(troubleshooting scenario): evidence grade for root cause hypotheses, disproof conditions, and remediation recommendations; do not write conclusions until the root cause is confirmed.

Situations that must stop

• The connection information contains plaintext passwords, private keys, or tokens that will be written into deliverables.
• Only write-permission accounts are available, so a read-only review boundary cannot be established.
• The task requires running destructive operations directly in the production environment (clearing partitions, overwriting, dropping tables, changing schedules), without explicit user approval for the specific operations.
• The source or target structure cannot be confirmed, so a verifiable incremental/idempotent plan cannot be written.
• Update policies and SLAs cannot be confirmed, yet are required to be hardcoded into the contract.
• The troubleshooting query cannot be scoped, so it may result in an unbounded scan.
• The requirement is essentially a final conclusion on table creation, quality checks, query interfaces, governance, or compliance: save the observation and return to using-data-dev to select the corresponding responsibility.