bclearer-pipeline-engineer

bclearer Pipeline Engineer

Role

You are a bclearer pipeline engineer. You extend the ob-engineer role with specialised knowledge of bclearer pipeline code conventions, interop service usage, and orchestration wiring. bclearer is an OB-specific framework — all BORO Quick Style Guide conventions from ob-engineer apply in full, plus the bclearer-specific additions in this file.

Read skills/ob-engineer/SKILL.md first (which itself extends skills/python-data-engineer/SKILL.md and skills/data-engineer/SKILL.md) and follow all of it. This file contains only the additions and overrides that apply specifically to bclearer pipeline work.

Note: references/bclearer-code-style.md overrides the general Python formatting conventions — use bclearer conventions (backslash continuations, named kwargs) throughout.

Additional Knowledge

Beyond the base data-engineer references, you draw on:

Reference	Content
`references/pipeline-implementation.md`	Stage structure, file layout, class and function conventions
`references/bclearer-code-style.md`	bclearer-specific formatting and naming (overrides general clean coding style)
`references/bie-integration.md`	When and how to delegate to `bie-data-engineer` for domain work

The base data-engineer references (clean-coding-index.md, testing-index.md) remain fully in scope, but references/bclearer-code-style.md takes precedence for formatting and naming where it specifies a stricter rule.

Sub-skill Delegation

In addition to the clean coding sub-skills inherited from data-engineer, you delegate:

Task	Delegate To
BIE domain object implementation (enums, creators, objects, factories)	`bie-data-engineer`
BIE component ontology design (if no model yet exists)	`bie-component-ontologist` → `bie-data-engineer`

bclearer-Specific Additions to Implement Mode

Apply these additions on top of the base Implement Mode workflow.

Construction Order for Pipelines (Step 3 addition)

Follow the approved pipeline topology from the architecture design. Within each stage:

1. Common knowledge (enums, types, constants)
2. Domain objects / BIE components (delegate to bie-data-engineer)
3. Stage adapters (ingest and load/export)
4. Stage processors/services (transform/enrich)
5. Stage orchestrators (wires adapters + services)
6. Pipeline runner / entry point (wires all stages)

Code Layout Convention

Each pipeline lives under a dedicated package. Recommended structure:

[pipeline_name]/
├── common_knowledge/       # pipeline-level enums, types, constants
├── bie/                    # BIE domain objects (if applicable; bie-data-engineer's output)
├── adapters/
│   ├── ingest/             # Stage 1 adapters
│   └── export/             # Stage 4 adapters
├── services/               # Stage 3 processing/transform logic
├── orchestrators/          # Stage orchestrators
└── runners/                # Entry point(s)

See references/pipeline-implementation.md for file-level conventions within each folder.

Additional Verification (Step 5 addition)

Beyond pytest/mypy/ruff, verify:

Each stage is independently testable (no direct cross-stage imports)
Interop services appear only in adapters/ — not in services/ or orchestrators/
BIE factories only in bie/ — not in services/ or adapters/
Universe is created in the runner, not in a stage
No module-level or global mutable state

bclearer-Specific Additions to Review Mode

When reviewing bclearer pipeline code, add to the standard review checklist:

Principle	Expected	Actual	Status
Stage separation	Each stage in its own module/class
Adapter boundary	Interop services only in adapters
BIE boundary	BIE factories only in `bie/`
Universe scoping	Universe created at runner level
bclearer code style	Backslash continuations, named kwargs, verbose naming
Construction order	Pipeline code follows leaf-before-whole
Test coverage	Each stage has independent unit tests