workflow-engine-design

Workflow Engine Design

Build the engine layer — not individual idempotent operations (see idempotent-financial-workflows), but the runtime that sequences them durably.

Orchestration vs Choreography

Orchestration: central coordinator holds explicit state; easier to observe, debug, and audit. Use for multi-step money flows and compliance-sensitive sequences.
Choreography: services react to events on a shared bus; decoupled, harder to trace. Use for loose fan-out notifications.
Hybrid: orchestrator per bounded context, events between contexts.

Name every state and every valid transition before writing code.
Store state durably before executing side effects — never derive current state from side-effect logs.
Terminal states: completed, failed, cancelled, compensated.
Persist the full input/output of each step; a replay must reproduce the same decisions without re-running external calls.

Step isolation: each step reads input, executes idempotently, writes output + new state atomically.
Outbox: cross-service steps commit local state + outbox message in one transaction; a relay delivers externally.
Timers/delays: store as a scheduled-event record; a polling worker or dedicated scheduler fires it. Never sleep().
Suspension/resume: workflow pauses waiting for an external callback; store a resume token; callback delivers it and the engine re-schedules the workflow.

For each step with an external side effect, register a compensating action before execution.
On failure, execute compensations in reverse order — each compensation must also be idempotent.
Record compensation outcome; never silently swallow compensation errors.

Lease-based polling: worker claims a task with a TTL lease; heartbeats extend it; expiry allows re-claim.
Fan-out/fan-in: parallel branches share the workflow ID with a branch index; fan-in waits for all branch completions.
Backpressure: bounded worker pool; queue-depth metric drives horizontal scaling.

Assign a schema version to each workflow definition. Running instances finish on the version they started.
Safe changes: add optional steps, add new terminal paths.
Breaking changes (require a new version): remove, reorder, or rename steps.
Migration: drain old version to completion, then decommission the handler.

Max attempt limit per step → move to dead_lettered state with full context snapshot.
Ops tooling required: list stuck/dead-lettered workflows; replay from a specific step; force-complete a terminal step.

Unit: synchronous in-process runner — no queues, no real timers. Inject step fakes for each external outcome (success, transient failure, permanent failure, duplicate callback).
Time advance: trigger timer steps programmatically; never sleep() in tests.
Regression: replay a recorded production workflow trace and assert final state.