Overview
Preserve freight correctness first. Freeze uncertain mutations, separate provider delivery from local processing, and reconstruct affected state with documented resource/event reads.
Prerequisites
- Incident owner, severity, start time, and affected tenant/workflow
- Last known good release and configuration
- Bounded reconciliation query and business owners for disputed state
Instructions
Step 1: Contain
Disable automatic bookings and affected writes. Keep authenticated webhook acceptance only if durable enqueue and deduplication remain correct.
Step 2: Bound the window
Identify first/last bad receipt by surface, operation, release, credential alias, and version without examining broad sensitive payloads.
Step 3: Split failure domains
Test OAuth, REST read, MCP connection, webhook ingress, queue processing, and downstream state independently.
Step 4: Reconcile read-only
Use /events or affected resource reads to compare provider state with durable local operation keys. Do not assume Flexport exposes a webhook replay endpoint.
Step 5: Repair deterministically
Reprocess authenticated durable events or apply approved state corrections exactly once; reconcile uncertain creates/bookings before retry.
Step 6: Recover and learn
Restore traffic by cohort, verify the full incident window, rotate exposed credentials if needed, and add a sanitized regression fixture.
Authentication
REST calls authenticate with a cached OAuth 2.0 client-credentials Bearer token using audience https://api.flexport.com, or an explicitly accepted broad API key. Use distinct credentials per workload and never log credentials or tokens. MCP calls use the authenticated connection to https://mcp.flexport.com/mcp and remain subject to each tool's documented account permissions.
Tool Discipline
Use Read and Grep for discovery and evidence. Use Write or Edit only for the approved artifact, code, configuration, test, or receipt described by this workflow; do not make an unapproved Flexport-side change.
Output
- Scoped decision or implementation artifact
- Redacted operation and validation receipt
- Failure, rollback, and follow-up ownership record
Return a machine-reviewable receipt in this shape; adapt the operation values, but never place credentials or provider payloads in it:
surface: rest-v3
operation: shipment-read
decision: approved
outcome: verified
evidence:
release_sha: recorded-out-of-band
provider_reference: redacted
rollback_owner: logistics-platform
Examples
A receiver deployment parsed bodies before signature validation and dropped events. The team restores the prior release, queries the bounded event/resource window, deduplicates by durable operation identity, and backfills only missing local transitions.
Error Handling
| Failure | Response |
|---|---|
| Provider reads unavailable | Maintain the mutation freeze and preserve the reconciliation window. |
| Duplicate business action found | Stop workers and choose one authoritative provider resource. |
| Event type newly additive | Update tolerant routing and replay only authenticated durable items. |
| Incident needs sensitive payload | Escalate access and minimize fields rather than copying broad logs. |