message-queues

Message Queues

Design and diagnose asynchronous delivery without inventing guarantees the broker or application does not provide. Make the delivery contract explicit: who owns a message, when an effect is safe to repeat, how poison messages stop blocking progress, and how recovery avoids replaying more than intended.

Target versions (October 2026):

• Apache Kafka 4.3.1 (2026-06-25; 4.x is KRaft-only, ZooKeeper removed since 4.0.0 released 2025-03-18)
• RabbitMQ 4.3.6 current, 4.2.10 previous lane (4.2 community support ended 2026-07-31; https://www.rabbitmq.com/release-information)

When to use

• Designing Kafka, RabbitMQ, or compatible producer and consumer delivery behavior
• Investigating consumer lag, growing queues, duplicate effects, redeliveries, or stuck messages
• Choosing acknowledgement, retry, dead-letter, ordering, backpressure, retention, or replay rules
• Reviewing an event-driven workflow for idempotency, observability, and failure recovery

When NOT to use

• The failing layer is unknown during a live outage; use debug-triage to localize it first
• Building generic telemetry pipelines, alerts, SLOs, or dashboards; use observability
• Modeling a database transaction, schema, outbox table, or data-store recovery; use databases
• Designing HTTP, GraphQL, or gRPC request contracts; use backend-api
• Editing Kubernetes, Terraform, or broker deployment configuration; use the matching infrastructure skill
• CPU, heap, allocation, or lock profiling of a producer or consumer process; use performance-debugging

Workflow

1. State the contract before changing settings

Record producer success, broker durability, consumer acknowledgement point, retry limit, ordering scope, retention, and the permitted duplicate/loss window. "Exactly once" only applies to a stated broker-to-consumer boundary; an external side effect still needs an idempotency key or a transactional outbox/inbox protocol.

2. Trace one message through normal and failure paths

For a representative message, trace publish -> broker acceptance -> delivery -> durable business effect -> acknowledgement/offset commit. Then trace process crash after the effect, timeout, malformed payload, dependency outage, and retry exhaustion. Place the acknowledgement only after the durable effect (auto-ack or early commit trades reliability for loss), and make duplicate delivery harmless.

3. Bound pressure and failure handling

• Limit in-flight work to consumer capacity. RabbitMQ prefetch bounds unacknowledged deliveries; Kafka consumers need bounded poll/worker handoff and processing that stays within the group timeout.
• Retry only transient, classified failures with capped attempts and delayed or scheduled backoff. Do not immediate-requeue every failure: it can create a hot redelivery loop.
• Send exhausted, malformed, or permanently rejected messages to a dead-letter destination with the original payload, stable message ID, failure reason, attempt count, and source location. A DLQ is a recovery queue, not a discard bin: assign ownership, alerting, and a repair/replay procedure.
• Preserve order only where the broker can: Kafka orders within a partition and a RabbitMQ queue has delivery order constraints that competing consumers, priorities, requeues, and retries can disturb. Choose a key/partition or a single ordered consumer deliberately; never claim global order by default.

4. Diagnose from read-only evidence

Compare ingress rate, egress/ack rate, consumer count, in-flight messages, retry/redelivery rate, oldest-message age, DLQ rate, and Kafka consumer-group lag. A rising backlog with steady consumer throughput suggests insufficient capacity or increased input; high in-flight work plus slow acks suggests downstream saturation; repeating message IDs suggest a retry or idempotency defect. Inspect one correlation/message ID across producer, broker, consumer, and effect records before changing prefetch, scaling consumers, resetting offsets, purging queues, or replaying data. Read-only starting points (the CLIs ship with the broker; detect with command -v kafka-consumer-groups.sh rabbitmqctl): kafka-consumer-groups.sh --bootstrap-server <host:port> --describe --group <group> and rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers.

5. Recover with a scoped, measurable plan

Offset reset, purge, requeue, and broad replay change delivery state. Copy and track:

• Scoped: finite message range, target group/queue, idempotency protection, side effects, stop condition
• Baselined: lag or backlog recorded; for Kafka, current offsets captured as the rollback record
• Rehearsed: replay on a non-production copy or a quarantined message; Kafka reset run with --dry-run
• Approved: exact command, scope, and rollback or containment plan explicitly authorized
• Executed unchanged: Kafka swaps only --dry-run for --execute; the group must have no active members
• Verified: lag/backlog re-measured; if it diverges from the stop condition, halt and return to Scoped

kafka-consumer-groups.sh --bootstrap-server <host:port> --describe --group <group> > offsets-before.txt
kafka-consumer-groups.sh --bootstrap-server <host:port> --group <group> \
  --topic <topic>:<partitions> --reset-offsets --to-offset <offset> --dry-run

AI Self-Check

• Delivery guarantee names match the chosen producer, broker, consumer, and side-effect boundary
• Acknowledgement/offset commit follows the durable effect and a crash can safely redeliver
• Retryable and terminal errors are classified; attempts, backoff, and DLQ ownership are explicit
• Message identity and idempotency cover producer retry, redelivery, and replay
• Ordering claim is limited to the actual partition, queue, key, and consumer topology
• Backpressure limits in-flight work rather than hiding saturation with unbounded buffers
• Lag/backlog diagnosis includes ingress, egress, age, retries, and downstream latency
• Any destructive recovery action has an exact scope, expected effects, measurement, and approval
• Cross-cutting agent hygiene applied - see references/agent-hygiene.md

Output Contract

See references/output-contract.md for the full contract.

• Skill name: MESSAGE-QUEUES
• Deliverable bucket: audits
• Mode: conditional. When invoked to audit or review queue-related repository content, emit the full contract and write the deliverable to docs/local/audits/message-queues/<YYYY-MM-DD>-<slug>.md. Design, diagnosis, and recovery guidance remains conversational.
• Severity scale: P0 | P1 | P2 | P3 | info

Sources

• Apache Kafka documentation - consumer groups, ordering, offsets, idempotent producers, and transactions
• RabbitMQ acknowledgements and confirms - manual acks, redelivery, publisher confirms, prefetch, and requeue behavior
• RabbitMQ quorum queues - dead-lettering guarantees

Rules

1. Assume at-least-once delivery unless the complete boundary proves otherwise. Consumers must tolerate duplicates; producer retries need stable message IDs.
2. Make retries bounded and observable. Include jitter/delay where supported; route poison messages away from the hot path.
3. Measure age as well as count. A small backlog containing old messages can be more urgent than a large fresh burst.
4. Do not alter live broker state during diagnosis. Read metrics, logs, and configuration first; request approval for offset resets, purges, requeues, topology edits, or production replay.