Skip to main content

RabbitMQ dead-letter queues: poison-message handling

Every RabbitMQ consumer job in Workers now dead-letters a message it can't process after retries exhaust, instead of the old ack-and-drop. Shipped 2026-08-25, replacing the earlier "no DLX, by design" stance once the real data-loss risk became clear — see Workers Resilience for the retry policy that runs before a message reaches this point.

Topology​

RabbitMqDeadLetterTopology.DeclareAsync(channel, queueName, querySuffix, ct) declares one shared dead-letter exchange per deployment (eos.dlx<QueueSuffix>, direct-typed) plus a per-consumer dead-letter queue (<queueName>.dlq) bound to it with a routing key equal to the queue's own name, and returns the x-dead-letter-exchange/x-dead-letter-routing-key arguments the caller passes into its own main-queue declare. All declare/bind calls are idempotent — whichever consumer job starts first wins.

Every consumer's ReceivedAsync handler tracks a local succeeded flag: ack on success (unchanged), BasicNackAsync(deliveryTag, false, requeue: false, ct) on final failure. RabbitMQ only routes a nacked/rejected message to the DLX when the queue itself carries the dead-letter arguments above. A message that fails to deserialize at all (bad JSON) is nacked/dead-lettered the same way; a successfully-deserialized-but-null message still acks, matching the prior silent-no-op behavior.

Admin surface​

IDeadLetterQueueAdmin/RabbitMqDeadLetterQueueAdmin (registered scoped — a fresh connection per call, since this is an infrequent admin action, not a hot path):

  • ListAsync reports each known consumer's DLQ name/source queue/current message count.
  • RequeueAsync(dlqName, maxMessages, ct) pulls up to maxMessages off a DLQ and republishes each one to its original source queue via the default exchange, then acks it off the DLQ. dlqName must be one of IDeadLetterQueueAdmin.KnownConsumerQueueNames — anything else throws before touching the broker. This is a fixed diagnostic tool over 4 known queues, not a general-purpose queue browser.

DeadLetterQueueController (Api/Controllers/Operations/Messaging/): GET api/v1/operations/deadletterqueues (list) / POST api/v1/operations/deadletterqueues/{dlqName}/requeue?maxMessages=N (requeue, capped at 500 per call).

:::note Gated differently from most Operations controllers This is gated by a manual CallerContext.FromClaims(User).IsSuperAdmin check inside the controller, not IAccessControlService/AccessPolicyRegistry — it's an ops/diagnostic tool over broker state with no MAS_MENU row, not a persisted resourceType/resourceId entity. Classified as a MechanismThreeOnlyController in AccessPolicyRegistryCoverageTests. :::

Redelivery safety​

A requeued-and-redelivered message is safe to reprocess because every consumer already has its own idempotency guard (see Workers Resilience) — AutomationActionExecutorJob/OutboundEmailConsumerJob via IProcessedDomainEventGuard, AuditLogConsumerJob via EOS_AUDIT_LOG.EventId's own UNIQUE constraint. AutomationTriggerConsumerJob stays deliberately un-deduped — same accepted-narrow-blast-radius reasoning as before.

:::caution Not built No automatic requeue/alerting on a growing DLQ — an operator has to check GET .../deadletterqueues (or a future alert on RabbitMQ's own queue-depth metric) and requeue manually today. :::

Reference files​

RabbitMqDeadLetterTopology.cs, IDeadLetterQueueAdmin.cs, RabbitMqDeadLetterQueueAdmin.cs, DeadLetterQueueController.cs, each consumer job's own doc comment.