Skip to main content

Workers resilience: retry, timeout, concurrency, idempotency

The 7 Logiswift.Eos.Workers background jobs (AutomationTriggerConsumerJob, AutomationActionExecutorJob, OutboundEmailConsumerJob, OutboxDrainJob, InboundEmailPollerJob, DatabaseHealthMonitorJob, DraftExpiryWorker) used to have inconsistent retry/timeout/concurrency behavior — some had no timeout at all, and timer-based jobs could overlap a slow tick with the next. Standardized 2026-08-18; full design rationale in WORKERS-RESILIENCE-STANDARDIZATION-PLAN.md, this page is the shipped shape.

Retry — RabbitMQ consumers only​

AutomationTriggerConsumerJob, AutomationActionExecutorJob, OutboundEmailConsumerJob, and AuditLogConsumerJob get bounded in-process retry via JobRetryPolicy.Create(maxAttempts, baseDelay, logger, jobName) — a thin Polly v8 ResiliencePipeline wrapper, distinct from the HTTP-specific wrapper described in Outbound Resilience. 3 attempts, exponential backoff, by default.

On final failure, the message is nacked (requeue: false) rather than acked and dropped — routing it to that consumer's own dead-letter queue. See Dead-Letter Queues for the full mechanism. This reopened a former "no DLX, by design" stance once the real risk became clear: a permanently-failing message across 15 independent client deployments used to be logged and silently dropped with no way to inspect or replay it.

Timeouts — every job​

Every job wraps its per-message/per-tick/per-item unit of work in CancellationTokenSource.CreateLinkedTokenSource(ct) + .CancelAfter(timeout). All timeout values are configurable via Workers:Timeouts:*Seconds in appsettings.json — 30s for automation/health/draft-repository work, 60s for email send/IMAP account connect, 15-30s for narrower per-item work.

:::note Internal test-covered methods keep their own default OutboxDrainJob.DrainOnceAsync / InboundEmailPollerJob.PollOnceAsync/ PollAccountAsync/ProcessMessageAsync take the timeout as a trailing TimeSpan? xTimeout = null parameter with an in-method fallback, specifically so existing internal-visibility unit tests calling these methods directly keep compiling unchanged. :::

Concurrency — timer-based jobs only​

Every timer-based job (OutboxDrainJob, InboundEmailPollerJob, DatabaseHealthMonitorJob, DraftExpiryWorker) is wrapped in a SkipIfBusyGuard (a SemaphoreSlim(1,1) probe): TryRunAsync(work) skips (logging a warning) and returns false if the previous tick is still running, rather than letting two ticks overlap. RabbitMQ consumers don't need this — a broker consumer only ever processes one message's callback at a time by construction.

Idempotency — make the action idempotent, not the transport​

Accept at-least-once delivery/execution everywhere. All three distinct dedup needs reuse one shared IProcessedDomainEventGuard/EfProcessedDomainEventGuard (TryMarkProcessedAsync(Guid id, ct) → bool, backed by EOS_PROCESSED_DOMAIN_EVENTS) rather than three separate mechanisms:

  • AutomationActionExecutorJob dedups on AutomationActionExecutionMessage.ActionExecutionId.
  • OutboundEmailConsumerJob dedups on EmailMessage.MessageId (added specifically for this).
  • AutomationTriggerConsumerJob is deliberately not deduped — its redelivery window is narrower in blast radius (a duplicate log row, not a duplicate side effect) since the downstream action-execution hop already has its own guard.

OutboxDrainJob/InboundEmailPollerJob/DatabaseHealthMonitorJob/ DraftExpiryWorker needed no new dedup mechanism — each already had its own natural idempotency (outbox rows marked processed/failed by row id, an existence check before insert, a health check/status sweep both naturally safe to re-run).

Reference files​

JobRetryPolicy.cs, SkipIfBusyGuard.cs, IProcessedDomainEventGuard.cs/ EfProcessedDomainEventGuard.cs, WORKERS-RESILIENCE-STANDARDIZATION-PLAN.md, Workers/appsettings.json (Workers:Retry/Workers:Timeouts keys).