Skip to main content

Distributed tracing (OpenTelemetry)

Closes the old correlation-ID model's gap — no distributed propagation without real OpenTelemetry instrumentation. All three composition roots migrated from the classic Microsoft.ApplicationInsights SDK to the Azure Monitor OpenTelemetry Distro — same Azure Monitor/App Insights backend, but real ActivitySource-based spans with W3C traceparent propagation instead of single-hop-only Activity.Current.

Building blocks​

  • EosActivitySource (Application/Common/EosActivitySource.cs) — the one process-wide ActivitySource (name "Logiswift.Eos") every host and Infrastructure share. Lives in Application, not Infrastructure, purely so both Infrastructure (RabbitMQ spans) and all three composition roots can reach it without a new project reference — it depends on nothing but System.Diagnostics.
  • Package swap: Api/Api.External use Azure.Monitor.OpenTelemetry.AspNetCore (bundles the OTel SDK, ASP.NET Core/HttpClient/SqlClient instrumentation, and the Azure Monitor exporter behind one AddOpenTelemetry().WithTracing(...).UseAzureMonitor(...) call). Workers has no ASP.NET Core request pipeline, so it composes the OpenTelemetry SDK manually instead.
  • Kill switch / sampling: ApplicationInsights:Enabled (default true) and an empty/missing ApplicationInsights:ConnectionString both skip registering the Azure Monitor exporter entirely — ActivitySource-based tracing still works locally, nothing ships. ApplicationInsights:SamplingMaxItemsPerSecond maps to a fixed SamplingRatio (itemsPerSecond / 100, clamped to [0.01, 1.0]) — a deliberately rough translation, not a like-for-like match of the old SDK's adaptive sampling.
  • TelemetryClient retired everywhere — its former injection sites (GlobalExceptionHandler, DatabaseHealthMonitorJob, OutboundEmailConsumerJob) record directly onto Activity.Current instead (AddException/AddEvent/SetStatus), all safe no-ops when Activity.Current is null.

RabbitMQ hop propagation — the actual cross-process fix​

Before this, no publisher set AMQP message properties/headers at all — CorrelationId rode only inside the JSON body, and trace context rode nowhere, so Activity.Current?.Id necessarily died at every queue hop.

RabbitMqTraceContext (Infrastructure/Messaging/) is the fix — two pure static methods operating on IDictionary<string, object?>, no RabbitMQ.Client types in the signature so they're unit-testable with no broker involved:

  • Inject(headers) — writes Activity.Current?.Id (W3C traceparent) and TraceStateString into the headers dictionary. No-op if there's no current activity.
  • TryExtractParentContext(headers, out context) — reads a previously- injected pair back out (decoding RabbitMQ's byte[]-encoded header values via UTF-8), returning false for missing/malformed input so the caller starts a new root activity instead of a bogus child one.

Every publish site (domain-event publisher, automation-action queue publisher, EmailDispatcher) now wraps its publish in a Producer-kind EosActivitySource span with Inject-populated headers. Every consumer job extracts the parent context at the top of its handler — before setting the business CorrelationId accessor, keeping that model fully independent — and wraps the rest of the handler in a Consumer-kind span parented to it: a genuine distributed trace stitched across the RabbitMQ boundary, not just a shared CorrelationId string.

:::caution RabbitMqDeadLetterQueueAdmin.RequeueAsync fixed along the way It used to republish only the dead-lettered message's Body, dropping its BasicProperties — fixed to forward them so a manually-requeued message keeps its traceparent/tracestate headers. Broker-side DLX routing already preserves headers for free on a nacked message; this was the one place code dropped them. :::

:::danger Not verified Actual end-to-end trace stitching in the Application Insights UI is unverified — no live Azure Monitor connection string exists in this environment. :::

Reference files​

EosActivitySource.cs, RabbitMqTraceContext.cs, RabbitMqTraceContextTests.cs, RabbitMqDomainEventPublisher.cs/RabbitMqAutomationActionQueuePublisher.cs/ EmailDispatcher.cs (the 3 publish sites), the 4 consumer jobs (AutomationTriggerConsumerJob, AutomationActionExecutorJob, AuditLogConsumerJob, OutboundEmailConsumerJob), RabbitMqDeadLetterQueueAdmin.cs, GlobalExceptionHandler.cs (both copies), Api/Program.cs/Api.External/Program.cs/Workers/Program.cs (the AddOpenTelemetry()/UseAzureMonitor() registration blocks).