Skip to main content

Outbound integration resilience: Polly retry/circuit-breaker wrapper

Every outbound HTTP call to an external partner/integration goes through one shared Polly-backed policy rather than a hand-rolled retry loop per integration. Built 2026-08-17.

OutboundIntegrationResilience.AddOutboundIntegrationResilience()​

The one extension method every current and future outbound-integration HttpClient registration chains onto (Infrastructure/Http/ OutboundIntegrationResilience.cs). Wraps Microsoft.Extensions.Http. Resilience's own AddStandardResilienceHandler() (rate limiter → total-request timeout → retry → circuit breaker → per-attempt timeout) rather than a hand-rolled Polly v8 pipeline, tuning only the circuit breaker (FailureRatio=0.5, MinimumThroughput=5, SamplingDuration/ BreakDuration=30s) down from the library's high-traffic-assuming defaults to something a low-volume client (occasional SSO logins, occasional webhook fires) can actually trip.

:::caution Circuit isolation is per client name, not per real-world partner Each named/typed client gets its own independent circuit breaker, matching IHttpClientFactory's per-client handler lifetime — but "AutomationWebhook" fires to a different URL per Scenario and they all share one circuit breaker, so a failing partner can trip every other, healthy partner behind that same name for the 30s break duration. Accepted for today's low webhook volume / single live SSO provider — revisit (key per-target-host instead) if concurrent multi-partner volume becomes real. :::

Current call sites​

OidcIdentityProvider's typed client (SSO discovery/token exchange) and the "AutomationWebhook" named client (CallWebhookAutomationAction's partner POSTs) — both registered in Infrastructure/DependencyInjection.cs. Add a new outbound integration by chaining this same extension; never re-implement retry/circuit-breaker logic per integration.

GraphEmailTransportSender's Microsoft Graph SDK calls are deliberately excluded — the Graph SDK manages its own retry/backoff internally, and wrapping it in a second, independent resilience layer would double-retry rather than add coverage.

Idempotency, deliberately not solved here​

The default retry predicate retries POSTs too (both current integrations are POST) — accepted as the same at-least-once shape as this codebase's other documented at-least-once windows (inbound-email redelivery, automation-engine ack timing). See Idempotency-Key Pattern for the matching guard on the receiving side. Do not add DisableForUnsafeHttpMethods() here without reconsidering this tradeoff first.

:::note A retired DI-ordering landmine AddStandardResilienceHandler() registers a keyed ServiceDescriptor that used to silently break the classic Microsoft.ApplicationInsights 2.22.0 SDK's own service-collection scan unless telemetry registration ran before AddInfrastructure(). The constraint no longer applies — the 2026-08-25 migration to the Azure Monitor OpenTelemetry Distro (see Distributed Tracing) retired the 2.x SDK entirely. Worth recognizing the failure mode (WebApplicationFactory tests dying with "No service for type 'X' has been registered", silent everywhere else) if a similarly narrow DI-ordering constraint shows up again with some other library. :::

Reference files​

OutboundIntegrationResilience.cs, OutboundIntegrationResilienceTests.cs, DependencyInjection.cs (OidcIdentityProvider/"AutomationWebhook" registrations).