Building Systems That Fail Gracefully
Distributed systems fail. Not "might fail" — they will fail. Networks partition, services crash, databases go offline, and deploys introduce bugs. The question isn't whether failures happen but how your system behaves when they do.
Three patterns come up repeatedly when building resilient distributed services: circuit breakers, retry strategies, and the saga pattern for distributed transactions. Getting these right is the difference between a 2 AM page and a smooth (to the user) degradation.
Circuit Breakers: Stop Hitting a Dead Service
Imagine your order service calls the payment service, which is down. Without a circuit breaker, every request to your order service makes a call to payment, waits for the timeout (say 30 seconds), and fails. Your order service is now effectively down too — all its threads are blocked waiting on a dead service. That's cascading failure.
A circuit breaker tracks recent failures and "trips open" when failures exceed a threshold. While open, calls fail immediately without even attempting the downstream request. After a cooldown period, the breaker enters a "half-open" state and allows a test request through. If it succeeds, the breaker closes. If it fails, it stays open.
Implementation
# Circuit breaker implementation
import time
from enum import Enum
class State(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Failing fast
HALF_OPEN = "half_open" # Testing recovery
class CircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=30,
half_open_max_calls=3):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.half_open_max_calls = half_open_max_calls
self.state = State.CLOSED
self.failure_count = 0
self.last_failure_time = 0
self.half_open_calls = 0
def call(self, func, *args, **kwargs):
if self.state == State.OPEN:
if time.time() - self.last_failure_time > self.recovery_timeout:
self.state = State.HALF_OPEN
self.half_open_calls = 0
else:
raise CircuitOpenError("Circuit is open")
if self.state == State.HALF_OPEN:
if self.half_open_calls >= self.half_open_max_calls:
raise CircuitOpenError("Half-open limit reached")
self.half_open_calls += 1
try:
result = func(*args, **kwargs)
self._on_success()
return result
except Exception as e:
self._on_failure()
raise
def _on_success(self):
if self.state == State.HALF_OPEN:
self.state = State.CLOSED
self.failure_count = 0
def _on_failure(self):
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = State.OPENTuning Parameters
The failure threshold depends on your traffic volume. For a service handling 1000 requests/second, 5 consecutive failures might be noise. You'd want something like "50% failure rate over 10 seconds" instead of a simple counter.
Recovery timeout should match how long your downstream service typically takes to recover. If the payment service usually bounces back in under a minute after a deploy, 30 seconds is reasonable. If it requires manual intervention, you might want 5 minutes.
In production, I'd recommend using a library rather than rolling your own. Resilience4j for Java, Polly for .NET, and pybreaker for Python are all battle-tested. For Go, sony/gobreaker is the standard choice.
Retry Strategies: When and How to Try Again
Not all failures are permanent. A network blip, a brief database lock, a momentary load spike — these resolve themselves. Retries can turn transient failures into invisible non-events.
But naive retries are dangerous. Retrying immediately, at full speed, against an already-overloaded service makes things worse. You'll create a retry storm that prevents recovery.
Exponential Backoff with Jitter
The standard approach: wait longer between each retry attempt, with randomization to prevent thundering herd.
import random
import time
def retry_with_backoff(func, max_retries=5, base_delay=0.5):
for attempt in range(max_retries):
try:
return func()
except TransientError:
if attempt == max_retries - 1:
raise
# Exponential backoff with full jitter
delay = base_delay * (2 ** attempt)
jitter = random.uniform(0, delay)
time.sleep(jitter)The jitter is critical. Without it, if 1000 clients all fail at the same time, they'll all retry at 1s, then all at 2s, then all at 4s — synchronized waves hammering the recovering service. Full jitter spreads retries uniformly across the backoff window.
What to Retry
Only retry on transient errors. HTTP 503 (Service Unavailable), 429 (Too Many Requests), network timeouts, connection refused — these are retry candidates. HTTP 400 (Bad Request) or 404 (Not Found) won't magically start succeeding on the next attempt.
And set a retry budget. I've seen systems where a single user request triggers a chain of retries across five services, turning one failure into 3^5 = 243 downstream calls. Limit total retry attempts per request chain, not just per hop.
Saga Pattern: Distributed Transactions Without Two-Phase Commit
In a monolith with a single database, you'd wrap a multi-step operation in a transaction. Either everything commits or everything rolls back. Clean.
In a microservices world, your "transaction" spans multiple databases owned by different services. Two-phase commit (2PC) technically works across databases but it's slow, brittle, and most modern services don't support it. The saga pattern is the practical alternative.
How Sagas Work
A saga breaks a distributed transaction into a sequence of local transactions. Each service performs its local transaction and publishes an event. If any step fails, compensating transactions undo the preceding steps.
Take an order workflow: (1) Create order, (2) Reserve inventory, (3) Process payment, (4) Confirm order. If payment fails at step 3, you need to release the inventory (compensate step 2) and cancel the order (compensate step 1).
Orchestration vs Choreography
Choreography: each service listens for events and decides what to do next. The order service publishes OrderCreated; the inventory service hears it and publishes InventoryReserved; the payment service hears that and publishes PaymentProcessed. No central coordinator.
Pros: simple, decoupled. Cons: hard to track the overall workflow state, difficult to debug, compensating transactions are scattered across services.
Orchestration: a central saga orchestrator tells each service what to do. It maintains the saga state machine and handles failures by invoking compensating transactions in reverse order.
Pros: workflow logic in one place, easier to understand and debug. Cons: the orchestrator is a potential bottleneck and single point of failure.
I'd argue orchestration is the better default for most teams. Yes, it adds a central component. But being able to look at one service and see the entire workflow — including failure handling — is worth the tradeoff. With choreography, understanding the full workflow requires reading code across every participating service.
Compensating Transactions Aren't Always Easy
Here's the uncomfortable truth about sagas: compensation isn't always straightforward. You can refund a payment, but can you un-send an email? Can you un-ship a package? Some actions are irreversible.
For irreversible steps, order your saga to place them last. Process payment before sending the shipping notification. If payment fails, you haven't notified anyone yet. If the notification fails after payment succeeds, that's a less critical failure — you can retry the notification separately.
Also design compensations to be idempotent. A compensation might execute multiple times due to retries, and applying a refund twice would be a very bad day for your accounting team.