Building distributed systems that remain maintainable after the third year is a different discipline than shipping the first version. In my experience, the teams that survive that transition are the ones who get ruthlessly precise about the language they use to describe change. Commands, events, and sagas are not interchangeable messaging patterns — they encode fundamentally different semantics about intent, fact, and time. Confusing them leads to subtle coupling, unreliable workflows, and event logs that cannot be trusted for replay. This article distills what I've learned implementing event-driven systems across e-commerce, logistics, and financial domains, with the concrete trade-offs I wish someone had shown me earlier.
Commands as Intent: Capturing What We Want to Happen
A command is a request to perform an action. It is imperative, it can be rejected, and it should have exactly one logical handler. If you take one idea from this section, let it be this: commands model intent in the present tense, not history. PlaceOrder, ReserveInventory, ProcessPayment — each expresses what the sender wants the receiver to do. Because they represent intent, commands can fail validation, fail authorization, and fail due to business invariants.
I've found that the most common failure in command design is making them too generic or too anemic. A command like UpdateOrder with a loose payload tells the handler nothing about the business operation. Instead, model the specific use case: ChangeShippingAddress, ApplyDiscountCode, CancelOrder. This forces you to encode the invariant that belongs to that operation directly in the handler, rather than branching on flags inside a single endpoint.
Anatomy of a Well-Formed Command
A good command carries just enough context to be validated without querying back to the caller. It should include a correlation ID, a causation ID, an actor/principal, and a deterministic idempotency key. The payload itself should be validated structurally at the edge (schema, types, required fields) and semantically inside the aggregate or handler (invariants, authorization). I enforce a team rule: commands are never published to a pub/sub fan-out. They are sent point-to-point to a single consumer, often via a queue with competing consumers where only one instance will process it. This constraint makes the single-handler guarantee explicit in your infrastructure.
// Command: explicit intent with idempotency and actor context
type PlaceOrderCommand = {
commandId: string; // uuid v7 for idempotency
correlationId: string;
actorId: string;
payload: {
customerId: string;
cartId: string;
shippingAddress: Address;
paymentMethodId: string;
};
metadata: {
occurredAt: string; // ISO-8601
source: "web-checkout";
}
};
// Handler: single owner, explicit rejection
async function handlePlaceOrder(cmd: PlaceOrderCommand): Promise<Result<OrderId, DomainError>> {
const cart = await cartRepository.load(cmd.payload.cartId);
if (!cart.isOwnedBy(cmd.payload.customerId)) {
return err(new AuthorizationError("Cart does not belong to actor"));
}
if (cart.isEmpty()) {
return err(new ValidationError("Cannot place order with empty cart"));
}
const order = Order.createFromCart(cart, cmd.payload.shippingAddress);
// Business invariant enforced inside the aggregate
const result = order.place(cmd.commandId);
if (result.isErr()) return result;
await orderRepository.save(order); // Outbox will publish OrderPlaced event
return ok(order.id);
}
This separation matters for system evolution. When we kept commands as thin DTOs routed through a generic CommandBus that allowed multiple subscribers, we recreated RPC with extra latency. When we constrained each command type to one handling service — the aggregate owner — our bounded contexts became genuinely autonomous. The order context owns PlaceOrder; inventory owns ReserveInventory. No other service handles them. If you need to fan out side-effects, you publish an event after the command succeeds.
Validation Boundaries and Rejection
Distinguish structural validation (is the payload well-formed?) from domain validation (is this operation allowed right now?). Structural validation belongs at the API gateway or command ingress and should return 400 Bad Request with a standard problem detail — see IETF RFC Standards RFC 9457 for a useful format. Domain validation belongs inside the aggregate and results in a domain rejection, not an exception. In my current systems, rejections are returned as explicit result types and, if needed, published as OrderPlacementRejected events for observability and user feedback, but they are not retried as failures.
Domain Events as Facts: Modeling What Has Already Happened
If commands are intent, events are facts. An event states that something has happened in the past: OrderPlaced, InventoryReserved, PaymentFailed. Once published, an event is immutable. You cannot un-publish it; you can only publish a compensating event like OrderCancelled or PaymentRefunded. This immutability is what makes events reliable for audit, replay, and temporal queries. In my experience, teams that treat events as mutable notifications inevitably lose the ability to rebuild state.
Two distinctions have saved us significant rework: domain events versus integration events, and fat versus thin events. Domain events stay inside the bounded context and can be rich, containing full state transitions relevant to that aggregate. Integration events cross context boundaries and should be stable, versioned contracts with minimal, intention-revealing payloads. Publishing your internal domain event directly to other teams couples you to your internal model.
Designing Events for Evolution
Events must be versioned. I use a simple convention: order.placed.v2 as the topic/type, with a schema registry enforcing backward compatibility. New consumers read v2; old consumers continue reading v1 via an upcaster that transforms the stored event on read. Payloads include eventId, aggregateId, aggregateVersion, eventType, occurredAt, and correlationId. I avoid putting derived state in events — e.g., don't put currentOrderTotal if it can be computed from the stream — because derived values drift when logic changes.
For transport, we often use an append-only log (Kafka, EventStoreDB, Kinesis) rather than a broker with destructive consumption. This choice enables replay. A practical nuance: browsers and edge services that need to observe events can use Server-Sent Events as a lightweight push mechanism, as documented in MDN Web Docs for Server-sent events, rather than polling a projection. It keeps the read path simple without exposing the log directly.
# Event: immutable fact with versioned schema
{
"eventId": "evt_2Xy9a8QpL1mN",
"eventType": "order.placed.v2",
"aggregateId": "ord_8f3b1a",
"aggregateVersion": 1,
"correlationId": "corr_9f1c2e",
"causationId": "cmd_a7b3c9",
"occurredAt": "2025-11-14T09:12:31.142Z",
"actorId": "cust_41b8e",
"payload": {
"customerId": "cust_41b8e",
"cartId": "cart_7c2a9",
"placedTotal": { "amount": "14900", "currency": "USD" },
"shippingAddress": { "city": "Berlin", "country": "DE" },
"lineItems": [
{ "sku": "MON-W-42", "qty": 1, "price": "14900" }
]
},
"metadata": {
"source": "order-service",
"schemaVersion": 2
}
}
Thin versus fat is a deliberate trade-off. Fat events (containing full payload) let consumers act without a callback, reducing coupling at read time but increasing message size and schema surface. Thin events (containing only IDs) keep messages small and hide internal structure but force consumers to query back, which reintroduces temporal coupling and failure modes. For cross-context integration events, I default to moderately thin: IDs plus a few stable attributes needed for routing and filtering, with a documented fetch API for enrichment. Inside a context, I prefer richer domain events because the team controls both sides.
Event Sourcing and CQRS: Persisting History Instead of State
Event sourcing persists the sequence of events that led to the current state rather than the state itself. The aggregate's state is a left-fold over its event stream. CQRS — Command Query Responsibility Segregation — is often paired with it, but it is a separate principle: the model for writes is distinct from the model for reads. You can do CQRS without event sourcing (different tables for read and write), and you can do event sourcing without CQRS (rebuilding the same model), but together they enable powerful consistency and scalability properties.
When I introduced event sourcing to a logistics service that tracked shipment lifecycle, the immediate win was not performance but diagnostics. We could replay any shipment's history to the exact point where a business rule was violated, and we could introduce a new projection for a new UI months later by reprocessing the log from the beginning. The cost was complexity in versioning and in educating the team that the event store is the source of truth, not the read projection. Treating the projection as disposable is the mental shift that matters.
The Write Side and the Read Side
On the write side, the aggregate loads its event stream, rehydrates state, validates the command against that state, and emits new events. The store operation is an atomic append with optimistic concurrency: WHERE aggregateVersion = expectedVersion. If two writers race, one fails and retries by reloading. This gives us strong consistency within an aggregate boundary without distributed locks. On the read side, asynchronous projectors subscribe to the log and materialize views optimized for queries — e.g., a denormalized order_summary table in Postgres, or an Elasticsearch index for search. Those views are eventually consistent.
The key design decision is aggregate granularity. Small aggregates reduce contention but make cross-aggregate invariants harder. Large aggregates enforce invariants naturally but limit throughput due to serial appends. I've found that aligning aggregates with transactional consistency needs — not with UI screens — yields the right size. If two pieces of data must be consistent immediately, they probably belong in the same aggregate. If eventual consistency is acceptable, split them.
| Concept | Semantics | Cardinality | Mutability & Retry |
|---|---|---|---|
| Command | Request / intent to do something; can be rejected | One sender → one handler (point-to-point) | Mutable intent; retried only if infrastructure failed, not on business rejection |
| Domain Event | Fact that something has happened; immutable history | One publisher → many subscribers (pub/sub) | Immutable; never retried for business logic, replayed for projections |
| Query (CQRS Read) | Request for data; no side-effects, no state change | One requester → one read model / projection | Cacheable and repeatable; safe to retry and scale horizontally |
This table is worth pinning to your architecture decision record. When a developer proposes to "publish a command to multiple services" or to "retry an event until it succeeds," the table makes the semantic violation obvious. Commands are directed and rejectable; events are broadcast facts; queries are side-effect free. Mixing those guarantees is how you end up with out-of-order state and phantom orders.
A housekeeping note that pays off operationally: pair your event store with the transactional outbox pattern. Inside the same database transaction that appends events, write the integration events to an outbox table. A relay process publishes from the outbox to the broker. This gives you atomicity between state change and publication without two-phase commit, and it is the simplest path to at-least-once delivery without losing events on crash. How you layer this outbox and the aggregate persistence fits naturally within a Clean Architecture approach where infrastructure details stay at the edges.
Orchestrating Long-Running Workflows with Sagas
Once you split a business process across services, you lose the ability to wrap it in a single ACID transaction. A customer placing an order needs inventory reserved, payment captured, and shipment scheduled — three services, three failure modes. The saga pattern manages this by coordinating a sequence of local transactions with compensating actions. There are two styles: choreography and orchestration. I've used both, and I now strongly prefer orchestration for any workflow with branching or required compensation.
In choreography, each service reacts to events and publishes the next event. OrderPlaced triggers inventory to emit InventoryReserved, which triggers payment to emit PaymentCaptured. It looks decoupled on a whiteboard, but in production it becomes invisible coupling: the workflow is scattered across four codebases, timeouts are hard to observe, and adding a new step requires touching every participant's event handling. Debugging a stuck order means tracing events across logs with no central view.
Orchestration: A Dedicated Saga Coordinator
Orchestration centralizes control in a saga manager that holds the workflow state machine, sends commands, and reacts to events. The manager persists its own state as an event-sourced entity, so it can recover after a crash and handle duplicates safely. When a step fails, the manager executes compensating commands in reverse order: if payment fails after inventory was reserved, it sends ReleaseInventory. Critically, compensations must be idempotent and must themselves be able to fail and be retried — you need a strategy for poison compensations, typically a manual intervention queue.
# Saga orchestrator: explicit state machine with compensation
class OrderFulfillmentSaga:
def __init__(self, saga_id, order_id):
self.saga_id = saga_id
self.order_id = order_id
self.state = "STARTED"
self.steps_completed = []
async def handle(self, event):
if event.type == "order.placed.v2" and self.state == "STARTED":
self.state = "RESERVING_INVENTORY"
await command_bus.send(ReserveInventory(
saga_id=self.saga_id, order_id=self.order_id,
items=event.payload["lineItems"],
idempotency_key=f"{self.saga_id}:reserve"
))
elif event.type == "inventory.reserved.v1" and self.state == "RESERVING_INVENTORY":
self.steps_completed.append("inventory")
self.state = "CAPTURING_PAYMENT"
await command_bus.send(CapturePayment(
saga_id=self.saga_id, order_id=self.order_id,
amount=event.payload["reservedTotal"],
idempotency_key=f"{self.saga_id}:capture"
))
elif event.type == "payment.captured.v1" and self.state == "CAPTURING_PAYMENT":
self.steps_completed.append("payment")
self.state = "COMPLETED"
await command_bus.send(ScheduleShipment(
saga_id=self.saga_id, order_id=self.order_id
))
elif event.type in ("inventory.out_of_stock.v1", "payment.failed.v1"):
await self.compensate(trigger=event.type)
async def compensate(self, trigger):
self.state = "COMPENSATING"
# Compensate in reverse order, idempotently
if "payment" in self.steps_completed:
await command_bus.send(RefundPayment(
saga_id=self.saga_id, order_id=self.order_id,
idempotency_key=f"{self.saga_id}:refund:{trigger}"
))
if "inventory" in self.steps_completed:
await command_bus.send(ReleaseInventory(
saga_id=self.saga_id, order_id=self.order_id,
idempotency_key=f"{self.saga_id}:release:{trigger}"
))
self.state = "COMPENSATED"
A production-grade orchestrator also needs timers. If InventoryReserved never arrives, the saga must time out and compensate rather than hang forever. I model timers as durable delayed commands: the orchestrator schedules a SagaTimeout command for T+30s when it enters a waiting state, and cancels it if the expected event arrives. This makes timeouts first-class and observable, rather than relying on broker visibility settings that are invisible to the domain.
When choosing between choreography and orchestration, my heuristic is simple: if the workflow has fewer than three steps, no branching, and no compensation, choreography is tolerable. If you need visibility, auditability, or compensations — which is most real-world order-to-cash, booking, or provisioning flows — use an orchestrator. The extra service is worth the clarity. This aligns with the broader decision framework for when distributed patterns are justified versus when a modular monolith with local transactions is simpler.
Delivery Guarantees, Idempotency, and Failure Modes in the Event Bus
Every serious incident I've seen in event-driven systems traced back to an assumption about delivery. Brokers do not provide exactly-once delivery end-to-end in a way you can depend on for business correctness. Expect at-least-once delivery at the transport layer, with duplicates due to retries, rebalances, and producer retries. Design your handlers to be idempotent, and your system will be stable; assume exactly-once and you will double-charge customers.
Idempotency starts with keys. For commands, the commandId is the idempotency key; the handler checks a processed_commands table with a unique constraint before applying side effects. For events, the eventId serves the same role in projectors: INSERT ... ON CONFLICT DO NOTHING or a conditional write on version. Also make your side effects idempotent where possible: setting a status to RESERVED is idempotent; incrementing a counter is not unless you guard it.
The Outbox, Inbox, and Dead Letters
On the producer side, the transactional outbox (mentioned earlier) solves the dual-write problem. On the consumer side, the inbox pattern solves duplicate processing: before handling an event, insert its ID into an inbox table within the same transaction that updates your projection. If the ID already exists, skip. This makes consumption effectively-once from the business perspective.
For poison messages — events that repeatedly fail to process due to a bug or schema mismatch — never block the partition indefinitely. After N retries with exponential backoff and jitter, route the event to a dead-letter topic with full headers and payload. Alert on dead-letter growth, and provide a replay tool that can reprocess after a fix, preserving ordering per aggregate key. I also insist on end-to-end observability: propagate correlationId and causationId through every command and event so a single trace can reconstruct the saga path in your observability platform.
Ordering is another nuance. Most logs guarantee ordering per partition key, typically aggregateId. If you need global ordering, you do not need a total order — you need causal ordering within the aggregates that participate in a saga. Partition by aggregateId and ensure your saga manager correctly handles out-of-order delivery between aggregates (e.g., payment succeeded arrives before inventory confirmation) by modeling state transitions that ignore or buffer unexpected events rather than crashing.
When Event-Driven Adds Value and When It Becomes Liability
Event-driven architecture shines when you have multiple readers with divergent query needs, temporal queries or audit requirements, and workflows that span teams with independent deploy cycles. In those settings, events decouple the write model from the read models and let teams evolve projections without coordinating deploys. Our analytics team could add a new orders_by_region projection without ever touching the order service, simply by replaying from the log.
It becomes a liability when applied as a default. If your domain has simple CRUD with no cross-service workflows and a single database comfortably handles load, an event log adds operational overhead: schema registry, broker clusters, outbox relay, projector lag monitoring, replay tooling, and versioned upcasters. Debugging a missing field becomes a hunt through schemas and projections rather than a single SELECT. I've seen teams adopt event sourcing for a user profile service that would have been a straightforward relational table with soft deletes — the result was slower development and no measurable benefit.
Team and Operational Readiness
Beyond technical fit, evaluate team readiness. Can the team reason about eventual consistency in the UI? Do you have patterns for showing "pending" states and reconciling reads after writes? Do you have runbooks for partition lag, consumer rebalances, and schema evolution? A useful exercise before committing is to map your bounded contexts and ask where you actually need decoupling. Many domains are better served by a well-modularized monolith with explicit internal events (in-process) and only a few well-chosen asynchronous boundaries. The patterns in modern design patterns for messaging and decoupling are valuable even inside a monolith — you do not need Kafka to benefit from command-handler separation.
If you do proceed, adopt incremental discipline: start with asynchronous projections for one read model while keeping writes on a relational table, then graduate that single aggregate to event sourcing once projection replay is routine. Verify your backup and restore story for the log — it is now your source of truth. And measure projector lag as a service-level indicator; when lag grows, user-visible inconsistency grows with it. In my experience, a week of investment in replay tooling, idempotency tests, and dead-letter reprocessing pays back months of incident time later.
Frequently Asked Questions
Should every microservice be event-sourced from day one?
No. Event sourcing is a persistence strategy with real costs: schema evolution, upcasters, replay tooling, and eventual consistency on reads. Start with CQRS at the application layer — separate command handlers from query handlers — and keep a relational write model. Introduce event sourcing only for aggregates where you need auditability, temporal queries, or flexible projections. I've found that one or two aggregates benefit strongly (order, shipment) while many others are best left as standard entities.
What is the practical difference between choreography and orchestration for sagas?
Choreography relies on each service reacting to events and publishing the next event, with no central coordinator. Orchestration uses a dedicated saga manager that sends commands and tracks state. Choreography has lower initial coordination but makes workflows invisible and compensation hard to reason about. Orchestration adds a service but gives you explicit state, timeouts, retries, and a single place to observe and compensate failures. For any workflow with compensation or branching, I recommend orchestration.
How do you handle event versioning without breaking consumers?
Version the event type (e.g., order.placed.v2) and enforce backward compatibility via a schema registry. New fields should be optional with sensible defaults; renames should be additive with deprecation. Store events immutably in v1 form and apply upcasters on read to transform old events to the latest shape for consumers that need it. Provide a replay path so consumers can reprocess the log after upgrading their projection logic, and test version transitions with contract tests.
How do you prevent duplicate processing when the broker delivers at-least-once?
Make handlers idempotent using an inbox or processed-set table. For commands, use a unique constraint on commandId; for events, use eventId. Perform the duplicate check inside the same transaction that applies the side effect. Pair this with the transactional outbox on the producer side so you never lose or double-publish due to crashes. Compensating actions and payment operations should also be idempotent via idempotency keys, so retries do not create double refunds or double charges.