When Events Beat Request-Response
Most systems start with synchronous request-response. Service A calls Service B, waits for a response, processes it, returns to the caller. It's straightforward and works well for simple architectures. But as you add more services that need to react to the same event, the request-response model starts to crack.
Picture an e-commerce order placement. The order service needs to: update inventory, process payment, send a confirmation email, notify the warehouse, update analytics, and trigger loyalty point calculation. With synchronous calls, the order service knows about — and directly calls — all six downstream services. That's tight coupling, and it'll make your on-call rotation miserable.
Event-driven architecture flips this around. The order service publishes an "OrderPlaced" event. Downstream services subscribe to events they care about and react independently. The order service doesn't know or care who's listening.
Kafka: The Distributed Commit Log
Apache Kafka isn't really a message queue, though people use it as one. It's a distributed, append-only commit log. That distinction matters.
How Kafka Works Internally
Messages go into topics, and topics are split into partitions. Each partition is an ordered, immutable sequence of records. Producers append to the end; consumers read from any position.
Here's what makes Kafka different from traditional message queues: messages aren't deleted after consumption. They stick around for a configurable retention period (default 7 days, but you can set it to forever). Multiple consumer groups can read the same topic independently, each tracking their own offset.
# Kafka producer example (Python, confluent-kafka 2.3)
from confluent_kafka import Producer
conf = {
'bootstrap.servers': 'kafka-1:9092,kafka-2:9092',
'acks': 'all', # Wait for all replicas
'retries': 3,
'linger.ms': 5, # Batch messages for 5ms
}
producer = Producer(conf)
def on_delivery(err, msg):
if err:
print(f"Delivery failed: {err}")
producer.produce(
topic='orders',
key=str(order_id).encode(),
value=json.dumps(order_data).encode(),
callback=on_delivery
)
producer.flush()Partition Strategy
Partition count determines your parallelism ceiling. You can't have more active consumers in a group than partitions. Got 12 partitions? Maximum 12 concurrent consumers in one group.
Partition key determines which partition a message lands in. Messages with the same key always go to the same partition, which guarantees ordering within that key. For order events, using customer_id as the partition key ensures all events for one customer are processed in order.
I'd suggest starting with 2-3x your current consumer count as your partition number. Increasing partitions later is possible but can break key-based ordering guarantees during the transition.
When Kafka Shines
Kafka's sweet spot is high-throughput event streaming. A 3-node Kafka cluster can handle 200,000+ messages per second with proper tuning. LinkedIn runs Kafka at 7 trillion messages per day across their infrastructure.
It's ideal for: event sourcing, change data capture (CDC), stream processing pipelines, activity tracking, and anything where you need to replay events from a specific point in time.
RabbitMQ: The Traditional Message Broker
RabbitMQ follows the AMQP protocol and behaves like a traditional message broker. Producers send messages to exchanges, exchanges route to queues based on bindings, and consumers pull from queues.
Exchange Types
Direct exchange routes by exact routing key match. Message with key "order.created" goes to queues bound with "order.created". Simple point-to-point routing.
Fanout exchange broadcasts to all bound queues regardless of routing key. Useful when every subscriber needs every message — like sending the same event to logging, analytics, and notification services.
Topic exchange routes by pattern matching. A queue bound with "order.*" gets both "order.created" and "order.cancelled". A queue bound with "#.error" gets "payment.error" and "shipping.error". This flexibility is one of RabbitMQ's strongest features.
# RabbitMQ consumer with acknowledgments (pika 1.3)
import pika
connection = pika.BlockingConnection(
pika.ConnectionParameters('rabbitmq-host')
)
channel = connection.channel()
channel.queue_declare(queue='order_processing', durable=True)
channel.basic_qos(prefetch_count=10)
def process_order(ch, method, properties, body):
try:
order = json.loads(body)
handle_order(order)
ch.basic_ack(delivery_tag=method.delivery_tag)
except Exception as e:
# Negative ack — requeue the message
ch.basic_nack(
delivery_tag=method.delivery_tag,
requeue=True
)
channel.basic_consume(
queue='order_processing',
on_message_callback=process_order
)
channel.start_consuming()Where RabbitMQ Wins
RabbitMQ excels at task distribution and complex routing. Its message acknowledgment system is battle-tested — consumers explicitly ack or nack messages, and unacknowledged messages get redelivered. Dead letter exchanges handle messages that fail processing repeatedly.
Priority queues, message TTLs, delayed message delivery — RabbitMQ supports all of these out of the box. Kafka requires workarounds for most of them.
For throughput, RabbitMQ handles around 20,000-50,000 messages per second on a single node. That's 4-10x less than Kafka, but still more than enough for many applications.
Kafka vs RabbitMQ: The Decision Framework
This isn't a "Kafka is better" or "RabbitMQ is better" discussion. They solve different problems.
Choose Kafka when: you need event replay, multiple independent consumers reading the same stream, very high throughput (100K+ msg/sec), stream processing with Kafka Streams or Flink, or you're building an event sourcing system.
Choose RabbitMQ when: you need complex routing logic, priority queues, message-level TTLs, traditional task queues with reliable acknowledgment, or your throughput needs are moderate. RabbitMQ is also simpler to operate — a 3-node cluster with quorum queues covers most use cases.
Choose neither when: you've got two services talking to each other and could just use a direct HTTP call or gRPC. Adding a message broker for simple point-to-point communication is over-engineering. You're adding operational complexity (another system to monitor, upgrade, and debug) for marginal architectural benefit.
Common Pitfalls
The "Everything Is an Event" Trap
Once teams adopt event-driven architecture, there's a tendency to make everything async. Don't. If a user is waiting for a response and you can provide it synchronously in 50ms, making it async adds complexity without benefit. Reserve events for cases where decoupling, fan-out, or async processing genuinely helps.
Event Schema Evolution
Your event schemas will change. A field gets added, a format changes, a field becomes required. Without schema management, you'll break consumers silently.
Use a schema registry (Confluent Schema Registry for Kafka, or a custom one for RabbitMQ). Define schemas with Avro or Protobuf, enforce backward compatibility, and version your events. It feels like overhead at first. It saves you at 3 AM when a breaking schema change would've taken down three services.
Idempotency
Messages can be delivered more than once. Network blip during acknowledgment? The message gets redelivered. Consumer crashes mid-processing? Redelivered. Your consumers must handle duplicate messages gracefully.
The simplest approach: track processed message IDs in your database and skip duplicates. For Kafka, you can store the consumer offset alongside your business data in a single transaction. That guarantees exactly-once processing semantics at the application level.