Most event-driven projects don't fail because Kafka fell over. They fail because a renamed field broke three consumers, a hot partition capped throughput, or a service fell forty minutes behind and nobody noticed until customers did. The pattern itself is proven at absurd scale: LinkedIn's Kafka deployment carries over 7 trillion messages a day and Netflix's Keystone pipeline handles trillions of events daily. The broker is rarely your bottleneck. Your design decisions are.
What is event-driven architecture? Event-driven architecture is a pattern where services communicate by publishing and consuming events, which are immutable records of things that already happened, rather than calling each other directly. A producer writes an event to a broker such as Apache Kafka and any number of consumers react asynchronously, so each side can scale, fail, and deploy independently.
Treat every event as a public API
No single decision shapes the architecture more than what your events look like, because every consumer inherits it. Thin notification events carry a reference and force consumers to call back for details, quietly reintroducing the synchronous coupling you were escaping; event-carried state transfer costs bandwidth but keeps consumers genuinely independent. Martin Fowler's comparison of the two patterns remains the clearest reference on the trade-off.
Whichever style you pick, three rules keep contracts from rotting. Run a schema registry, such as Confluent Schema Registry, with compatibility checks enforced at publish time. Give breaking changes a new major version. And name events as past-tense facts: OrderPlaced records history, while PlaceOrder is a command in disguise that turns your broker into hidden RPC.
Partition keys set your scalability ceiling
Kafka system design reduces to one structural fact: ordering is guaranteed only within a partition, and each partition is read by at most one consumer in a group. Your partition key therefore decides both ordering and maximum parallelism. Key by the identity that must stay ordered, such as order_id for order events. Key by something skewed, like a customer ID dominated by one enterprise account, and you get a hot partition that no amount of extra consumers can drain.
The platform ceiling is rarely the issue, since LinkedIn already runs 7 million partitions across more than 4,000 brokers. Provision more partitions than you currently need, because adding them later reshuffles the key-to-partition mapping and breaks ordering. And never design around global ordering; if two events on different keys must be sequenced, remodel the aggregate rather than fight the broker.
Design for at-least-once, because duplicates will happen
Retries exist, so duplicates will reach somebody. Kafka offers exactly-once semantics through idempotent producers and transactions, but the guarantee ends the moment a side effect leaves Kafka, such as an email or a payment API call. In practice, at-least-once delivery with idempotent consumers is the sensible default, with true end-to-end exactly-once reserved for payments, ledgers, and anything money-shaped. The durable habit is idempotence: record processed event IDs, skip repeats, and prefer upserts to inserts.
The write side has a matching trap. Updating your database and publishing to Kafka as two separate steps works 99.9% of the time, which at a million events a day means a thousand silent inconsistencies. The fix is the transactional outbox: write the event to an outbox table inside the same database transaction as the state change, then relay it to the broker via Debezium.
Watch consumer lag, and treat event sourcing as a power tool
Event systems fail silently: producers keep publishing while a struggling consumer drifts further behind and users see stale data. Consumer lag, the gap between the newest offset and the last committed offset, is the most honest operational metric you have. Alert on its growth rate rather than its absolute value, and route poison messages to a dead-letter queue after bounded retries, because one unparseable event will otherwise block its partition indefinitely. The headroom you are monitoring against is enormous: LinkedIn's deployment grew from 1 billion messages a day in 2011 to 7 trillion by 2019 on the same core design.
Restraint matters at the top of the stack too. Event sourcing, where every state change is stored as an immutable event and state is rebuilt by replay, delivers a complete audit trail, but schema changes then mean migrating history and compliance deletion is hard in an append-only store. Microsoft's architecture guidance is candid that it suits neither simple domains nor small teams. Most organisations do better with event-driven microservices backed by strong contracts and an outbox, reserving event sourcing for aggregates such as a ledger, and delivering real-time updates through WebSockets fed by a filtering consumer rather than exposing Kafka to browsers.
FAQ
When should you not use event-driven architecture?
Skip it when your workflow is fundamentally request-response, or when your team cannot yet operate a broker plus the surrounding observability. The costs only pay off once coupling and scale are genuine problems, and a CRUD application behind a load balancer does not have them.
Is Kafka overkill for small applications?
Often, yes. Kafka earns its keep when you need replayable history, several independent consumers of one stream, or serious throughput headroom. For modest workloads, managed message queues such as SQS or RabbitMQ are simpler to run.
Your rollout checklist
Start with a genuinely asynchronous workflow such as notifications, not your checkout path.
Define contracts first: past-tense names, explicit versions, a schema registry enforced in CI.
Choose partition keys from ordering requirements and test them against real traffic skew.
Implement the transactional outbox, make consumers idempotent, and wire dead-letter queues before the first poison message.
Instrument consumer lag from day one, alert on growth rate, and give every topic a named owning team.
Event systems reward teams that respect four details: contracts, keys, idempotency, and lag. BeyondPixl Studio builds event-driven backbones for startups and enterprises, so if your system is outgrowing request-response, talk to our engineering team about an architecture review.