Apache Kafka is one of the most reliable event-streaming platforms available, yet a surprising number of production deployments struggle within months of going live. The problem is rarely Kafka itself - it's how teams configure, monitor, and scale it once real traffic hits.
Below are the failure patterns that show up most often, and what actually fixes them.
1. Under-Provisioned Partitions
Partition count is one of the most consequential decisions in a Kafka cluster, and it's usually made too early, with too little data. Teams set partition counts based on initial load estimates and never revisit them as throughput grows. The result: consumer lag that creeps up silently until a dashboard alert turns into an incident.
Fix: Partition counts should be reviewed quarterly against actual throughput, not set once and forgotten. Over-partitioning has costs too (more file handles, slower leader elections), so this is a balancing act, not a "more is better" problem.
2. Consumer Group Rebalancing Storms
When consumers in a group restart frequently — due to deploys, crashes, or aggressive autoscaling — Kafka triggers a rebalance every time. In a group with many consumers, this can pause processing for seconds or longer, repeatedly, during peak hours.
Fix: Moving to cooperative sticky rebalancing (available since Kafka 2.4) dramatically reduces the blast radius of each rebalance. Pairing this with longer session timeouts and graceful shutdown handling in consumer code prevents most storms before they start.
3. Broker Configuration Left at Defaults
Default broker settings are designed for compatibility, not performance. Default retention policies, default replication factor, and default min. insync.Replica settings often don't match what a production workload actually needs — until a broker goes down and the team discovers, in the middle of an incident, that durability guarantees weren't what they assumed.
Fix: Replication factor and min.insync.replicas should be explicitly set based on the durability the business actually requires, not left to chance. This is a five-minute conversation that prevents multi-hour incidents.
4. No Real Observability Into Lag and Throughput
Many teams monitor whether Kafka is "up," but not whether it's healthy. Consumer lag, under-replicated partitions, and request queue time are the metrics that actually predict incidents — and they're frequently missing from dashboards until after the first outage.
Fix: Lag and replication health need to be first-class metrics with alerting thresholds, not an afterthought bolted on after something breaks.
5. Schema Changes Without a Compatibility Strategy
As producers and consumers evolve independently, schema drift becomes inevitable. Without a schema registry and compatibility rules, a single upstream change can silently corrupt downstream consumers for days before anyone notices.
Fix: A schema registry with backward-compatibility enforcement turns this from a recurring fire drill into a non-event.
The Common Thread
None of these issues is exotic. They're operational gaps — the kind that show up only after a cluster has been running real traffic for a while, and that internal teams often don't have the bandwidth to catch proactively while also shipping features.
This is exactly the gap that managed Kafka support exists to close: ongoing tuning, proactive monitoring, and incident response handled by people who've already seen these failure patterns play out elsewhere. If your Kafka cluster is showing any of the symptoms above, it's worth getting a second set of eyes on the configuration before the next incident forces the conversation.
Sign in to leave a comment.