A Low-Downtime Kafka Migration

Some Kafka migrations are mechanical — add a broker, reassign partitions, remove the old one. This writeup is about the kind that isn't, because the nodes you're replacing are the ones holding the consensus quorum, and the quorum doesn't want to change.

The problem

A self-managed KRaft cluster needs to be replaced. Not restarted, not reconfigured — replaced. The brokers are also the controllers, and the storage they sit on can't be modified in place. New nodes, new volumes, same cluster.

In the ZooKeeper era, this is a partition-reassignment exercise. In KRaft, the brokers are the metadata quorum. Replacing them means replacing the voters in a Raft consensus group, and that's a fundamentally different operation than moving partition data.

Why you can't just roll through it

KRaft in static voter mode hardcodes the voter list into every node's configuration. All voters must agree on who the voters are. Changing that list one node at a time sounds safe and isn't.

The arithmetic: three voters, majority is two. You want to add a fourth, making majority three. Restart the first node with the new four-voter list. During its restart, two nodes are alive under the new math — but the new list needs three for quorum. Meanwhile the other two still have the old three-voter list, still see two of three alive, and keep going. Two groups, two different ideas of who's in charge. That's split-brain, and the protocol doesn't resolve it.

There is no safe way to incrementally change a static voter list. The only option is a coordinated stop: shut down all controllers, update all configs, start them together. That's a full cluster outage by definition.

The question becomes: how small can you make it?

The observer trick

The insight that makes this tractable: brokers already have the metadata.

KRaft's cluster state lives in a single internal topic. Controllers replicate it via Raft. But brokers — even ones that aren't voters — tail this topic as observers, maintaining a local copy of the metadata log for their own use.

Stand up the new nodes as broker-only first. They join the existing cluster, start observing the metadata log, and accept partition replicas as you reassign data to them. By the time you're ready to promote them to controllers, they already have the full metadata state on disk. Promotion is a config change, not a data transfer.

This turns the scary part — forming a new quorum — into the fast part. The slow part is the data reassignment, and that runs with zero downtime.

Making the outage small

The migration has three phases, and only the last one requires downtime.

Move the data first. Partition reassignment runs live, driven by the existing controllers. Replicas move from old brokers to new ones concurrently with normal traffic. This takes as long as it takes — minutes for small partitions, potentially longer for large ones — but it's invisible to clients.

Stop the old controllers. This is where the outage starts. The cluster is down.

Start the new controllers. They boot with the voter list pointing only at each other, find the metadata log they've been observing for hours (or days), form a new quorum, and elect a leader. The cluster comes back. Writes that failed during the gap retry; consumers reconnect and catch up.

The gap between stopping the old and the new cluster being ready was about three minutes. Brief enough that retrying producers and reconnecting consumers handled it without intervention.

The dynamic voter false start

KRaft has a feature designed for exactly this — KIP-853, dynamic voters. Instead of a hardcoded list, the voter set is stored in metadata and managed at runtime. Add a voter, wait for it to catch up, remove an old one. No coordinated restart, no outage.

We tried it. The upgrade command appeared to succeed but never finalized — the feature version stayed at zero despite the metadata epoch advancing. The likely cause: the cluster was originally formatted without the identity records that dynamic voters need. The transition from static to dynamic mode has undocumented prerequisites that we didn't have time to investigate on a production cluster.

So we took the three-minute hit.

What broke afterward

The cluster itself was fine — all partitions healthy, ISR full, correct voter set. What broke was everything around it that referenced the old nodes by name.

DNS records generated from infrastructure-as-code still pointed at the old instances. Monitoring probes had hardcoded hostnames. A metrics exporter installed manually months earlier had broker addresses baked into its systemd unit. None of these were caught in pre-flight checks because nobody thought to grep every config surface for the old hostnames.

The pattern is worth naming: the blast radius of a cluster migration is wider than the cluster. Every system that references a broker by name is a downstream you have to update, and the dangerous ones are the ones not managed in code.

What it cost

Three minutes of planned outage. Zero data loss. A few hours of follow-up work for the downstream references. Against the alternatives — an unsafe rolling restart, or a week of yak-shaving the dynamic voter upgrade on a production system — the small coordinated outage was the honest answer.

The general lesson: when a system's consensus layer doesn't support incremental membership changes, the right move is often to accept a brief, planned stop rather than engineering around it unsafely. Three minutes is a long time if you're watching the dashboard. It's nothing compared to the incident you'd have from a split-brain.