// HACKER NEWS — CYBERSECURITY
Data liberation: Apache Kafka's native cluster mirroring
Apache Kafka excels at moving data within a cluster. Leaders replicate to followers, consumers pull from any replica, and the entire machinery runs with minimal operational overhead. Moving data between clusters has never been that simple.
Organizations run multiple Kafka clusters for good reasons: Geographic distribution, compliance boundaries, team isolation, version segregation. But once data lands in a cluster, getting a faithful copy into another has always required external tooling, careful coordination, and a healthy tolerance for operational surprises.
KIP-1279 changes this. Cluster mirroring embeds cross-cluster replication directly into the Kafka broker. No external processes, no offset translation tables, no recompression. A destination broker fetches committed records from a source cluster using the same fetch protocol that followers already use, and appends them to local partition logs byte for byte. The result is a mirror that preserves offsets, compression, and consumer group state, making failover as simple as stopping the mirror and redirecting clients.
This article covers the high-level architecture behind cluster mirroring, the state machine that governs mirror partitions, and the consistency guarantees that hold it all together. I then walk through 2 practical scenarios (disaster recovery and cluster migration), and end with a video demo.
MirrorMaker 2 (MM2) has served as the standard tool for cross-cluster replication since Kafka 2.4, running as a set of Kafka Connect workers that consume from a source cluster and produce to a destination cluster. Cluster mirroring takes a fundamentally different approach by embedding replication directly into the broker.
The following table summarizes the key differences:
With cluster mirroring, destination brokers become active participants in cross-cluster replication. Each one fetches data directly from the source cluster using the standard fetch protocol and appends raw record batches to local partition logs. Source and destination partitions share the same topic ID.
Beyond data replication, the broker also handles metadata discovery, configuration syncing, groups offset syncing, and ACL propagation. Bandwidth control works on both sides. The destination broker enforces a configurable replication rate limit. On the source side, mirror fetch traffic presents as standard consumer requests, so existing client quota mechanisms apply without modification.
There are 3 main components collaborating within each destination broker. Figure 1 shows how they connect to each other and to the source cluster.
MirrorMetadataManager (MMM) is the orchestrator. Running on every broker, it implements the MetadataPublisher interface to react to changes in the KRaft metadata log. When the controller writes a MirrorTopicStateChangeRecord, the MMM on the affected partition's leader drives the corresponding state transition that triggers a specific operation (create, start, stop, pause, resume, recover, delete).