There are many scenarios in which it can be useful or even necessary to replicate data from one Kafka cluster into a second cluster:
The tool of choice for mirroring data between clusters is called MirrorMaker and is distributed with Kafka. The documentation gives a brief introduction to its usage and what it does (and doesn't) do.
A notable limitation of MirrorMaker seems to be, that it can only mirror messages into a topic with the same name on the target cluster. There can however sometimes be scenarios, in which it would be preferable to change the target topic name, if you want to merge different topics into one during mirroring for example.
Alex from Confluent recently mentioned to me, how this can be achieved with MirrorMaker and after checking out this excellent example I decided to play around a bit with this. You need to implement a custom message handler which can change every mirrored message on the fly. MirrorMaker accepts a class name to use as message handler via the parameter --message.handler and a configuration string for the handler via --message.handler.args.
Your handler class will have to implement MirrorMakerMessageHandler - it is worth noting, that this interface changed in Kafka 0.10 (see 0.9.0.1 & 0.10.0.0) so be sure to code against the version you will be using.
The basic procedure is, that every message will be handed to your implementation of MirrorMakerMessageHandler which then returns a list of ProducerRecords, which allows you to pretty much perform any transformations you want during mirroring. If there is any configuration necessary that you want to pass into your object, you need to provide a constructor with one argument of type String. How you use this string is then entirely up to you.
I have created an example implementation which allows you to pass in a list of topics that should be renamed along with the names of new topics. Any topic that is whitelisted but not present in the message handler configuration will be mirrored into a topic of the same name, as per usual MirrorMaker behavior. Apart from changing the topic name, this handler copies the code that the defaultMirrorMakerMessageHandler uses in order to keep the behavior as close to the default as possible.
The repository contains a readme with instructions on how to install and configure the code. Feel free to give this a try and if you have any further questions please don't hesitate to let me know!