Foreword

I assume that you are familiar with Kafka and the concepts around it. I will not go into detail on that.

My use case: We have a Confluent Cloud instance with Kafka in it. We wanted to update one topic from retention policy delete to compact. Confluent is not really clear on their rules:

• At one place in the docs, it says that going from delete to delete, compact is impossible and I have to take the detour delete -> compact -> delete, compact.
• Control Center itself says “Changing the cleanup policy from delete to compact is not supported”. This also holds true in expert mode altough the docs suggest that you can edit anything you want in expert mode.

Long story short, changing retention from delete to compact is impossible from a Confluent point of view (altough I did not get an explanation why that is the case).

Now comes the interesting part: Choose any other Kafka Admin Client - and it just works! I tried the kafka-topics command line tool and also AKHQ, both changed the setting without problem. As far as I can see, the compaction was also done.

Now I still was unclear about if everything would work out. So I decided to poll a backup of the topic to a dev cluster running locally so that I can reproduce in case I destroy something. Enter Mirror Maker.

My experiences

First thing to learn was: There exists Mirror Maker 2.0 which is incompatible with the previous versions. So obviously, let’s take that.

How does it work? Neatly, everything is defined and configured in one configuration file they call mm2.properties (altough you can use any other name, obviously). I entered the following:

clusters = local, cloud

Nice. I just define the names for the clusters I want to use.

local.bootstrap.servers = localhost:9092

Configuring a local cluster is easy too: Just define the bootstrap servers, prepended with a local.

cloud.bootstrap.servers = {cluster-id}.{location}.{cloud}.confluent.cloud:9092
cloud.security.protocol=SASL_SSL
cloud.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='redacted' password='redacted';
cloud.sasl.mechanism=PLAIN

Naturally, adding a Confluent Cloud cluster is a bit more complicated, altough finally not that difficult at all.

Note that the security settings are done on cluster level, and not on producer or consumer level. I am not quite sure what will happen if I have a producer and a consumer with different ACLs in this case.

Defining the topics to mirror is easy too:

# Just mirror all topics
topics = .*

# OR: Mirror a subset of topics
topics = <regular expression>

One can also easily define properties for consumers or producers:

# When producing to the local cluster, use gzip compression
local.producer.compression.type = gzip

# Not creative enough for a real example here, sorry...
cloud.consumer.property = value

Finally, we have to define what is mirrored from where to where:

cloud->local.enabled = true

# Overwrite the topics to mirror
cloud->local.topics = <regular expression>

As easy as that! I a really excited about how easy it is to configure!

Learnings

I will stay in the setting that I mirror from a cloud cluster to a local cluster.

Replication factor

Just like other applications (looking at ksql), Mirror Maker creates a set of topics it uses. Those are…

• mm2-config.local.something on the cloud cluster
• mm2-config.cloud.something on the local cluster
• … same for some more topics like offset (so, mm2-offset-local.something), status etc.

When creating these topics, Mirror Maker by default uses a replication factor of 3 which usually is difficult in a development setting (I just have one broker in my Docker environment). We have multiple options for that:

• Just create the topics by hand, beforehand. They should be compacted, and there should be 25 or 50 partitions. Compact seems to be important, about the number of partitions, I don’t know…

• Define the correct properties in the Mirror Maker configuration:

  offset.storage.replication.factor = 1
  status.storage.replication.factor = 1
  config.storage.replication.factor = 1
  checkpoints.topic.replication.factor= 1
  heartbeats.topic.replication.factor= 1
  offset-syncs.topic.replication.factor= 1
  

  This is a good idea, as long as all the clusters support that. Fun Fact: Confluent Cloud requires every topic to have replication factor 3. Mirror Maker now again fails because it is unable to create its topics on the cloud cluster.

  It seems to be impossible to set those configurations on a per-cluster basis. Altough I did not go into detail too much.

I finally did a workaround: First, define the replication factor 1 for the topics and let some pseudo-topic be replicated from and to my local cluster. Then, drop those settings again and bring the Cloud cluster in. This works out - because then the topics already exist locally and Mirror Maker does not attempt to create them while still creating the topics in the Cloud with replication factor 3.

Message Size

Message Sizes have to be considered! In case of a too large message, Mirror Maker fails and does not recover.

Flooded logs

A real problem which, would I have realized before, could have saved me loads of trouble shooting in the problems mentioned before. I believe that Mirror Maker by default sends all the logs from all the producers and consumers it generates - because it prints loads of information over and over again.

And altough it is set to logging level INFO, the important informations (usually, the ERRORs) are just lost in the floods of information that is logged.

Next steps

• I still have to check if the compaction (my initial problem) actually works now; I still am not sure. And I also do not know how to check it properly.
• I (now) know of the concept of cluster Linking. I am not sure if this is a Kafka OSS feature or a Confluent feature but I have to try that out - maybe it is another tool for making mirroring easier.