Kafka MirrorMaker - How to replicate topics between two Kafka services

Objective

Kafka MirrorMaker continuously copies the messages of Kafka topics from a source Kafka service to a target Kafka service. You can use it to migrate to a new Kafka service or version, or to keep a copy of your topics in another service.

This guide explains how to replicate a topic between two Kafka services of the same Public Cloud project with the OVHcloud API.

Requirements

• A Public Cloud project in your OVHcloud account
• Two Kafka services in this project: a source and a target. They can run different Kafka versions; this guide was tested with a Kafka 3.9 source and a Kafka 4.3 target.
• A topic to replicate on the source service, here inventory
• A Kafka MirrorMaker service in the same project
• Access to the OVHcloud API

Instructions

Step 1 - Linking both Kafka services to MirrorMaker

On your MirrorMaker service, create one integration for the source service and one for the target service:

{
  "type": "kafkaMirrorMaker",
  "sourceServiceId": "<source Kafka service ID>",
  "destinationServiceId": "<MirrorMaker service ID>"
}

{
  "type": "kafkaMirrorMaker",
  "sourceServiceId": "<target Kafka service ID>",
  "destinationServiceId": "<MirrorMaker service ID>"
}

In both cases, sourceServiceId is the Kafka service and destinationServiceId is the MirrorMaker service. Keep the id returned for each integration.

Step 2 - Creating the replication flow

Create a replication flow from the source integration to the target integration:

{
  "sourceIntegration": "<source integration ID>",
  "targetIntegration": "<target integration ID>",
  "topics": ["inventory"],
  "syncGroupOffsets": true,
  "syncInterval": 60,
  "emitHeartbeats": true,
  "enabled": true
}

The API returns the replication flow with its default replication policy:

{
  "id": "<replication ID>",
  "sourceIntegration": "<source integration ID>",
  "targetIntegration": "<target integration ID>",
  "replicationPolicyClass": "org.apache.kafka.connect.mirror.DefaultReplicationPolicy",
  "topics": ["inventory"],
  "topicExcludeList": [],
  "syncGroupOffsets": true,
  "syncInterval": 60,
  "emitHeartbeats": true,
  "enabled": true
}

In this replication flow:

• topics lists the topics to replicate, and topicExcludeList the topics to leave out;
• syncGroupOffsets copies the offsets of the consumer groups to the target service, every syncInterval seconds;
• emitHeartbeats makes MirrorMaker write heartbeat messages to the target service.

Step 3 - Checking the replicated topic

With the default replication policy, the replicated topic is created on the target service with the alias of the source service as a prefix. For the inventory topic, the target service contains a topic of the form xxxxxxxxx.kafka-xxxxxxxx.inventory, with the same messages as the source topic.

MirrorMaker also creates its own topics on the target service: heartbeats, xxxxxxxxx.kafka-xxxxxxxx.checkpoints.internal, and topics starting with mm2-.

To read the replicated messages, grant your users a read ACL on the replicated topic of the target service. See the ACL guide.

Step 4 - Managing the replication flow

Read or delete a replication flow:

Go further

Capabilities and Limitations of Analytics with Kafka MirrorMaker

We want your feedback!

We would love to help answer questions and appreciate any feedback you may have.

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

Are you on Discord? Connect to our channel at https://discord.gg/ovhcloud and interact directly with the team that builds our Analytics service!

Join our community of users.