Kafka - How to enable the REST API

Objective

Apache Kafka is an open-source, distributed event streaming platform designed for real-time, large-scale data processing with high scalability, durability, and low latency.

The Kafka REST API lets applications produce and consume messages over HTTPS, without a Kafka client library.

This guide explains how to enable the REST API of your Kafka service with the OVHcloud API, then how to produce and consume a first message with curl.

Requirements

• A Public Cloud project in your OVHcloud account
• A Kafka service accepting incoming connections from your IP address
• A topic, and a user with the readwrite permission on it (see the ACL guide)
• Access to the OVHcloud API

Instructions

Enabling the REST API

Update your service with restApi set to true (or false to disable the REST API):

{
  "restApi": true
}

Retrieving the REST API endpoint

Once the REST API is enabled, a kafkaRestApi endpoint is added to your service. You can find it in the endpoints list of the service details:

{
  "component": "kafkaRestApi",
  "domain": "kafka-xxxxxxxx-xxxxxxxxx.database.cloud.ovh.net",
  "port": 20188,
  "scheme": "https",
  "ssl": true,
  "uri": "https://<username>:<password>@kafka-xxxxxxxx-xxxxxxxxx.database.cloud.ovh.net:20188"
}

The REST API uses HTTPS with the CA certificate of your service, and the username and password of a Kafka user. Download the CA certificate as ca.pem (see Connect to a Kafka cluster with CLI).

Listing topics

curl --cacert ca.pem -u 'app-orders:<password>' https://<host>:<port>/topics

["orders","__consumer_offsets"]

Producing a message

Send a JSON message with the key order-1 to the orders topic:

curl --cacert ca.pem -u 'app-orders:<password>' \
  -H 'Content-Type: application/vnd.kafka.json.v2+json' \
  -X POST https://<host>:<port>/topics/orders \
  -d '{"records":[{"key":"order-1","value":{"amount":42}}]}'

The REST API returns the partition and offset of the message:

{"key_schema_id":null,"offsets":[{"offset":0,"partition":0}],"value_schema_id":null}

Consuming messages

Consuming takes three steps: create a consumer instance in a consumer group, subscribe it to the topic, then read the records.

# 1. Create the consumer instance "c1" in the consumer group "orders-readers"
curl --cacert ca.pem -u 'app-orders:<password>' \
  -H 'Content-Type: application/vnd.kafka.v2+json' \
  -X POST https://<host>:<port>/consumers/orders-readers \
  -d '{"name":"c1","format":"json","auto.offset.reset":"earliest"}'

# 2. Subscribe it to the "orders" topic
curl --cacert ca.pem -u 'app-orders:<password>' \
  -H 'Content-Type: application/vnd.kafka.v2+json' \
  -X POST https://<host>:<port>/consumers/orders-readers/instances/c1/subscription \
  -d '{"topics":["orders"]}'

# 3. Read the records
curl --cacert ca.pem -u 'app-orders:<password>' \
  -H 'Accept: application/vnd.kafka.json.v2+json' \
  https://<host>:<port>/consumers/orders-readers/instances/c1/records

The REST API returns the messages of the topic:

[{"key":"order-1","offset":0,"partition":0,"timestamp":1791971865910,"topic":"orders","value":{"amount":42}}]

When you are done, delete the consumer instance:

curl --cacert ca.pem -u 'app-orders:<password>' \
  -H 'Content-Type: application/vnd.kafka.v2+json' \
  -X DELETE https://<host>:<port>/consumers/orders-readers/instances/c1

Go further

Kafka - How to use Access Control Lists (ACLs)

Kafka - How to enable the Schema Registry

We want your feedback!

We would love to help answer questions and appreciate any feedback you may have.

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

Are you on Discord? Connect to our channel at https://discord.gg/ovhcloud and interact directly with the team that builds our Analytics service!

Join our community of users.