Consuming a Kafka topic from Azure Confluent Kafka Cloud using Azure Databricks (Spark)

This post explains how to use Azure Databricks to read/consume a Kafka topic hosted on Azure Confluent Kafka Cloud.

Before you try this example, please have a look in my previous post about how to set up Azure Confluent Kafka Cloud in you own Microsoft Azure subscription.

The next step is to connect Azure Databricks together with Azure Confluent Kafka Cloud.

Push some sample data to your Kafka topic, my topic is called dc01_caiotopic1

Push some sample data to your Kafka topic

Use Azure Databricks to read the topic, see example below:

Azure Databricks to read the topic

Use the User Interface to create your Kafka API Key and change the Scala Code with your API key.

Special thank you to Gianluca Natali from Confluent.