Collect data from Apache Kafka

Most recent version: v2.1.1

See the changelog of this Listener type here.

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Apache Kafka.

Apache Kafka is a distributed, fault-tolerant, high-throughput, and scalable streaming platform. It's used for building real-time data pipelines and streaming applications.

Select Apache Kafka from the list of Listener types and click Configuration to start.

Prerequisites

In order to use this Listener, you must activate the following environment variable in your distributor using docker compose: KAFKA_LISTENER_EXECUTION_ENABLED=true

Apache Kafka Setup

You will need to set up a running Kafka cluster, with optional group IDs and Topics.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Apache Kafka Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

Enter the Bootstrap servers. These are the host-port pairs that act as the starting point to access the full set of alive servers in the cluster. Enter your value with format host:port and click Add element to add as many elements as required.

5

Enter the Group ID string, which uniquely identifies the group of consumer processes. Find this in your Kafka Cluster at Home > Configuration > Consumer Properties.

6

We need to let the Listener know the Topics to connect to. Use kafka-topics --bootstrap-server :9092 --describe and write the result here. Click Add element to add as many topics as required.

7

Now, you must choose an Auto offset reset policy*. This policy defines the behavior when there are no committed positions available or when an offset is out of range. Choose between Earliest, Latest, or None.

8

Next, you must define the Authentication configuration, or select None if no authentication is required. Choose between:

9

• Plain - Enter your Username* and select your Password* from your Secrets or create a new one.

• Scram - Enter the required information:

  • Username* - Enter your username.

  • Password* - Select your password from your Secrets or create a new one.

  • SCRAM mechanism* - Choose either SHA-256 or SHA-512.

• mTLS - Enter the required information:

  • CA Certificate* - Select your CA certificate from your Secrets or create a new one.

  • Client certificate* - Select your client certificate from your Secrets or create a new one.

  • Client key* - Select your client key from your Secrets or create a new one.

  • Skip verify - Select true to skip or false to require verification.

  • Server name - Enter the name of the server to connect to.

  • Minimum TLS version - Select the required minimum version from the menu.

10

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

11

Click Create listener when you're done.