WebsiteBlogLogin

Collect data from Apache Kafka

Most recent version: v2.1.1

See the changelog of this Listener type here.

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Apache Kafka.

Apache Kafka is a distributed, fault-tolerant, high-throughput, and scalable streaming platform. It's used for building real-time data pipelines and streaming applications.

Select Apache Kafka from the list of Listener types and click Configuration to start.

Prerequisites

In order to use this Listener, you must activate the environment variable in your distributor using docker compose (KAFKA_LISTENER_EXECUTION_ENABLED=true)

Apache Kafka Setup

You will need to set up a running Kafka cluster, with optional group IDs and Topics.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Apache Kafka Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

Enter the Bootstrap servers. These are the host-port pairs that act as the starting point to access the full set of alive servers in the cluster. Enter your value with format host:port and click Add element to add as many elements as required.

5

Enter the Group ID string, which uniquely identifies the group of consumer processes. Find this in your Kafka Cluster at Home > Configuration > Consumer Properties.

6

We need to let the Listener know the Topics to connect to. Use kafka-topics --bootstrap-server :9092 --describe and write the result here. Click Add element to add as many topics as required.

7

Auto offset reset policy*

This policy defines the behavior when there are no committed positions available or when an offset is out of range. Choose between Earliest, Latest, or None.

8

Next we define the Authentication settings below, or select None if no authentication is required.

For the Plain, Scram, or mTLS settings, some parameters will need to be added as secrets (see below for details)

Click New secret to create a new one:

  • Give the secret a Name.

  • Turn off the Expiration date option.

  • Click Add new value and paste the secret corresponding value.

  • Click Save.

Learn more about secrets in Onum in this article.

9

You can now select the secret you just created in the following fields:

  • Plain - Enter your Username* and select your Password* from your Secrets or create a new one.

  • Scram - Enter the required information:

    • Username* - Enter your username.

    • Password* - Select your password from your Secrets or create a new one.

    • SCRAM mechanism* - Choose either SHA-256 or SHA-512.

  • mTLS - Enter the required information:

    • CA Certificate* - Select your CA certificate from your Secrets or create a new one.

    • Client certificate* - Select your client certificate from your Secrets or create a new one.

    • Client key* - Select your client key from your Secrets or create a new one.

    • Skip verify - Select true to skip or false to require verification.

    • Server name - Enter the name of the server to connect to.

    • Minimum TLS version - Select the required minimum version from the menu.

10

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

11

Click Create listener when you're done.

PreviousListener IntegrationsNextCollect data from AWS products

Last updated

Was this helpful?