WebsiteBlogLogin

Collect data from Apache Kafka

Most recent version: v2.1.0

See the changelog of this Listener type here.

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Apache Kafka.

Apache Kafka is a distributed, fault-tolerant, high-throughput, and scalable streaming platform. It's used for building real-time data pipelines and streaming applications.

Select Apache Kafka from the list of Listener types and click Configuration to start.

Apache Kafka Setup

You will need to set up a running Kafka cluster, with optional group IDs and Topics.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Apache Kafka Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

Enter the Bootstrap servers: the initial host-port pair that acts as the starting point to access the full set of alive servers in the cluster. This is a comma-separated list of host and port pairs using : as the separator, e.g. localhost:9092,another.host:9092

Enter your value and click Add element to add the required elements.

5

Enter the group ID string, which uniquely identifies the group of consumer processes. Find this in your Kafka Cluster at Home > Configuration > Consumer Properties.

6

We need to let the Listener know the topic(s) to connect to. Use kafka-topics --bootstrap-server :9092 --describe and write the result here. Click Add element to add as many topics as required.

7

Auto offset reset policy*

This policy defines the behavior when there are no committed positions available or when an offset is out of range. Choose between Earliest, Latest, or None.

8

Next we define the Authentication settings below, or select None if no authentication is required.

For the Plain, Scram, or TLS settings, some parameters will need to be added as secrets (see below for details)

Click New secret to create a new one:

  • Give the secret a Name.

  • Turn off the Expiration date option.

  • Click Add new value and paste the secret corresponding value.

  • Click Save.

Learn more about secrets in Onum in this article.

9

You can now select the secret you just created in the following fields:

  • Plain - Enter your Username* and select your Password* from your Secrets or create a new one.

  • Scram - Enter the required information:

    • Username* - Enter your username.

    • Password* - Select your password from your Secrets or create a new one.

    • SCRAM mechanism* - Choose either SHA-256 or SHA-512.

  • TLS - Enter the required information:

    • CA Certificate* - Select your CA certificate from your Secrets or create a new one.

    • Client certificate* - Select your client certificate from your Secrets or create a new one.

    • Client key* - Select your client key from your Secrets or create a new one.

    • Skip verify - Select true to skip or false to require verification.

    • Server name - Enter the name of the server to connect to.

    • Minimum TLS version - Select the required minimum version from the menu.

10

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

Click Create listener when you're done.

PreviousListener IntegrationsNextCollect data from AWS products

Last updated

Was this helpful?