Send data to Apache Kafka
Most recent version: v0.0.1
Overview
Onum supports integration with Apache Kafka.
The Apache Kafka Data sink lets you send data messages to a Kafka topic, using information from events to construct messages. This component supports various configurations, including Kafka brokers, topic partitions, authentication methods, and message compression, providing a flexible and scalable approach for integrating with Kafka.
Prerequisites
You will need to set up a running Kafka cluster, with optional group IDs and Topics.
Onum Setup
Log in to your Onum tenant and click Data Sinks> New Data sink.
Double-click the Apache Kafka Sink.
Enter a Name for the new Data Sink. Optionally, add a Description and some Tags to identify the Sink.
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
Kafka Brokers
The broker hosts the partitions and handles incoming requests. Find this in the Instances area. The format is Kafka installed Machine HostName:Port Number.
Host*
Port*
Click Add element to add the required number of brokers.
In the Authentication section, if needed, here is where you enter your authentication parameters (either SASL or SSL/TLS). Otherwise, select None.
Click New secret to create a new one:
Give the secret a Name.
Turn off the Expiration date option.
Click Add new value and paste the secret corresponding to the JWT token you generated before. Remember that the token will be added in the Zscaler configuration.
Click Save.
Learn more about secrets in Onum in this article.
Configure your Topic & Partition:
Kafka topic*
Use
kafka-topics --broker :9092 --describeand write the result here.The Partition is the subdivision of the topic, designed to allow parallelism, scalability, and fault tolerance in message processing. This is where it is contained within the specified broker. Enter
-1for automatic selection.0- [msg1, msg4, msg7]1- [msg2, msg5, msg8]
The Acknowledgment (acks) setting controls how producers handle message durability and delivery confirmation from brokers. It defines how many replicas must confirm the write before the producer considers it successful.
All - Full replication. The leader waits for all in-sync replicas (ISR) to acknowledge before confirming.
-1 - This is the same as All.
0 - No acknowledgment. The producer sends data but does not wait for confirmation.
1 - Leader acknowledgment. The leader broker confirms receipt without waiting for replicas.
Batch configuration
Toggle on this switch button to enable batch settings. The values will be filled in by default; however, you can change them if needed.
Batch size
The batch size (in bytes) to allow through. The default value is 1000000.
Batch number of messages
The number of messages to allow through. The default value is 10000.
Queue max number of messages
Enter how many events to collect until considered full. The default value is 100000.
Queue max size
Enter the maximum number of events in KB to collect until considered full. The default value is 102400.
Linger
Enter the number of seconds to wait before considering the batch completed. The default value is 5.
Max in-flight requests per connection
Controls how many unacknowledged produce requests can be sent at the same time on a single TCP connection between a Kafka producer and a broker. The default value is 1000000.
Delivery timeout
The maximum amount of time to wait for a message to be successfully sent before it is marked as failed. The default value is 300000.
Request timeout
The number of milliseconds to wait before considering the batch request a timeout. The default value is 30000.
Use compression
Toggle on this switch if you need to compress your data and choose the required type between Gzip, Snappy, LZ4, or Zstd.
Enter the number of times to retry the server before deeming the request as timed out. The default value is 3, and the minimum value is 0.
Click Create data sink when complete.
Your new Data sink will appear in the Data sinks area list.
Pipeline configuration
When it comes to using this Data sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.
Output configuration
Message*
Choose the field containing the message to send.
Advanced configuration
Message key
If the message has a key, choose the field containing it here.
Click Save to save your configuration.
Last updated
Was this helpful?