Apache Kafka
Most recent version: v0.0.1
Overview
Onum supports integration with Apache Kafka.
The Apache Kafka Data sink lets you send data messages to a Kafka topic, using information from events to construct messages. This component supports various configurations, including Kafka brokers, topic partitions, authentication methods, and message compression, providing a flexible and scalable approach for integrating with Kafka.
Select Apache Kafka from the list of Data sink types and click Configuration to start.
Data sink configuration
Now you need to specify how and where to send the data, and how to establish a connection with Kafka.
Metadata
Enter the basic information for the new Data sink.
Name*
Enter a name for the new Data sink.
Description
Optionally, enter a description for the Data sink.
Tags
Add tags to easily identify your Data sink. Hit the Enter key after you define each tag.
Metrics
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
Configuration
Now add the configuration to establish the connection.
Kafka Brokers
The broker hosts the partitions and handles incoming requests. Find this in the Instances area. The format is Kafka installed Machine HostName:Port Number.
Host*
Enter the host of your Kafka broker(s).
Port*
Enter the port of your Kafka broker(s).
Click Add element to add the required number of brokers.
Authentication
If needed, here is where you enter your authentication parameters (either SASL or SSL/TLS). Otherwise, select None.
Topic & Partition
Kafka topic*
The topic to connect to. Use kafka-topics --broker :9092 --describe and write the result here.
Partition
The subdivision of the topic, designed to allow parallelism, scalability, and fault tolerance in message processing. This is where it is contained within the specified broker. Enter -1 for automatic selection.
0- [msg1, msg4, msg7]
1- [msg2, msg5, msg8]
Acknowledgements
Acks level*
The acknowledgment (acks) setting controls how producers handle message durability and delivery confirmation from brokers. It defines how many replicas must confirm the write before the producer considers it successful.
All - Full replication. The leader waits for all in-sync replicas (ISR) to acknowledge before confirming.
-1 - This is the same as All.
0 - No acknowledgment. The producer sends data but does not wait for confirmation.
1 - Leader acknowledgment. The leader broker confirms receipt without waiting for replicas.
Advanced configuration
Batch configuration
Toggle on this switch button to enable batch settings. The values will be filled in by default; however, you can change them if needed.
Batch size
The batch size (in bytes) to allow through. The default value is 1000000.
Batch number of messages
The number of messages to allow through. The default value is 10000.
Queue max number of messages
Enter how many events to collect until considered full. The default value is 100000.
Queue max size
Enter the maximum number of events in KB to collect until considered full. The default value is 102400.
Linger
Enter the number of seconds to wait before considering the batch completed. The default value is 5.
Max in-flight requests per connection
Controls how many unacknowledged produce requests can be sent at the same time on a single TCP connection between a Kafka producer and a broker. The default value is 1000000.
Delivery timeout
The maximum amount of time to wait for a message to be successfully sent before it is marked as failed. The default value is 300000.
Request timeout
The number of milliseconds to wait before considering the batch request a timeout. The default value is 30000.
Use compression
Toggle on this switch if you need to compress your data and choose the required type:
Type
Choose between Gzip, Snappy, LZ4, or Zstd.
Number of retries
Retries
Enter the number of times to retry the server before deeming the request as timed out. The default value is 3, and the minimum value is 0.
Click Finish when complete. Your new Data sink will appear in the Data sinks area list.
Pipeline configuration
When it comes to using this Data sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.
Output configuration
Message*
Choose the field containing the message to send.
Advanced configuration
Message key
If the message has a key, choose the field containing it here.
Click Save to save your configuration.
Last updated
Was this helpful?