Azure Event Hubs
Most recent version: v1.0.0
This is a Pull Listener and therefore should not be used in environments with more than one cluster.
Overview
Onum supports integration with Azure Event Hubs
The Azure Event Hubs Listener receives messages from an Azure Event Hub for real-time data streaming, providing support for message batching, retries, and secure connection options.
Select Azure Event Hubs from the list of Listener types and click Configuration to start.
Listener configuration
Now you need to specify how and where to collect the data, and how to establish a connection with Microsoft Azure Event Hubs.
Metadata
Enter the basic information for the new Listener.
Name*
Enter a name for the new Listener.
Description
Optionally, enter a description for the Listener.
Tags
Add tags to easily identify your Listener. Hit the Enter key after you define each tag.
Configuration
Now, add the configuration to establish the connection.
Event Hub Connection
Event Hub Namespace*
Enter the namespace to connect to (e.g. my-namespace.servicebus.windows.net). You can find this in the top left-hand corner of you Azure area.
Event Hub Name*
Click your Event Hubs namespace to view the Hubs it contains in the middle pane. Alternatively, click Event Hub to create one.
Consumer Group
In the left-hand menu, scroll down to Entities and click Consumer groups to see the names. This value is $Default when empty.
Authentication
Authentication Method*
Connection String
Connection String* The URL for your Event Hub. To get it:
Click your Event Hubs namespace to view the Hubs it contains.
Scroll down to the bottom and click the specific event hub to connect to.
In the left menu, go to Shared Access Policies.
If there is no policy created for an event hub, create one with Manage, Send, or Listen access.
Select the policy from the list.
Select the copy button next to the Connection string-primary key field.
Depending on the version of Azure you are using, the corresponding field may have a different name, so to help you find it, look for a string with the same format:
Endpoint=sb://.servicebus.windows.net/; SharedAccessKeyName=RootManageSharedAccessKey; SharedAccessKey=
Entra ID - enter the following credentials from the Certificates & Secrets area
Tenant ID*
Client ID*
Client Secret*
Checkpointing & Processor
When multiple consumer instances read from the same Event Hub and consumer group, a cooperative processor coordinates partition ownership and progress using a checkpoint store (Azure Blob Storage).
Ensures at-least-once processing without duplicates when instances restart: committed checkpoints allow new owners to resume from the last processed offset instead of re-reading the whole partition.
Evenly distributes partitions across active instances (load balancing): with the
balancedstrategy, ownership is redistributed as instances join/leave;greedytries to acquire as many partitions as possible.Enables safe horizontal scaling: adding instances increases throughput by processing multiple partitions in parallel.
Learn more in the Azure Event Hubs documentation:
It is recommended you only choose False if you are sure that your worker only uses a single distributor. If you suspect your worker is connected to more than one distributor, it is important to select true and configure the fields below.
Storage Container Name*
In the left-hand menu, scroll down to Resource groups, where you will see a list of all the storage containers within your Event Hub. In Onum, enter the name of the blob container to persist checkpoints and ownership.
Processor Options
Load Balancing Strategy
Choose how to distribute the work evenly across the server to avoid overload.
Balanced - distributes load evenly across all servers.
Greedy - assigns each new task immediately to the currently leadt-loaded server.
Update Interval (ms)
How often a processor renews partition ownership; defaults to 10000ms if unset.
Partition Expiration Duration (ms)
Enter a time limit in milliseconds, after which the load partition will be considered expired and can be claimed by other instances.
Use batch
When false, the handler processes events one-by-one using internal defaults (maxBatchSize=1, maxWaitTimeMs=500). When true, batch processing settings apply.
Max Batch Size*
Enter the maximum bytes for the batch.
Max Wait Time*
Enter the maximum amount of milliseconds to wait before considering the batch as complete.
Backoff Settings
Add the settings regarding how long to wait before retrying a request after failure.
Error Backoff (ms)
Enter the amount of milliseconds to wait after an error before retrying.
Idle Backoff (ms)
Enter the amount of milliseconds to wait before trying again to send a request.
Message Processing
Decompression
Choose the method used to restore a compressed message to its original form before being processed (none, gzip or zlib)
Split Strategy
Choose the method of dividing the data or requests from the following delimiter options:
None to ignore
Newline
JSON array
JSON object
Custom Delimiter
Custom Delimiter - enter your custom delimiter here.
Click Create labels to move on to the next step and define the required Labels if needed.
Last updated
Was this helpful?