Collect data from Azure Event Hubs

Most recent version: v3.0.0

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

See the changelog of the Azure Event Hubs Listener here.

Overview

Onum supports integration with Azure Event Hubs

The Azure Event Hubs Listener receives messages from an Azure Event Hub for real-time data streaming, providing support for message batching, retries, and secure connection options.

Prerequisites

To enhance performance and resilience, we highly recommend enabling the Client Checkpointing option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using SINGLETON_LISTENER_EXECUTOR=true.

For Listener versions before v3.0.0, you must use the AZURE_EVENTHUB_LISTENER_EXECUTION_ENABLED environment variable, even if the Client Checkpointing option is enabled.

Azure Event Hubs Setup

There are various management credentials that Onum needs to communicate with the event hub:

Register a Microsoft application and generate a client secret

1

In the Microsoft Azure portal, search and select Microsoft Entra ID.

2

In the Microsoft Entra ID Overview page, click + Add > App registration. The Register an application page opens.

3

On the Register an application page, enter this info:

• Name - Enter an application name. Save this name to enter in a later step.

• Supported account types - Choose the account type based on your organization's requirements. We recommend choosing Accounts in this organizational directory only based on ​​least privilege access. For more info, see Identity and account types for single- and multitenant apps.

4

Click Register. The Application page opens with a Successfully created application notification.

5

In the Essentials section on the Application page, copy and save the Application (client) ID and the Directory (tenant) ID values to use in a later step.

6

In the navigation menu, click Manage > Certificates & secrets. The Certificates & secrets page appears.

7

Click the Client secrets tab and click + New client secret. The Add a client secret dialog opens.

8

In the dialog, enter a description for the client secret and a client secret expiration time. The expiration interval is based on your environment and determines how often the client secret needs to be regenerated.

9

Click Add. Your new client is now listed in the Client secrets tab with a Successfully updated application credentials notification.

10

Copy the Value field and save it somewhere safe to enter in a later step.

This sensitive info is displayed only once and is required for Listener configuration in a later step.

Create an Event Hub Namespace

If you already have an Event Hub Namespace and Event Hub created, skip to Step 4.

1

In the Microsoft Azure portal, search for and select Event Hubs. The Event Hub page opens.

2

Click + Create.The Create Namespace page opens.

3

In the Basics tab, set or input the Project and Instance Details:

• Subscription - Select your Azure subscription.

• Resource Group - Choose an existing resource group or click Create new, enter a Name for this resource group, and then click OK.

• Namespace name - Enter a unique name. Save this Event Hub Namespace name to enter in a later step.

• Location - Select a region to support availability zones for this namespace.

• Pricing Tier - Select a plan. Minimum required plan is the Standard tier. Based on your tier, select additional configuration options:

  • Throughput Units - Select the number of units. Default is 1.

4

In the Advanced tab, select Security measures:

• Minimum TLS version - Select a minimum TLS version. We recommend Version 1.2.

• Local Authentication - Select Enabled or Disabled based on your requirements.

5

In the Networking tab, choose a Connectivity method.

6

Confirm successful namespace creation with the Your deployment is complete message on screen.

Create an Event Hub

1

In the Next steps section, click Go to resource. The namespace Overview page opens.

2

Click + Event Hub. The Create Event Hub page opens.

3

In the Basics tab, enter Event Hub Details and set Retention settings:

• Name - Enter a name. Save this event hub name to enter in a later step.

• Partition count - Select the number of partitions. For more info, see Partitions.

As a best practice for processing large volumes of data, we recommended using the highest number of partitions.

• Cleanup policy - Select Delete or Compact based on your requirements. If you choose Compact, complete these tasks:

  • Select Infinite retention time.

  • Set a Tombstone retention time in hours. For more info, see Configure cleanup policy.

• Retention time (hrs) - As events are sent to the Listener for consumption when they are created, we recommend setting a 168 hour (7 day) retention time. If your tier does not allow this retention period, then set the max retention period for your tier. For more info, see Configure retention time.

4

In the Capture tab, turn Capture On or Off. For more info, see Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage.

5

Go to the Event Hubs Namespace > Settings > Shared access policies. Ensure the RootManageSharedAccessKey policy is present.

Add role assignment

1

In the main menu of the Event Hub Namespace page, click Access control (IAM).

If you’re using an existing Event Hub and the Event Hub Namespace contains other Event Hubs that you do not wish to give this role assignment, you can create the role assignment in the Event Hub instead. Go to the Event Hub page and click Access control (IAM).

2

Click the Role assignments tab to see the role assignments for this subscription.

3

To add a new role assignment, click + Add > Add role assignment.

4

In the Role tab, select the Azure Event Hub Data Receiver role, then click Next.

5

In the Members tab, click + Select Members.

6

In the Select Members dialog, search and select the application name value that you saved in Step 1, then click Select.

7

In the Review + assign tab, review the role assignment details.

8

Click Review + assign. The resource group Access IAM page opens with the role assigned to the selected scope.

Create a consumer group

As a best practice, if multiple applications are reading the same data in your Event Hub, we recommend creating a Consumer Group for each application or purpose. Each consumer group allows up to 5 concurrent readers with different processing requirements. For more info, see Consumer Groups.

Create a Consumer Group within your Event Hub:

1

Go to the Event Hub Namespace

2

Select your Event Hub.

3

In the main menu, click Consumer Groups.

4

Click + Consumer Group.

5

Enter a name for your consumer group.

6

Click Create.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Azure Event Hubs Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

Establish the Event Hub Connection:

• Event Hub Namespace* - You can find this in the top left-hand corner of you Azure area (e.g. mynamespace.servicebus.windows.net).

• Event Hub Name* - In your Azure console, click your Event Hubs namespace to view the Hubs it contains in the middle pane and enter it in the field. Alternatively, click Event Hub to create one.

• Consumer Group - In the left-hand menu of your Azure console, scroll down to Entities and click Consumer groups to see the names. This value is $Default when empty.

5

In the Authentication section, choose between Connection String and Entra ID as the Authentication Method.

• Connection String

  • Connection String* - The URL for your Event Hub. To get it:

    1. Click your Event Hubs namespace to view the Hubs it contains.

    2. Scroll down to the bottom and click the specific event hub to connect to.

    3. In the left menu, go to Shared Access Policies.

    4. If there is no policy created for an event hub, create one with Manage, Send, or Listen access.

    5. Select the policy from the list.

    6. Select the copy button next to the Connection string-primary key field.

    Depending on the version of Azure you are using, the corresponding field may have a different name, so to help you find it, look for a string with the same format: Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=;SharedAccessKey=

• Entra ID - Enter the following credentials from the Certificates & Secrets area:

  • Tenant ID*

  • Client ID*

  • Client Secret*

Open the Secret fields and click New secret to create a new one:

• Give the token a Name.

• Turn off the Expiration date option.

• Click Add new value and paste the secret corresponding to the JWT token you generated before. Remember that the token will be added in the Zscaler configuration.

• Click Save.

Learn more about secrets in Onum in this article.

6

You can now select the secret you just created in the corresponding fields.

7

Checkpointing & Processor

When multiple consumer instances read from the same Event Hub and consumer group, a cooperative processor coordinates partition ownership and progress using a checkpoint store (Azure Blob Storage).

• Ensures at-least-once processing without duplicates when instances restart: committed checkpoints allow new owners to resume from the last processed offset instead of re-reading the whole partition.

• Evenly distributes partitions across active instances (load balancing): with the balanced strategy, ownership is redistributed as instances join/leave; greedy tries to acquire as many partitions as possible.

• Enables safe horizontal scaling: adding instances increases throughput by processing multiple partitions in parallel.

Learn more in the Azure Event Hubs documentation:

• Checkpointing overview

• Tutorial on checkpoints and rewinding

To enhance performance and resilience, we highly recommend enabling the Client Checkpointing option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using SINGLETON_LISTENER_EXECUTOR=true.

For this, you will first need to create a blob storage container:

1. Click Go to resource or find your storage account in the resources list.

2. In the left menu, under Data storage, select Containers

3. Click + Container. This creates the blob container to persist checkpoints and ownership. Enter a name for your container & click Create.

4. Switch to Onum and enter the Storage Container Name* in the Listener configuration.

5. In the Azure storage account again, go to Access keys. Copy the connection string or key to enter into the Connection String parameter in the Listener later on.

6. The Storage Connection String* parameter is a secret, therefore you must add this string in the Secrets area, or select it from the list if you have already done so.

   See here for where to find it in the Azure portal.

7. Then, configure the Processor Options:

   • Load Balancing Strategy - Choose how to distribute the work evenly across the server to avoid overload.

     • Balanced - distributes load evenly across all servers.

     • Greedy - assigns each new task immediately to the currently least-loaded server.

   • Update Interval (ms) - How often a processor renews partition ownership; defaults to 10000ms if unset.

   • Partition Expiration Duration (ms) - Enter a time limit in milliseconds, after which the load partition will be considered expired and can be claimed by other instances.

8

Decide whether to Use batch settings.

When false, the handler processes events one-by-one using internal defaults (maxBatchSize=1, maxWaitTimeMs=500). When true, batch processing settings apply.

• Max Batch Size* - Enter the maximum number of events for the batch.

• Max Wait Time* - Enter the maximum amount of milliseconds to wait before considering the batch as complete.

9

The Start Position defines where to begin reading the event stream.

• Latest (End of Stream)

  • Onum begins reading from the next event that is enqueued after Onum starts. It skips all existing events currently in the partition.

• Earliest (Start of Stream)

  • Onum begins reading from the very first event currently retained in the partition. Events are only available up to the Event Hub's data retention period (e.g., 1 to 7 days for Standard, up to 90 days for Premium/Dedicated). You cannot read events older than the retention limit.

• Sequence Number

  • Onum begins listening at a specific event identified by its unique, increasing sequence number within that partition.

    • This will show a new field where you can enter the Sequence Number*, and an Inclusive* drop-down where true includes this value and false listens from this value onwards.

• Minutes ( FromEnqueuedTime)

  • Onum begins listening from the first event that was enqueued on or after a specified UTC date/time.

    • This will show a new field where you can enter the Minutes Ago*, and an Inclusive* drop-down where true includes this value and false listens from this value onwards.

If you have configured Checkpoint & Processing options, the Start Position only applies the first time you run the Listener. From then on, the checkpoint is used.

10

Add the Backoff Settings regarding how long to wait before retrying a request after failure.

• Error Backoff (ms) - Enter the amount of milliseconds to wait after an error before retrying.

• Idle Backoff (ms) - Enter the amount of milliseconds to wait before trying again to send a request.

11

Choose the Decompression method used to restore a compressed message to its original form before being processed (none, gzip or zlib).

12

Choose the Split Strategy method of dividing the data or requests from the following delimiter options:

• None to ignore

• Newline

• JSON array

• JSON object

• Custom Delimiter - Enter your custom delimiter here.

13

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabeled. Click Create listener when you're done.

Learn more about labels in this article.

14

Click Create listener when you're done.