Collect data from Azure Event Hubs

See the changelog of the Azure Event Hubs Listener here.

The Azure Event Hubs Listener is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Azure Event Hubs

The Azure Event Hubs Listener receives messages from an Azure Event Hub for real-time data streaming, providing support for message batching, retries, and secure connection options.

Prerequisites

To enhance performance and resilience, we highly recommend enabling the Client Checkpointing option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using SINGLETON_LISTENER_EXECUTOR=true.

For Listener versions before v3.0.0, you must use the AZURE_EVENTHUB_LISTENER_EXECUTION_ENABLED environment variable, even if the Client Checkpointing option is enabled.

Azure Event Hubs Setup

There are various management credentials that Onum needs to communicate with the event hub:

Step 1: Register a Microsoft application and generate a client secret

1

In the Microsoft Azure portal, search and select Microsoft Entra ID.

2

In the Microsoft Entra ID Overview page, click + Add > App registration. The Register an application page opens.

3

On the Register an application page, enter this info:

• Name - Enter an application name. Save this name to enter in a later step.

• Supported account types - Choose the account type based on your organization's requirements. We recommend choosing Accounts in this organizational directory only based on ​​least privilege access. For more info, see Identity and account types for single- and multitenant apps.

4

Click Register. The Application page opens with a Successfully created application notification.

5

In the Essentials section on the Application page, copy and save the Application (client) ID and the Directory (tenant) ID values to use in a later step.

6

In the navigation menu, click Manage > Certificates & secrets. The Certificates & secrets page appears.

7

Click the Client secrets tab and click + New client secret. The Add a client secret dialog opens.

8

In the dialog, enter a description for the client secret and a client secret expiration time. The expiration interval is based on your environment and determines how often the client secret needs to be regenerated.

9

Click Add. Your new client is now listed in the Client secrets tab with a Successfully updated application credentials notification.

10

Copy the Value field and save it somewhere safe to enter in a later step.

This sensitive info is displayed only once and is required for Listener configuration in a later step.

Step 2: Create an Event Hub Namespace

If you already have an Event Hub Namespace and Event Hub created, skip to Step 4.

1

In the Microsoft Azure portal, search for and select Event Hubs. The Event Hub page opens.

2

Click + Create.The Create Namespace page opens.

3

In the Basics tab, set or input the Project and Instance Details:

• Subscription - Select your Azure subscription.

• Resource Group - Choose an existing resource group or click Create new, enter a Name for this resource group, and then click OK.

Only choose an existing resource group if that group hosts resources that share the same lifecycle, access controls, environment, or location as the resources in your Namespace.

• Namespace name - Enter a unique name. Save this Event Hub Namespace name to enter in a later step.

• Location - Select a region to support availability zones for this namespace.

• Pricing Tier - Select a plan. Minimum required plan is the Standard tier. Based on your tier, select additional configuration options:

  • Throughput Units - Select the number of units. Default is 1.

  • Enable auto-inflate - Check the box if you want throughput units to automatically increase based on your usage. For more info, see the Microsoft article Enable Auto inflate on a namespace.

Sending data from Event Hubs to this connector is limited to a maximum of 2MB per second per throughput unit. All Event Hubs and consumer groups in the same namespace share this limit. If egress capacity is exceeded, data delivery slows down silently and no errors are produced. Enable auto-inflate and monitor OutgoingBytes in Azure Monitor. For more info, see the Microsoft articles Automatically scale up Azure Event Hubs throughput units and Metrics.

4

In the Advanced tab, select Security measures:

• Minimum TLS version - Select a minimum TLS version. We recommend Version 1.2. For more info, see Configure the minimum TLS version for an Event Hubs namespace.

• Local Authentication - Select Enabled or Disabled based on your requirements. For more info about local authentication, see Authenticating Event Hubs Consumers with SAS.

5

In the Networking tab, choose a Connectivity method. For more info about connectivity methods, see Allow access to Azure Event Hubs namespaces from specific IP addresses or ranges.

6

In the Tags tab, add tags as needed.

7

In the Review + create tab, complete these tasks:

1. Review the namespace details.

2. Confirm the Validation succeeded message.

3. Click Create.

4. The deployment Overview page opens. Confirm successful namespace creation with the Your deployment is complete message on screen.

Step 3: Create an Event Hub

1

In the Next steps section, click Go to resource. The namespace Overview page opens.

2

Click + Event Hub. The Create Event Hub page opens.

3

In the Basics tab, enter Event Hub Details and set Retention settings:

• Name - Enter a name. Save this event hub name to enter in a later step.

Avoid using the same name for both the Event Hub and Event Hub Namespace.

• Partition count - Select the number of partitions. For more info, see Partitions.

As a best practice for processing large volumes of data, we recommended using the highest number of partitions.

• Cleanup policy - Select Delete.

• Retention time (hrs) - As events are sent to the Listener for consumption when they are created, we recommend setting a 168 hour (7 day) retention time. If your tier does not allow this retention period, then set the max retention period for your tier. For more info, see Configure retention time.

4

In the Capture tab, turn Capture On or Off. For more info, see Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage.

5

In the Review + Create tab, complete these tasks:

1. Review the instance details.

2. Confirm the Validation succeeded message.

3. Click Create.

4. The Event Hubs Namespace Overview page opens. Confirm successful Event Hub creation with the Successfully created Event Hub message notification.

Step 4: Add role assignment

1

In the main menu of the Event Hub Namespace page, click Access control (IAM).

If you’re using an existing Event Hub and the Event Hub Namespace contains other Event Hubs that you do not wish to give this role assignment, you can create the role assignment in the Event Hub instead. Go to the Event Hub page and click Access control (IAM).

2

Click the Role assignments tab to see the role assignments for this subscription.

3

To add a new role assignment, click + Add > Add role assignment.For more info about assigning roles, see Assign Azure roles using the Azure portal.

4

In the Role tab, select the Azure Event Hub Data Receiver role, then click Next.

5

In the Members tab, click + Select Members.

6

In the Select Members dialog, search and select the application name value that you saved in Step 1, then click Select.

7

In the Review + assign tab, review the role assignment details.

8

Click Review + assign. The resource group Access IAM page opens with the role assigned to the selected scope.

Step 5: Create a consumer group

As a best practice, if multiple applications are reading the same data in your Event Hub, we recommend creating a Consumer Group for each application or purpose. Each consumer group allows up to 5 concurrent readers with different processing requirements. For more info, see Consumer Groups.

Create a Consumer Group within your Event Hub:

1

Go to the Event Hub Namespace

2

Select your Event Hub.

3

In the main menu, click Consumer Groups.

4

Click + Consumer Group.

5

Enter a name for your consumer group.

6

Click Create.

Step 6: Configure data forwarding to your Event Hub

1

In the Microsoft Azure portal, search for and select Microsoft Entra ID.

2

In the Services area, click Microsoft Entra ID.

3

In Monitoring, select Diagnostic settings.

4

Click + Add diagnostic setting.

5

Diagnostic setting name - Enter a unique name.

6

Logs Categories - Select these logs to send to your Event Hub.

• AuditLogs

• SignInLogs

• ProvisioningLogs

• MicrosoftGraphActivityLogs

• NonInteractiveUserSignInLogs

• RiskyUsers

• UserRiskEvent

• ManagedIdentitySignInLogs

• ServicePrincipalSignInLogs

7

Destination details - Select Stream to an event hub.

8

Subscription - Select your Azure subscription.

• EventHub Namespace - Enter the Event Hub Namespace name that you created earlier.

• EventHub Name (Optional) - Enter the Event Hub name that you created earlier.

• EventHub policy name - Select the Event Hub policy name.

9

Click Save.

Step 7: Verify successful Event Hubs configuration

Verify data is successfully streaming to your event hub:

1

In the Azure Portal, search for and select Event Hub.

2

Click the new Event Hub namespace that you created in Step 2.

3

In the Overview page, look at the Messages chart and verify incoming messages.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Azure Event Hubs Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

Establish the Event Hub Connection:

• Event Hub Namespace - Enter the name of your existing Event Hubs Namespace or the Namespace name that you saved in Step 2.

• Event Hub Name - Enter the name of your existing Event Hub or the name that you saved in Step 3.

• Consumer Group - Enter the name of the Event Hub Consumer Group you created in Step 5. This value is $Default when empty.

5

In the Authentication section, choose between Connection String and Entra ID as the Authentication Method.

• Connection String

  • Connection String - The URL for your Event Hub. To get it:

    1. Click your Event Hubs namespace to view the Hubs it contains.

    2. Scroll down to the bottom and click the specific event hub to connect to.

    3. In the left menu, go to Shared Access Policies.

    4. If there is no policy created for an event hub, create one with Manage, Send, or Listen access.

    5. Select the policy from the list.

    6. Select the copy button next to the Connection string-primary key field.

    Depending on the version of Azure you are using, the corresponding field may have a different name, so to help you find it, look for a string with the same format: Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=;SharedAccessKey=

• Entra ID - Enter the following credentials from the Certificates & Secrets area:

  • Tenant ID - Enter the Directory (Tenant) ID value that you saved in Step 1.

  • Client ID - Enter the Application (Client) ID value that you saved in Step 1.

  • Client Secret - Enter the client secret value that you saved Step 1.

Open the Secret fields and click New secret to create a new one:

• Give the token a Name.

• Turn off the Expiration date option.

• Click Add new value and paste the secret corresponding Tenant ID/Client ID/Client Secret.

• Click Save.

You can now select the secret you just created in the corresponding fields.

Learn more about secrets in Onum in this article.

6

In the Decompression section, choose the method used to restore a compressed message to its original form before being processed (none, gzip or zlib).

7

In the Message Format section, choose how you want incoming messages to be parsed:

Raw

Choose this option if incoming messages are in raw format.

JSON

Choose this option if incoming messages are JSON objects. Enter the following:

• Path - Enter the path of the JSON element you want to retrieve. The path are keys separated by dots. For example one.two.three will select the element in {"one":{"two":{"three":[1,2,3]}}. To select the root, you can leave the path empty or enter a single dot . (default option).

• Array Unroll - Activate this toggle if you want to generate one event for each element in the array. The element that the path points to must be a JSON array.

JSON Lines

Choose this option if incoming messages are JSON Lines.

Delimiter

Choose this option if you want to consider a delimiter in your messages. Choose one the following:

• Non-Transparent Framing (newline) - Uses \n as the message delimiter.

• Non-Transparent Framing (zero) - Uses \0 as the message delimiter.

• Non-Transparent Framing (custom) - Uses a user-defined character sequence to separate messages. Enter the specific delimiter in the Custom Trailer Characters field.

• Octet Counting (message length) - Instead of using a delimiter to mark the end of a message, it prefixes each message with its exact byte length.

CSV

Choose this option if incoming messages are CSV objects. Enter the following:

• Header Row - Select true to include a header for your CSV rows.

• Delimiter - Decide between comma, semicolon and tab.

• Text Encoding - Choose the scheme (or key) that maps the characters used to store and read the data in the CSV file.

• Output Format - Choose either CSV or JSON.

  • JSON:

    • Converts CSV records to structured JSON objects

    • Field names are derived from header row (if present) or auto-generated (field_0, field_1, etc.)

    • Provides structured data for easier processing in Pipelines.

  • CSV:

    • Preserves original CSV formatting.

    • Each CSV record becomes a separate event containing the raw CSV line.

    • Useful when you want to maintain the original CSV structure.

• Trim Leading Space - Select true to remove any whitespace characters that appear immediately before the first non-whitespace character in a cell.

• Lazy Quotes - Select true to allow double quotes to appear in fields without strictly following the formal CSV escaping rules.

• Fields per Record - The number of fields expected in each row (record) of your CSV file.

• Comment Character - Use the hash symbol (#) to designate lines of text that should be ignored during the data parsing process.

If you were using an Azure Event Hubs Listener with version prior to v4.0.0, note that the Split Strategy and Custom Delimiter fields do not exist anymore from this version on. These are the equivalent field options from version v4.0.0:

• Split Strategy - None --> Message Format - Raw

• Split Strategy - New Line --> Message Format - Delimiter / Framing Method - Non-Transparent Framing (newline)

• Split Strategy - JSON Array --> Message Format - JSON / Array Unroll - true

• Split Strategy - JSON Object --> Message Format - JSON Lines

• Split Strategy - Custom Delimiter / Custom Delimiter - "x" --> Message Format - Delimiter / Framing Method - Non-Transparent Framing (custom) / Custom Trailer Characters - "x"

8

Checkpointing & Processor

When multiple consumer instances read from the same Event Hub and consumer group, a cooperative processor coordinates partition ownership and progress using a checkpoint store (Azure Blob Storage).

• Ensures at-least-once processing without duplicates when instances restart: committed checkpoints allow new owners to resume from the last processed offset instead of re-reading the whole partition.

• Evenly distributes partitions across active instances (load balancing): with the balanced strategy, ownership is redistributed as instances join/leave; greedy tries to acquire as many partitions as possible.

• Enables safe horizontal scaling: adding instances increases throughput by processing multiple partitions in parallel.

Learn more in the Azure Event Hubs documentation:

• Checkpointing overview

• Tutorial on checkpoints and rewinding

To enhance performance and resilience, we highly recommend enabling the Client Checkpointing option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using SINGLETON_LISTENER_EXECUTOR=true.

For this, you will first need to create a blob storage container:

1. Click Go to resource or find your storage account in the resources list.

2. In the left menu, under Data storage, select Containers

3. Click + Container. This creates the blob container to persist checkpoints and ownership. Enter a name for your container & click Create.

4. Switch to Onum and enter the Storage Container Name in the Listener configuration.

5. In the Azure storage account again, go to Access keys. Copy the connection string or key to enter into the Connection String parameter in the Listener later on. See here for where to find it in the Azure portal.

Back in Onum, the Storage Connection String parameter is a secret, therefore you must add this string in the Secrets area, or select it from the list if you have already done so. Enter the name in the Storage Container Name field.

9

Then, configure the Processor Options:

• Load Balancing Strategy - Choose how to distribute the work evenly across the server to avoid overload.

  • Balanced - distributes load evenly across all servers.

  • Greedy - assigns each new task immediately to the currently least-loaded server.

• Update Interval (ms) - How often a processor renews partition ownership; defaults to 10000ms if unset.

• Partition Expiration Duration (ms) - Enter a time limit in milliseconds, after which the load partition will be considered expired and can be claimed by other instances.

10

Decide whether to Use batch settings.

When false, the handler processes events one-by-one using internal defaults (maxBatchSize=1, maxWaitTimeMs=500). When true, batch processing settings apply.

• Max Batch Size - Enter the maximum number of events for the batch.

• Max Wait Time - Enter the maximum amount of milliseconds to wait before considering the batch as complete.

11

The Start Position defines where to begin reading the event stream.

• Latest (end of stream) - Onum begins reading from the next event that is enqueued after Onum starts. It skips all existing events currently in the partition.

• Earliest (start of stream) - Onum begins reading from the very first event currently retained in the partition. Events are only available up to the Event Hub's data retention period (e.g., 1 to 7 days for Standard, up to 90 days for Premium/Dedicated). You cannot read events older than the retention limit.

• Sequence Number - Onum begins listening at a specific event identified by its unique, increasing sequence number within that partition. This will show a new field where you can enter the Sequence Number, and an Inclusive drop-down where true includes this value and false listens from this value onwards.

• Minutes (From enqueued time) - Onum begins listening from the first event that was enqueued on or after a specified UTC date/time. This will show a new field where you can enter the Minutes Ago, and an Inclusive drop-down where true includes this value and false listens from this value onwards.

If you have configured Checkpoint & Processing options, the Start Position only applies the first time you run the Listener. From then on, the checkpoint is used.

12

Add the Backoff Settings regarding how long to wait before retrying a request after failure.

• Error Backoff (ms) - Enter the amount of milliseconds to wait after an error before retrying.

• Idle Backoff (ms) - Enter the amount of milliseconds to wait before trying again to send a request.

13

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabeled.

Learn more about labels in this article.

14

Click Create listener when you're done.

Output Ports

The Azure Event Hubs Listener has two output ports:

• Default port - Events are sent through this port if no error occurs while processing them.

• Error port - Events are sent through this port if an error occurs while processing them.