> For the complete documentation index, see [llms.txt](https://docs.onum.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.onum.com/listeners/listener-integrations/collect-data-from-azure-event-hubs.md).

# Collect data from Azure Event Hubs

{% hint style="info" %}
See the changelog of the **Azure Event Hubs** Listener [here](/listeners-changelog/azure-event-hubs-listener.md).
{% endhint %}

{% hint style="warning" %}
The **Azure Event Hubs** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster.
{% endhint %}

## Overview

Onum supports integration with [Azure Event Hubs](https://azure.microsoft.com/products/event-hubs) 

The **Azure Event Hubs** Listener receives messages from an Azure Event Hub for real-time data streaming, providing support for message batching, retries, and secure connection options.

## Prerequisites

To enhance performance and resilience, we highly recommend enabling the **Client Checkpointing** option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using `SINGLETON_LISTENER_EXECUTOR=true`.

{% hint style="warning" %}
For Listener versions before `v3.0.0`, you must use the `AZURE_EVENTHUB_LISTENER_EXECUTION_ENABLED` environment variable, even if the **Client Checkpointing** option is enabled.
{% endhint %}

## Azure Event Hubs Setup

There are various management credentials that Onum needs to communicate with the event hub:

### Step 1: Register a Microsoft application and generate a client secret <a href="#azureeventhubstemplate-step1-registeramicrosoftapplicationandgenerateaclientsecret" id="azureeventhubstemplate-step1-registeramicrosoftapplicationandgenerateaclientsecret"></a>

{% stepper %}
{% step %}
In the Microsoft Azure portal, search and select **Microsoft Entra ID**.
{% endstep %}

{% step %}
In the Microsoft Entra ID **Overview** page, click **+ Add** **>** **App registration**. The **Register** **an application** page opens.&#x20;
{% endstep %}

{% step %}
On the **Register** **an application** page, enter this info:

* **Name** - Enter an application name. Save this name to enter in a later step.
* **Supported account types** - Choose the account type based on your organization's requirements. We recommend choosing **Accounts** **in this organizational directory only** based on ​​least privilege access. For more info, see [Identity and account types for single- and multitenant apps](https://learn.microsoft.com/en-us/security/zero-trust/develop/identity-supported-account-types).
  {% endstep %}

{% step %}
Click **Register**. The **Application** page opens with a **Successfully created application** notification.
{% endstep %}

{% step %}
In the **Essentials** section on the **Application** page, copy and save the **Application (client) ID** and the **Directory (tenant) ID** values to use in a later step.
{% endstep %}

{% step %}
In the navigation menu, click **Manage** **>** **Certificates & secrets**. The **Certificates & secrets** page appears.&#x20;
{% endstep %}

{% step %}
Click the **Client secrets** tab and click **+** **New client secret**. The **Add a client secret** dialog opens.
{% endstep %}

{% step %}
In the dialog, enter a description for the client secret and a client secret expiration time. The expiration interval is based on your environment and determines how often the client secret needs to be regenerated.
{% endstep %}

{% step %}
Click **Add**. Your new client is now listed in the **Client secrets** tab with a **Successfully updated application credentials** notification.
{% endstep %}

{% step %}
Copy the **Value** field and save it somewhere safe to enter in a later step.
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
This sensitive info is displayed only once and is required for Listener configuration in a later step.
{% endhint %}

### Step 2: Create an Event Hub Namespace <a href="#azureeventhubstemplate-step2-createaneventhubnamespace" id="azureeventhubstemplate-step2-createaneventhubnamespace"></a>

{% hint style="warning" %}
If you already have an Event Hub Namespace and Event Hub created, skip to Step 4.
{% endhint %}

{% stepper %}
{% step %}
In the Microsoft Azure portal, search for and select **Event Hubs**. The **Event Hub** page opens.
{% endstep %}

{% step %}
Click **+** **Create**.The **Create Namespace** page opens.
{% endstep %}

{% step %}
In the **Basics** tab, set or input the **Project** and **Instance Details**:

* **Subscription** - Select your Azure subscription.
* **Resource Group** - Choose an existing resource group or click **Create new**, enter a **Name** for this resource group, and then click **OK**.

{% hint style="warning" %}
Only choose an existing resource group if that group hosts resources that share the same lifecycle, access controls, environment, or location as the resources in your **Namespace**.
{% endhint %}

* **Namespace name** - Enter a unique name. Save this Event Hub Namespace name to enter in a later step.
* **Location** - Select a region to support availability zones for this namespace.&#x20;
* **Pricing Tier** - Select a plan. Minimum required plan is the **Standard** tier. Based on your tier, select additional configuration options:
  * **Throughput Units** - Select the number of units. Default is 1.
  * **Enable auto-inflate** - Check the box if you want throughput units to automatically increase based on your usage. For more info, see the Microsoft article [Enable Auto inflate on a namespace](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-auto-inflate#enable-auto-inflate-on-a-namespace).

{% hint style="warning" %}
Sending data from Event Hubs to this connector is limited to a maximum of 2MB per second per throughput unit. All Event Hubs and consumer groups in the same namespace share this limit. If egress capacity is exceeded, data delivery slows down silently and no errors are produced. Enable auto-inflate and monitor `OutgoingBytes` in Azure Monitor. For more info, see the Microsoft articles [Automatically scale up Azure Event Hubs throughput units](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-auto-inflate) and [Metrics](https://learn.microsoft.com/en-us/azure/event-hubs/monitor-event-hubs-reference#metrics).
{% endhint %}
{% endstep %}

{% step %}
In the **Advanced** tab, select **Security** measures:

* **Minimum TLS version** - Select a minimum TLS version. We recommend Version 1.2. For more info, see [Configure the minimum TLS version for an Event Hubs namespace](https://learn.microsoft.com/en-us/azure/event-hubs/transport-layer-security-configure-minimum-version).
* **Local Authentication** - Select **Enabled** or **Disabled** based on your requirements. For more info about local authentication, see [Authenticating Event Hubs Consumers with SAS.](https://learn.microsoft.com/en-us/azure/event-hubs/authenticate-shared-access-signature#authenticating-event-hubs-consumers-with-sas)
  {% endstep %}

{% step %}
In the **Networking** tab, choose a **Connectivity method**. For more info about connectivity methods, see [Allow access to Azure Event Hubs namespaces from specific IP addresses or ranges](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-ip-filtering).
{% endstep %}

{% step %}
In the **Tags** tab, add tags as needed.
{% endstep %}

{% step %}
In the **Review + create** tab, complete these tasks:

1. Review the namespace details.
2. Confirm the **Validation succeeded** message.
3. Click **Create**.
4. The deployment **Overview** page opens. Confirm successful namespace creation with the **Your deployment is complete** message on screen.
   {% endstep %}
   {% endstepper %}

### Step 3: Create an Event Hub <a href="#azureeventhubstemplate-step3-createaneventhub" id="azureeventhubstemplate-step3-createaneventhub"></a>

{% stepper %}
{% step %}
In the **Next steps** section, click **Go to resource.** The namespace **Overview** page opens.
{% endstep %}

{% step %}
Click **+ Event Hub**. The **Create Event Hub** page opens.
{% endstep %}

{% step %}
In the **Basics** tab, enter **Event Hub Details** and set **Retention** settings:

* **Name** - Enter a name. Save this event hub name to enter in a later step.

{% hint style="info" %}
Avoid using the same name for both the Event Hub and Event Hub Namespace.
{% endhint %}

* **Partition count** - Select the number of partitions. For more info, see [Partitions](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#partitions).

{% hint style="info" %}
As a best practice for processing large volumes of data, we recommended using the highest number of partitions.&#x20;
{% endhint %}

* **Cleanup policy** - Select **Delete**.
* **Retention time (hrs)** - As events are sent to the Listener for consumption when they are created, we recommend setting a 168 hour (7 day) retention time. If your tier does not allow this retention period, then set the max retention period for your tier. For more info, see [Configure retention time](https://learn.microsoft.com/en-us/azure/event-hubs/configure-event-hub-properties#configure-cleanup-policy).
  {% endstep %}

{% step %}
In the **Capture** tab, turn Capture **On** or **Off**. For more info, see [Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview).
{% endstep %}

{% step %}
In the **Review + Create** tab, complete these tasks:

1. Review the instance details.
2. Confirm the **Validation succeeded** message.
3. Click **Create**.
4. The Event Hubs Namespace **Overview** page opens. Confirm successful Event Hub creation with the **Successfully created Event Hub message** notification.
   {% endstep %}
   {% endstepper %}

### Step 4: Add role assignment <a href="#azureeventhubstemplate-step4-addroleassignment" id="azureeventhubstemplate-step4-addroleassignment"></a>

{% stepper %}
{% step %}
In the main menu of the Event Hub Namespace page, click **Access control (IAM)**.

{% hint style="info" %}
If you’re using an existing Event Hub and the Event Hub Namespace contains other Event Hubs that you do not wish to give this role assignment, you can create the role assignment in the Event Hub instead. Go to the Event Hub page and click **Access control (IAM)**.
{% endhint %}
{% endstep %}

{% step %}
Click the **Role assignments** tab to see the role assignments for this subscription.
{% endstep %}

{% step %}
To add a new role assignment, click **+ Add > Add role assignment**.For more info about assigning roles, see [Assign Azure roles using the Azure portal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal).
{% endstep %}

{% step %}
In the **Role** tab, select the **Azure Event Hub Data Receiver** role, then click **Next**.
{% endstep %}

{% step %}
In the **Members** tab, click **+ Select Members**.
{% endstep %}

{% step %}
In the **Select Members** dialog, search and select the application name value that you saved in Step 1, then click **Select**.
{% endstep %}

{% step %}
In the **Review + assign** tab, review the role assignment details.
{% endstep %}

{% step %}
Click **Review + assign**. The resource group **Access IAM** page opens with the role assigned to the selected scope.
{% endstep %}
{% endstepper %}

### Step 5: Create a consumer group <a href="#azureeventhubstemplate-step5-createaconsumergroup" id="azureeventhubstemplate-step5-createaconsumergroup"></a>

As a best practice, if multiple applications are reading the same data in your Event Hub, we recommend creating a Consumer Group for each application or purpose. Each consumer group allows up to 5 concurrent readers with different processing requirements. For more info, see [Consumer Groups](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups).

Create a Consumer Group within your Event Hub:

{% stepper %}
{% step %}
Go to the Event Hub Namespace
{% endstep %}

{% step %}
Select your Event Hub.
{% endstep %}

{% step %}
In the main menu, click **Consumer Groups**.
{% endstep %}

{% step %}
Click **+ Consumer Group**.
{% endstep %}

{% step %}
Enter a name for your consumer group.
{% endstep %}

{% step %}
Click **Create**.
{% endstep %}
{% endstepper %}

### Step 6: Configure data forwarding to your Event Hub

{% stepper %}
{% step %}
In the Microsoft Azure portal, search for and select **Microsoft Entra ID**.
{% endstep %}

{% step %}
In the **Services** area, click **Microsoft Entra ID.**
{% endstep %}

{% step %}
In **Monitoring**, select **Diagnostic settings**.
{% endstep %}

{% step %}
Click **+ Add diagnostic setting**.
{% endstep %}

{% step %}
**Diagnostic setting name** - Enter a unique name.
{% endstep %}

{% step %}
**Logs Categories** - Select these logs to send to your Event Hub.

* `AuditLogs`
* `SignInLogs`
* `ProvisioningLogs`
* `MicrosoftGraphActivityLogs`
* `NonInteractiveUserSignInLogs`
* `RiskyUsers`
* `UserRiskEvent`
* `ManagedIdentitySignInLogs`
* `ServicePrincipalSignInLogs`
  {% endstep %}

{% step %}
**Destination details** - Select **Stream to an event hub**.
{% endstep %}

{% step %}
**Subscription** - Select your Azure subscription.

* **EventHub Namespace** - Enter the **Event Hub Namespace** name that you created earlier.
* **EventHub Name (Optional)** - Enter the **Event Hub** name that you created earlier.
* **EventHub policy name** - Select the **Event Hub policy** name.
  {% endstep %}

{% step %}
Click **Save**.
{% endstep %}
{% endstepper %}

### Step 7: Verify successful Event Hubs configuration

Verify data is successfully streaming to your event hub:

{% stepper %}
{% step %}
In the Azure Portal, search for and select **Event Hub**.
{% endstep %}

{% step %}
Click the new Event Hub namespace that you created in Step 2.
{% endstep %}

{% step %}
In the **Overview** page, look at the **Messages** chart and verify incoming messages.
{% endstep %}
{% endstepper %}

## Onum Setup

{% stepper %}
{% step %}
Log in to your Onum tenant and click **Listeners > New listener**.
{% endstep %}

{% step %}
Double-click the **Azure Event Hubs** Listener.
{% endstep %}

{% step %}
Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener.
{% endstep %}

{% step %}
Establish the **Event Hub Connection**:

* **Event Hub Namespace** - Enter the name of your existing Event Hubs Namespace or the Namespace name that you saved in Step 2.
* **Event Hub Name** - Enter the name of your existing Event Hub or the name that you saved in Step 3.
* **Consumer Group** - Enter the name of the Event Hub Consumer Group you created in Step 5. This value is `$Default` when empty.
  {% endstep %}

{% step %}
In the **Authentication** section, choose between **Connection String** and **Entra ID** as the **Authentication Method**.&#x20;

* **Connection String**
  * **Connection String** - The URL for your Event Hub. To get it:

    1. Click your **Event Hubs namespace** to view the Hubs it contains.
    2. Scroll down to the bottom and click the specific event hub to connect to.
    3. In the left menu, go to **Shared Access Policies**.
    4. If there is no policy created for an event hub, create one with **Manage**, **Send**, or **Listen** access.
    5. Select the policy from the list.
    6. Select the **copy** button next to the **Connection string-primary key** field.

    Depending on the version of Azure you are using, the corresponding field may have a different name, so to help you find it, look for a string with the same format: `Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=;SharedAccessKey=`
* **Entra ID** - Enter the following credentials from the [**Certificates & Secrets**](https://learn.microsoft.com/en-us/entra/fundamentals/how-to-find-tenant) area:
  * **Tenant ID** - Enter the Directory (Tenant) ID value that you saved in Step 1.
  * **Client ID** - Enter the Application (Client) ID value that you saved in Step 1.
  * **Client Secret** - Enter the client secret value that you saved Step 1.

Open the **Secret** fields and click **New secret** to create a new one:

* Give the token a **Name**.
* Turn off the **Expiration date** option.
* Click **Add new value** and paste the secret corresponding Tenant ID/Client ID/Client Secret.
* Click **Save**.

You can now select the secret you just created in the corresponding fields.

{% hint style="info" %}
Learn more about secrets in Onum in [this article](/settings/organization-settings/secrets-management.md).
{% endhint %}
{% endstep %}

{% step %}
In the **Decompression** section, choose the method used to restore a compressed message to its original form before being processed (**none**, **gzip** or **zlib**).
{% endstep %}

{% step %}
In the **Message Format** section, choose how you want incoming messages to be parsed:

<details>

<summary>Raw</summary>

Choose this option if incoming messages are in raw format.

</details>

<details>

<summary>JSON</summary>

Choose this option if incoming messages are JSON objects. Enter the following:

* **Path** - Enter the path of the JSON element you want to retrieve. The path are keys separated by dots. For example `one.two.three` will select the element in `{"one":{"two":{"three":[1,2,3]}}`. To select the root, you can leave the path empty or enter a single dot `.` (default option).
* **Array Unroll** - Activate this toggle if you want to generate one event for each element in the array. The element that the path points to must be a JSON array.

</details>

<details>

<summary>JSON Lines</summary>

Choose this option if incoming messages are JSON Lines.

</details>

<details>

<summary>Delimiter</summary>

Choose this option if you want to consider a delimiter in your messages. Choose one the following:

* **Non-Transparent Framing (newline)** - Uses `\n` as the message delimiter.
* **Non-Transparent Framing (zero)** - Uses `\0` as the message delimiter.
* **Non-Transparent Framing (custom)** - Uses a user-defined character sequence to separate messages. Enter the specific delimiter in the **Custom Trailer Characters** field.
* **Octet Counting (message length)** - Instead of using a delimiter to mark the end of a message, it prefixes each message with its exact byte length.

</details>

<details>

<summary>CSV</summary>

Choose this option if incoming messages are CSV objects. Enter the following:

* **Header Row** - Select **true** to include a header for your CSV rows.
* **Delimiter** - Decide between **comma**, **semicolon** and **tab**.
* **Text Encoding** - Choose the scheme (or key) that maps the characters used to store and read the data in the CSV file.
* **Output Format** - Choose either **CSV** or **JSON**.
  * **JSON:**
    * Converts CSV records to structured JSON objects
    * Field names are derived from header row (if present) or auto-generated (`field_0`, `field_1`, etc.)
    * Provides structured data for easier processing in Pipelines.
  * **CSV:**
    * Preserves original CSV formatting.
    * Each CSV record becomes a separate event containing the raw CSV line.
    * Useful when you want to maintain the original CSV structure.
* **Trim Leading Space** - Select **true** to remove any whitespace characters that appear immediately before the first non-whitespace character in a cell.
* **Lazy Quotes** - Select **true** to allow double quotes to appear in fields without strictly following the formal CSV escaping rules.
* **Fields per Record** - The number of fields expected in each row (record) of your CSV file.
* **Comment Character** - Use the hash symbol (`#`) to designate lines of text that should be ignored during the data parsing process.

</details>

{% hint style="warning" %}
If you were using an **Azure Event Hubs** Listener with version prior to **v4.0.0**, note that the **Split Strategy** and **Custom Delimiter** fields do not exist anymore from this version on. These are the equivalent field options from version v4.0.0:

* **Split Strategy** - `None` --> **Message Format** - `Raw`
* **Split Strategy** - `New Line` --> **Message Format** - `Delimiter` / **Framing Method** - `Non-Transparent Framing (newline)`
* **Split Strategy** - `JSON Array` --> **Message Format** - `JSON` / **Array Unroll** - `true`
* **Split Strategy** - `JSON Object` --> **Message Format** - `JSON Lines`
* **Split Strategy** - `Custom Delimiter` / **Custom Delimiter** - `"x"` --> **Message Format** - `Delimiter` / **Framing Method** - `Non-Transparent Framing (custom)` / **Custom Trailer Characters** - `"x"`
  {% endhint %}
  {% endstep %}

{% step %}
**Checkpointing & Processor**

When multiple consumer instances read from the same Event Hub and consumer group, a cooperative processor coordinates partition ownership and progress using a checkpoint store (Azure Blob Storage).

* Ensures at-least-once processing without duplicates when instances restart: committed checkpoints allow new owners to resume from the last processed offset instead of re-reading the whole partition.
* Evenly distributes partitions across active instances (load balancing): with the `balanced` strategy, ownership is redistributed as instances join/leave; `greedy` tries to acquire as many partitions as possible.
* Enables safe horizontal scaling: adding instances increases throughput by processing multiple partitions in parallel.

Learn more in the Azure Event Hubs documentation:

* [Checkpointing overview](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#checkpointing)
* [Tutorial on checkpoints and rewinding ](https://learn.microsoft.com/en-us/agent-framework/tutorials/workflows/checkpointing-and-resuming?pivots=programming-language-csharp)

{% hint style="warning" %}
To enhance performance and resilience, we highly recommend enabling the **Client Checkpointing** option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using `SINGLETON_LISTENER_EXECUTOR=true`.
{% endhint %}

For this, you will first need to create a blob storage container:

1. Click **Go to resource** or find your storage account in the resources list.
2. In the left menu, under **Data storage**, select **Containers**
3. Click **+ Container**. This creates the blob container to persist checkpoints and ownership. Enter a name for your container & click **Create.**
4. Switch to Onum and enter the **Storage Container Name** in the Listener configuration.
5. In the Azure storage account again, go to **Access keys**. Copy the connection string or key to enter into the **Connection String** parameter in the Listener later on. [See here](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) for where to find it in the Azure portal.

Back in Onum, the **Storage Connection String** parameter is a secret, therefore you must add this string in the [Secrets](/settings/organization-settings/secrets-management.md) area, or select it from the list if you have already done so. Enter the name in the **Storage Container Name** field.
{% endstep %}

{% step %}
Then, configure the **Processor Options**:

* **Load Balancing Strategy** - Choose how to distribute the work evenly across the server to avoid overload.
  * **Balanced** - distributes load evenly across all servers.
  * **Greedy** - assigns each new task immediately to the currently least-loaded server.
* **Update Interval (ms)** - How often a processor renews partition ownership; defaults to 10000ms if unset.
* **Partition Expiration Duration (ms)** - Enter a time limit in milliseconds, after which the load partition will be considered expired and can be claimed by other instances.
  {% endstep %}

{% step %}
Decide whether to **Use batch** setting&#x73;**.**

When **false**, the handler processes events one-by-one using internal defaults (`maxBatchSize=1`, `maxWaitTimeMs=500`). When **true**, batch processing settings apply.

* **Max Batch Size** - Enter the maximum number of events for the batch.
* **Max Wait Time** - Enter the maximum amount of milliseconds to wait before considering the batch as complete.
  {% endstep %}

{% step %}
The **Start Position** defines where to begin reading the event stream.&#x20;

* **Latest** (end of stream) - Onum begins reading from the **next event** that is enqueued *after* Onum starts. It skips all existing events currently in the partition.
* **Earliest** (start of stream) - Onum begins reading from the **very first** event currently retained in the partition. Events are only available up to the Event Hub's data retention period (e.g., 1 to 7 days for Standard, up to 90 days for Premium/Dedicated). You cannot read events older than the retention limit.
* **Sequence Number** - Onum begins listening at a **specific** event identified by its unique, increasing sequence number within that partition. This will show a new field where you can enter the **Sequence Number**, and an **Inclusive** drop-down where true includes this value and false listens from this value onwards.
* **Minutes** (From enqueued time) - Onum begins listening from the **first** event that was enqueued *on* or *after* a specified UTC date/time. This will show a new field where you can enter the **Minutes Ago**, and an **Inclusive** drop-down where true includes this value and false listens from this value onwards.

{% hint style="info" %}
If you have configured **Checkpoint & Processing** options, the **Start Position** only applies the first time you run the Listener. From then on, the checkpoint is used.&#x20;
{% endhint %}
{% endstep %}

{% step %}
Add the **Backoff Settings** regarding how long to wait before retrying a request after failure.

* **Error Backoff (ms)** - Enter the amount of milliseconds to wait after an error before retrying.
* **Idle Backoff (ms)** - Enter the amount of milliseconds to wait before trying again to send a request.
  {% endstep %}

{% step %}
Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**.

{% hint style="info" %}
Learn more about labels in [this article](/listeners/labels.md).
{% endhint %}
{% endstep %}

{% step %}
Click **Create listener** when you're done.
{% endstep %}
{% endstepper %}

## Output Ports <a href="#ports" id="ports"></a>

The **Azure Event Hubs** Listener has two output ports:

* **Default port** - Events are sent through this port if no error occurs while processing them.
* **Error port** - Events are sent through this port if an error occurs while processing them.