> For the complete documentation index, see [llms.txt](https://docs.onum.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.onum.com/listeners/listener-integrations/collect-data-from-azure-blob-storage.md).

# Collect data from Azure Blob Storage

{% hint style="info" %}
See the changelog of the **Azure Blob Storage** Listener [here](/listeners-changelog/azure-blob-storage-listener.md).
{% endhint %}

## Overview

Onum supports integration with [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs).

The **Azure Blob Storage** Listener connects to your Azure Storage account and detects when new files are uploaded. It works by monitoring an Azure Storage Queue that receives notifications from Azure Event Grid whenever a blob is created. The Listener then retrieves the file content and makes it available for processing in your workflows.

## Prerequisites

Depending on your authentication method, you'll need the following permissions:

* **Connection String**: Storage account access key
* **Service Principal**: Azure AD application with these assigned roles:
  * `Storage Blob Data Reader` (minimum)
  * `Storage Queue Data Contributor` (minimum)

## Azure Blob Storage Setup

Event Hubs Capture enables you to specify your own Azure Blob storage account and container to  store the data. These accounts can be in the same region as your event hub or in another region.

### Prerequisites

* An active Azure subscription
* Azure portal access with sufficient permissions (Contributor or Storage Account Contributor)

### Configure Azure Storage Account

1. **Access the Azure Portal**
   * Go to [portal.azure.com](https://portal.azure.com/)
   * Sign in with your Azure credentials
2. **Create a new Storage Account**
   * Click **Create a resource**
   * Search for **Storage account** and select it
   * Click **Create**
3. **Configure basic settings**
   * **Subscription**: Select your Azure subscription
   * **Resource group**: Create new or select existing
   * **Storage account name**: Enter a unique name&#x20;
   * **Region**: Select your region
   * **Configure advanced settings**
4. **Click Create**

### Create a Blob Storage Container

1. **Navigate to your Storage Account**
   * Once deployment is complete, click **Go to resource** or find your storage account in the resources list
2. **Access Blob service**
   * In the left menu, under **Data storage**, select **Containers**
3. **Create a new container**
   * Click **+ Container**
   * **Name**: Enter a name for your container&#x20;
   * **Public access level**: Choose based on your security requirements.
   * Click **Create**

### Configure Access and Security

**Set up authentication**

* **Access Keys**:
  * In the storage account, go to **Access keys**
  * Copy the connection string or key for to enter into the ***Connection String*** parameter in the listener later on.
* **Azure AD**&#x20;
  * In the storage account, go to **Access Control (IAM)**
  * Assign appropriate roles to users or applications
* Enable Azure Monitor metrics

Your Azure Storage Account with Blob Storage container is now ready to use. Save the connection strings, container names, and access policies for use in the Listener.

## Onum Setup

{% stepper %}
{% step %}
Log in to your Onum tenant and click **Listeners > New listener**.
{% endstep %}

{% step %}
Double-click the **Azure Blob Storage** Listener.
{% endstep %}

{% step %}
Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener.
{% endstep %}

{% step %}
In the **Authentication** section, choose between:

<details>

<summary>Connection String</summary>

Use your storage account's connection string as your authentication method. This method is straightforward but requires managing the connection string securely.

Follow these steps to get your connection string:

1. Click your **Event Hubs namespace** to view the Hubs it contains.
2. Scroll down to the bottom and click the specific event hub to connect to.
3. In the left menu, go to **Shared Access Policies.**
4. If there is no policy created for an event hub, create one with **Manage**, **Send**, or **Listen** access.
5. Select the policy from the list.
6. Select the **copy** button next to the **Connection string-primary key** field. Depending on the version of Azure you are using, the corresponding field may have a different name, so to help you find it, look for a string with the same format:&#x20;

`Endpoint=sb://.servicebus.windows.net/; SharedAccessKeyName=RootManageSharedAccessKey; SharedAccessKey=`&#x20;

Now that you got it, open the **Connection String** field and click **New secret**. In the window that appears, give your secret a **Name** and turn off the **Expiration date** toggle if not needed. Then, click **Add new value** and paste the connection string. Click **Save** when you're done.

Now, select the token you have just created in the **Connection String** field.

{% hint style="info" %}
Learn more about secrets in [this article](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management).
{% endhint %}

</details>

<details>

<summary>Client Secret</summary>

Use Azure Active Directory authentication with a registered application and client secret. This provides better security and access control. We recommend to use this method for production environments and multi-tenant applications.

Enter your **Storage Account Name** and get the following credentials from the [**Certificates & Secrets**](https://learn.microsoft.com/en-us/entra/fundamentals/how-to-find-tenant) area:

* **Tenant ID** - Azure AD tenant identifier.
* **Client ID** - Azure AD application (service principal) identifier.
* **Client Secret** - Secret key for your service principal. To add it, open the field and click **New secret**. In the window that appears, give your secret a **Name** and turn off the **Expiration date** toggle if not needed. Then, click **Add new value** and paste your client secret. Click **Save** when you're done. Now, select the token you have just created in the **Client Secret** field.

{% hint style="info" %}
Learn more about secrets in [this article](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management).
{% endhint %}

</details>

<details>

<summary>Certificate</summary>

Use Azure Active Directory authentication with a certificate instead of a secret. This is the most secure option. We recommend to use this method for high-security production environments and compliance requirements.

Enter your **Storage Account Name** and get the following credentials from the [**Certificates & Secrets**](https://learn.microsoft.com/en-us/entra/fundamentals/how-to-find-tenant) area:

* **Tenant ID** - Azure AD tenant identifier.
* **Client ID** - Azure AD application (service principal) identifier.
* **Certificate** - PEM-encoded certificate with private key\
  Open the field and click **New secret**. In the window that appears, give your secret a **Name** and turn off the **Expiration date** toggle if not needed. Then, click **Add new value** and paste your certificate. Click **Save** when you're done. Now, select the token you have just created in the **Certificate** field.

{% hint style="info" %}
Learn more about secrets in [this article](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management).
{% endhint %}

</details>
{% endstep %}

{% step %}
In the **Retry Configuration** section, set the maximum number of attempts a failed Azure read should be retried (**Max Retries**) and the wait time before sending the next request after the last response was received and empty (**Idle Backoff Time**).
{% endstep %}

{% step %}
In the **Queue Configuration** section, enter the **Queue Name** of the queue that is receiving blob events.
{% endstep %}

{% step %}
In the **Limit & Timeout** section,  enter the following:

* **Message Limit** - Number of messages to retrieve per polling cycle. The minimum value is `1`, and the maximum value is `32`.
* **Visibility Timeout** - Number of seconds messages should stay hidden from other consumers while processing. The minimum value is `1`, and the maximum value is `604,800` (7 days).
  {% endstep %}

{% step %}
In the **Advanced configuration** section, you can optionally configure the following:

* **Use delivery format** - Activate to select the required delivery format of your data:
  * **Multiple events from JSON array**
  * **Multiple events from stacked JSON**
  * **Multiple events with delimiter** - Split file content into multiple messages using a delimiter. The default value is `\n` for line-by-line processing.
* **Use compression** - Activate this toggle if you want to listen for compressed files. Choose between **Auto**, **Gzip** or **Bzip2**.
  {% endstep %}

{% step %}
Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**.

{% hint style="info" %}
Learn more about labels in [this article](/listeners/labels.md).
{% endhint %}
{% endstep %}

{% step %}
Click **Create listener** when you're done.
{% endstep %}
{% endstepper %}

## Output Ports <a href="#ports" id="ports"></a>

The **Azure Blob Storage** Listener has only a single output port:

* **Default port** - Events are sent through this port if no error occurs while processing them.
* **Error port** - Events are sent through this port if an error occurs while processing them.