Collect data from Microsoft Entra ID

See the changelog of the Microsoft Entra ID Listener here.

Overview

Onum supports integration with Microsoft Entra ID.

The Microsoft Entra ID Listener receives messages from an Azure Event Hub for real-time data streaming, providing support for message batching, retries, and secure connection options.

Prerequisites

To enhance performance and resilience, we highly recommend enabling the Client Checkpointing option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using SINGLETON_LISTENER_EXECUTOR=true.

Microsoft Entra ID Setup

There are various management credentials that Onum needs to communicate with Microsoft Entra ID:

Step 1: Register a Microsoft application and generate a client secret

1

In the Microsoft Azure portal, search and select Microsoft Entra ID.

2

In the Microsoft Entra ID Overview page, click + Add > App registration. The Register an application page opens.

3

On the Register an application page, enter this info:

• Name - Enter an application name. Save this name to enter in a later step.

• Supported account types - Choose the account type based on your organization's requirements. We recommend choosing Accounts in this organizational directory only based on ​​least privilege access. For more info, see Identity and account types for single- and multitenant apps.

4

Click Register. The Application page opens with a Successfully created application notification.

5

In the Essentials section on the Application page, copy and save the Application (client) ID and the Directory (tenant) ID values to use in a later step.

6

In the navigation menu, click Manage > Certificates & secrets. The Certificates & secrets page appears.

7

Click the Client secrets tab and click + New client secret. The Add a client secret dialog opens.

8

In the dialog, enter a description for the client secret and a client secret expiration time. The expiration interval is based on your environment and determines how often the client secret needs to be regenerated.

9

Click Add. Your new client is now listed in the Client secrets tab with a Successfully updated application credentials notification.

10

Copy the Value field and save it somewhere safe to enter in a later step.

This sensitive info is displayed only once and is required for Listener configuration in a later step.

Step 2: Create an Event Hub Namespace

If you already have an Event Hub Namespace and Event Hub created, skip to Step 4.

1

In the Microsoft Azure portal, search for and select Event Hubs. The Event Hub page opens.

2

Click + Create.The Create Namespace page opens.

3

In the Basics tab, set or input the Project and Instance Details:

• Subscription - Select your Azure subscription.

• Resource Group - Choose an existing resource group or click Create new, enter a Name for this resource group, and then click OK.

Only choose an existing resource group if that group hosts resources that share the same lifecycle, access controls, environment, or location as the resources in your Namespace.

• Namespace name - Enter a unique name. Save this Event Hub Namespace name to enter in a later step.

• Location - Select a region to support availability zones for this namespace.

• Pricing Tier - Select a plan. Minimum required plan is the Standard tier. Based on your tier, select additional configuration options:

  • Throughput Units - Select the number of units. Default is 1.

  • Enable auto-inflate - Check the box if you want throughput units to automatically increase based on your usage. For more info, see the Microsoft article Enable Auto inflate on a namespace.

Sending data from Event Hubs to this connector is limited to a maximum of 2MB per second per throughput unit. All Event Hubs and consumer groups in the same namespace share this limit. If egress capacity is exceeded, data delivery slows down silently and no errors are produced. Enable auto-inflate and monitor OutgoingBytes in Azure Monitor. For more info, see the Microsoft articles Automatically scale up Azure Event Hubs throughput units and Metrics.

4

In the Advanced tab, select Security measures:

• Minimum TLS version - Select a minimum TLS version. We recommend Version 1.2. For more info, see Configure the minimum TLS version for an Event Hubs namespace.

• Local Authentication - Select Enabled or Disabled based on your requirements. For more info about local authentication, see Authenticating Event Hubs Consumers with SAS.

5

In the Networking tab, choose a Connectivity method. For more info about connectivity methods, see Allow access to Azure Event Hubs namespaces from specific IP addresses or ranges.

6

In the Tags tab, add tags as needed.

7

In the Review + create tab, complete these tasks:

1. Review the namespace details.

2. Confirm the Validation succeeded message.

3. Click Create.

4. The deployment Overview page opens. Confirm successful namespace creation with the Your deployment is complete message on screen.

Step 3: Create an Event Hub

1

In the Next steps section, click Go to resource. The namespace Overview page opens.

2

Click + Event Hub. The Create Event Hub page opens.

3

In the Basics tab, enter Event Hub Details and set Retention settings:

• Name - Enter a name. Save this event hub name to enter in a later step.

Avoid using the same name for both the Event Hub and Event Hub Namespace.

• Partition count - Select the number of partitions. For more info, see Partitions.

As a best practice for processing large volumes of data, we recommended using the highest number of partitions.

• Cleanup policy - Select Delete.

• Retention time (hrs) - As events are sent to the Listener for consumption when they are created, we recommend setting a 168 hour (7 day) retention time. If your tier does not allow this retention period, then set the max retention period for your tier. For more info, see Configure retention time.

4

In the Capture tab, turn Capture On or Off. For more info, see Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage.

5

In the Review + Create tab, complete these tasks:

1. Review the instance details.

2. Confirm the Validation succeeded message.

3. Click Create.

4. The Event Hubs Namespace Overview page opens. Confirm successful Event Hub creation with the Successfully created Event Hub message notification.

Step 4: Add role assignment

1

In the main menu of the Event Hub Namespace page, click Access control (IAM).

If you’re using an existing Event Hub and the Event Hub Namespace contains other Event Hubs that you do not wish to give this role assignment, you can create the role assignment in the Event Hub instead. Go to the Event Hub page and click Access control (IAM).

2

Click the Role assignments tab to see the role assignments for this subscription.

3

To add a new role assignment, click + Add > Add role assignment.For more info about assigning roles, see Assign Azure roles using the Azure portal.

4

In the Role tab, select the Azure Event Hub Data Receiver role, then click Next.

5

In the Members tab, click + Select Members.

6

In the Select Members dialog, search and select the application name value that you saved in Step 1, then click Select.

7

In the Review + assign tab, review the role assignment details.

8

Click Review + assign. The resource group Access IAM page opens with the role assigned to the selected scope.

Step 5: Create a consumer group

As a best practice, if multiple applications are reading the same data in your Event Hub, we recommend creating a Consumer Group for each application or purpose. Each consumer group allows up to 5 concurrent readers with different processing requirements. For more info, see Consumer Groups.

Create a Consumer Group within your Event Hub:

1

Go to the Event Hub Namespace

2

Select your Event Hub.

3

In the main menu, click Consumer Groups.

4

Click + Consumer Group.

5

Enter a name for your consumer group.

6

Click Create.

Step 6: Configure data forwarding to your Event Hub

1

In the Microsoft Azure portal, search for and select Microsoft Entra ID.

2

In the Services area, click Microsoft Entra ID.

3

In Monitoring, select Diagnostic settings.

4

Click + Add diagnostic setting.

5

Diagnostic setting name - Enter a unique name.

6

Logs Categories - Select these logs to send to your Event Hub.

• AuditLogs

• SignInLogs

• ProvisioningLogs

• MicrosoftGraphActivityLogs

• NonInteractiveUserSignInLogs

• RiskyUsers

• UserRiskEvent

• ManagedIdentitySignInLogs

• ServicePrincipalSignInLogs

7

Destination details - Select Stream to an event hub.

8

Subscription - Select your Azure subscription.

• EventHub Namespace - Enter the Event Hub Namespace name that you created earlier.

• EventHub Name (Optional) - Enter the Event Hub name that you created earlier.

• EventHub policy name - Select the Event Hub policy name.

9

Click Save.

Step 7: Verify successful Event Hubs configuration

Verify data is successfully streaming to your event hub:

1

In the Azure Portal, search for and select Event Hub.

2

Click the new Event Hub namespace that you created in Step 2.

3

In the Overview page, look at the Messages chart and verify incoming messages.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Microsoft Entra ID Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

Establish the Event Hub Connection:

• Event Hub Namespace - Enter the name of your existing Event Hubs Namespace or the Namespace name that you saved in Step 2.

• Event Hub Name - Enter the name of your existing Event Hub or the name that you saved in Step 3.

• Consumer Group - Enter the name of the Event Hub Consumer Group you created in Step 5. This value is $Default when empty.

5

In the Authentication section, choose between Connection String and Entra ID as the Authentication Method.

• Connection String

  • Connection String - The URL for your Event Hub. To get it:

    1. Click your Event Hubs namespace to view the Hubs it contains.

    2. Scroll down to the bottom and click the specific event hub to connect to.

    3. In the left menu, go to Shared Access Policies.

    4. If there is no policy created for an event hub, create one with Manage, Send, or Listen access.

    5. Select the policy from the list.

    6. Select the copy button next to the Connection string-primary key field.

    Depending on the version of Azure you are using, the corresponding field may have a different name, so to help you find it, look for a string with the same format: Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=;SharedAccessKey=

• Entra ID - Enter the following credentials from the Certificates & Secrets area:

  • Tenant ID - Enter the Directory (Tenant) ID value that you saved in Step 1.

  • Client ID - Enter the Application (Client) ID value that you saved in Step 1.

  • Client Secret - Enter the client secret value that you saved Step 1.

Open the Secret fields and click New secret to create a new one:

• Give the token a Name.

• Turn off the Expiration date option.

• Click Add new value and paste the secret corresponding Tenant ID/Client ID/Client Secret.

• Click Save.

Learn more about secrets in Onum in this article.

6

You can now select the secret you just created in the corresponding fields.

7

Checkpointing & Processor

When multiple consumer instances read from the same Event Hub and consumer group, a cooperative processor coordinates partition ownership and progress using a checkpoint store (Azure Blob Storage).

• Ensures at-least-once processing without duplicates when instances restart: committed checkpoints allow new owners to resume from the last processed offset instead of re-reading the whole partition.

• Evenly distributes partitions across active instances (load balancing): with the balanced strategy, ownership is redistributed as instances join/leave; greedy tries to acquire as many partitions as possible.

• Enables safe horizontal scaling: adding instances increases throughput by processing multiple partitions in parallel.

Learn more in the Azure Event Hubs documentation:

• Checkpointing overview

• Tutorial on checkpoints and rewinding

To enhance performance and resilience, we highly recommend enabling the Client Checkpointing option in the Listener configuration. If it's not enabled, you must set the environment variable in your distributor with Docker Compose using SINGLETON_LISTENER_EXECUTOR=true.

For this, you will first need to create a blob storage container:

1. Click Go to resource or find your storage account in the resources list.

2. In the left menu, under Data storage, select Containers

3. Click + Container. This creates the blob container to persist checkpoints and ownership. Enter a name for your container & click Create.

4. Switch to Onum and enter the Storage Container Name in the Listener configuration.

5. In the Azure storage account again, go to Access keys. Copy the connection string or key to enter into the Connection String parameter in the Listener later on.

6. The Storage Connection String parameter is a secret, therefore you must add this string in the Secrets area, or select it from the list if you have already done so.

   See here for where to find it in the Azure portal.

7. Then, configure the Processor Options:

   • Load Balancing Strategy - Choose how to distribute the work evenly across the server to avoid overload.

     • Balanced - distributes load evenly across all servers.

     • Greedy - assigns each new task immediately to the currently least-loaded server.

   • Update Interval (ms) - How often a processor renews partition ownership; defaults to 10000ms if unset.

   • Partition Expiration Duration (ms) - Enter a time limit in milliseconds, after which the load partition will be considered expired and can be claimed by other instances.

8

Enter your Batch Processing settings:

• Max Batch Size - Enter the maximum number of events for the batch.

• Max Wait Time - Enter the maximum amount of milliseconds to wait before considering the batch as complete.

9

The Start Position defines where to begin reading the event stream.

• Latest (end of stream) - Onum begins reading from the next event that is enqueued after Onum starts. It skips all existing events currently in the partition.

• Earliest (start of stream) - Onum begins reading from the very first event currently retained in the partition. Events are only available up to the Event Hub's data retention period (e.g., 1 to 7 days for Standard, up to 90 days for Premium/Dedicated). You cannot read events older than the retention limit.

• Sequence Number - Onum begins listening at a specific event identified by its unique, increasing sequence number within that partition. This will show a new field where you can enter the Sequence Number, and an Inclusive drop-down where true includes this value and false listens from this value onwards.

• Minutes (From enqueued time) - Onum begins listening from the first event that was enqueued on or after a specified UTC date/time. This will show a new field where you can enter the Minutes Ago, and an Inclusive drop-down where true includes this value and false listens from this value onwards.

If you have configured Checkpoint & Processing options, the Start Position only applies the first time you run the Listener. From then on, the checkpoint is used.

10

Add the Backoff Settings regarding how long to wait before retrying a request after failure.

• Error Backoff (ms) - Enter the amount of milliseconds to wait after an error before retrying.

• Idle Backoff (ms) - Enter the amount of milliseconds to wait before trying again to send a request.

11

In the Message Processing section, choose the method used to restore a compressed message to its original form before being processed (none, gzip or zlib).

12

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabeled. Click Create listener when you're done.

Learn more about labels in this article.

13

Click Create listener when you're done.

Output Ports

The Azure Event Hubs Listener has two output ports:

• Default port - Events are sent through this port if no error occurs while processing them.

• Error port - Events are sent through this port if an error occurs while processing them.