# Welcome Falcon Onum helps security and IT leaders focus on the most important data. Gain control of your data by cutting through the noise for deep insights in real-time. {% embed url="" %} ### Quick links {% content-ref url="/pages/64bhgtTdXk4iZE8AU0lG" %} [Getting Started with Falcon Onum](/getting-started/getting-started-with-falcon-onum) {% endcontent-ref %} {% content-ref url="/pages/4Z3TFv3t7fJmUZYeA2E4" %} [Understanding The Essentials](/getting-started/understanding-the-essentials) {% endcontent-ref %} {% content-ref url="/pages/DYAGllTGDiM6UCbYQZw4" %} [Pipelines](/the-workspace/pipelines) {% endcontent-ref %} ### Most popular
ListenersLearn about how to set up and use Listeners/pages/GYyURyXe1A9niyvozKTO
PipelinesDiscover Pipelines to manage and customize your data
Data SinksAdd the final piece of the puzzle for simpler data
# About Falcon Onum Observability & Orquestration in real time. Any format. Any source. ## Overview The exponential growth of data ingestion volumes can lead to reduced performance, slow response times, and increased costs. With this comes the need to implement optimization strategies & volume reduction control. We help you cut the noise of large data streams and reduce infrastructure by up to 80%. Gain deep insights from any type of data, using any format, from any source. All of this... ### @ the Edge By collecting and observing that data at the edge, as close as possible to where it’s being generated, gain real-time observations and take decisive action to prevent network downtime, payment system failures, malware infections, and more. Unlike most tools that provide data observation and orchestration, Onum is not a data analytics space, which is already served well by security information and event management (SIEM) vendors and other analytics tools. Instead, Onum sits as close as possible to where the data is generated, and well in front of your analytics platforms, to collect and observe data across every aspect of your hybrid network. ## Start with the basics {% content-ref url="/pages/Iyl7WuGlyRGJ8SjfJ7sr" %} [Key Terminology](/getting-started/key-terminology) {% endcontent-ref %} {% content-ref url="/pages/4Z3TFv3t7fJmUZYeA2E4" %} [Understanding The Essentials](/getting-started/understanding-the-essentials) {% endcontent-ref %} # Architecture Designed for the Edge, created in the Cloud Easy, flexible deployment in any environment while keeping them as close as possible to where the data is produced delivers unparalleled speed and efficiency, enabling you to cut the infrastructure you have dedicated to orchestration by up to 80%. The Onum infrastructure consists of: * **Distributor**: this is the service that hosts the Listener before forwarding it to Workers. * **Worker**: this is the service that runs the Pipelines, receiving data from its Distributor and contained within a Cluster. * **Cluster**: a container grouping Distributors and Workers. You can have as many clusters as required per Tenant. Listeners are hosted within Distributors and are placed as close as possible to where data is generated. The Distributor pulls tasks from the data queue passing through the pipeline and distributes it to the next available worker in a Cluster. As soon as a Worker completes a task it becomes available again, and the Distributor in turn will assign it the next task from the queue. The installation process creates the Distributor and all Workers for each data source in the cluster.
## How it works
Any format. Any source.

Collect data from anywhere it’s generated, across every aspect of the network.

All data is aggregated, observed, and seamlessly routed to any destination.

/files/m0xz6eDpG95XBgfOlF33
Edge observabilityListeners are placed right on the edge to collect all data as close as possible to where it’s generated. /files/k8vfe7NXVrYHDF7jOIRn
Centralized managementOnum receives data from Listeners and observes and optimizes the data from all nodes. All data is then sent to the proper data sink./files/24bdNsyBxZQzrzhy4fro
## Deployment types The Onum Platform supports any deployment type ― including `on-premises`, the Onum public cloud, or your own private `cloud`. In a typical SaaS-based deployment, most processing activities are conducted in the Cloud.
Client-side components can be deployed on a Linux machine or on a Kubernetes cluster for easy, flexible deployment in any environment. Onum supports all major cloud environments, including Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. When the deployment type is on-premises, the communication between the management console and the process cluster will be encrypted with TLS and controlled by pull updates from the process cluster at configurable intervals. {% hint style="info" %} Learn more about Deployment requirements [here](/getting-started/deployment). {% endhint %} ## Delivery methods Onum supports all major standards such as Netflow, Syslog, and Kafka to orchestrate data streams to any desired destination, including popular data analytics tools such as Splunk and Devo, as well as storage environments such as S3.
# Deployment Falcon Onum installation process ## Overview Once you’ve obtained an Onum account, just a few steps are needed to complete the installation, depending on the type of deployment you require. Onum supports flexible deployment options, including both on-premises and cloud environments. In case you have any question regarding the deployment and installation process, please [contact us](/support). {% hint style="warning" %} **Supported browsers** Onum supports the following browsers: * Google Chrome {% endhint %} ### Cloud deployment For cloud-based installations, CrowdStrike or a partner will activate Onum access within the Falcon platform. All the necessary infrastructure will be set up based on estimated usage metrics. The deployment process is fully automated, ensuring quick and streamlined provisioning and configuration. {% hint style="warning" %} **Cloud Listeners** Note that the Listener configuration process is slightly different if you are using a Cloud deployment. Learn more [in our Listeners articles](/the-workspace/listeners/listener-integrations). {% endhint %} {% hint style="warning" %} **Outbound IPs** You might need to whitelist our IPs for some of our pull-based Listeners. For Falcon-based cloud Onum deployements, our outbound IPs are: * `54.76.153.190` * `52.209.110.159` * `3.255.0.156` For other cloud-based deployments, please contact Customer Sucess to gather information about the outbound IPs. {% endhint %} ### On-premises deployment In on-premises deployments, either a CrowdStrike team or a partner will set up the new account. Appropriate access permissions are granted to allow Onum to perform the installation. A validation script is run to confirm all prerequisites are met and connectivity is established, ensuring a smooth installation process. Once installed, you can access your tenant, start ingesting data, invite users, and take full advantage of Onum’s capabilities. You must allocate the disk space to `/opt` on the installation machine. {% hint style="info" %} **Dependencies:** * Docker * Packages: * `gpg` * `curl` * `ipvsadm` * `ca-certificates` * SIEM access * Access to sources {% endhint %} ## Hardware requirements Hardware (per Virtual Machine): * Distribution: Linux (Debian or Red Hat) * Server Hardware: 16GB RAM and 8 CPU * Disk Storage: At least 500GB ## Access In case of upcoming system maintenance, we kindly seek permission to access the customer infrastructure. We aim to ensure seamless operations and address any potential issues promptly. # Getting Started with Falcon Onum Welcome to Falcon Onum! This guide will help you start working with Onum, a powerful tool designed to enhance your data analysis and processing capabilities. {% embed url="" %} ## Accessing Onum Once you get your Onum credentials, you only have to go to [console.onum.com](https://console.onum.com/) and enter them to access your **Tenant**. A Tenant is a domain that contains a set of data in your organization. You can use one or various Tenants and grant access to as many as required. Learn more about working with Tenants [in this article](/administration/tenant-menu). ### Logging in Once in [console.onum.com](https://console.onum.com/), there are several ways to log in: * Log in with email address and password. Your password must be a minimum of 10 characters and include a combination of uppercase letters, lowercase letters, numbers, and symbols. * Two-factor authentication * Single Sign-On (SSO) with SAML * Single Sign-On (SSO) with OpenID Learn more about the different authentication types in [this section](/administration/global-settings/tenant/authentication). {% hint style="info" %} An inactive session will be automatically logged out after one hour. {% endhint %} ## Navigating the interface When you access the Onum app, you'll see [the **Home** page](/the-workspace/home), where you can see an overview of the activity in your Tenant. You can access the rest of the areas in Onum using the left panel.
## Create your first Listener Onum receives any data through **Listeners**. These are logical entities created within a Distributor, acting as the gateway to the Onum system. Configuring a Listener involves defining an IP address, a listening port, and a transport layer protocol, along with additional settings depending on the type of Listener specialized in the data it will receive. Access the **Listeners** area to start working with them. Learn how to create your first Listener [in this article](/the-workspace/listeners). ## Create your first Data Sink Onum outputs data via **Data sinks**. Use them to define where and how to forward the results of your streamlined data. Access the **Data sinks** area to start working with them. Learn how to create your first Data sink [in this article](/the-workspace/pipelines/data-sinks). ## Build your first Pipeline Use **Pipelines** to start transforming your data and build a data flow. Pipelines are made of the following components: * [Listeners](/the-workspace/listeners) * [Actions](/the-workspace/pipelines/actions) * [Data sinks](/the-workspace/pipelines/data-sinks) Learn more about Pipelines [in this section](/the-workspace/pipelines). ### Use cases Do you want to check the essential steps in Onum through specific Pipelines? Explore the most common use cases in [this section](https://docs.onum.com/usecases/). # Understanding The Essentials Get to grips with the important concepts & best practices of the Falcon Onum application. These articles contain information on functionalities across the entire platform.
Cards and Table Views/files/lRejjywuq0H2AwnLoSU3/pages/TepBA51V9byyFmkaofix
Data Types/files/cKxNW28CS4i0YQHX7rAN/pages/GigDqk7hAxltMMo8kSKl
Graph Calculations/files/D8Ic156H73eDvLPNGAHQ/pages/Cpi5yLyDONU9vuISuUnw
The Time Range Selector/files/3OumYnJHOuHTmWMLWra2/pages/Ffzy8DEy77jdr2yY5FE4
# Cards and Table Views Viewing and modifying elements in the table. ## Overview In the [Listeners](/the-workspace/listeners), [Pipelines](/the-workspace/pipelines), [Data sinks](/the-workspace/pipelines/data-sinks) and [Enrichment](/the-workspace/pipelines/actions/enrichment) areas, you can view all the resources in your Tenant as cards or in a table. Use the icons at the top right corner of the elements to switch between both modes. In both views, you can: * Use the search box to look for specific elements in the list. * Display all the elements individually in a list or grouped by types, and order the elements as required. These grouping options vary depending on the area you are. ## Cards view In this view, each element is displayed as a card that shows details about it. * Click a card to open the details window, or double-click it to access the element and edit it. In the details window, you can click the **Add tag** button and add the required tags to an element. For each tag you enter in the box, hit the `Enter` key. Click **Save** to add the tags. * Click the ellipsis button on each card to edit the element, copy its ID, or remove it.
## Table view In the **Table** view, you can click the cog icon to begin customizing the table settings. You can reorder the columns in the table, hide or display the required ones or pin them. Changes will be automatically applied. Click the **Reset** button to recover the original configuration.
* Click a row to open the details window, or double-click it to access the element and edit it. In the details window, you can cick the **Add tag** button and add the required tags to an element. For each tag you enter in the box, hit the `Enter` key. Click **Save** to add the tags. * Click the icon next to each column name to sort the table content or pin columns. * Click the ellipsis button on each row to edit the element, copy its ID, or remove it. ## Order and group elements In both cards and table views, you can display your elements using the following options: * **Order by** - Order the elements by a specific criterion (alphabetically, creation date or update date). * **Group by** - Group the elements in different categories. The categories you can choose may vary in each area of the platform (status, type, etc). # Data Types Easily identify data types using the color legend Since Onum can process any data type, you may be wondering how to identify which is which. See the color legend below:
Field typeDescriptionExample
A sequence of characters that is used primarily for textual data representation.
"hello world"
A list of string values separated by commas.
"hello", "my", "name", "is", "John"
Used to represent whole numbers without any fractional or decimal component. Integers can be positive, negative, or zero.
25
A list of integer values separated by commas.
1, 2, 3, 4
Used to represent real numbers with fractional parts, allowing for the representation of a wide range of values, including decimals.
1.2
A list of float values separated by commas.
0.1, -1.0, 2.0
Sequence of characters or encoded information that identifies the precise time at which an event occurred.
2024-05-17T14:30:00Z
A list of timestamps separated by commas.
2024-05-17T14:30:00Z, 2022-10-19T14:30:04Z, 1998-04-10T14:49:00Z
A fundamental data type in computer programming that represents one of two possible values: true or false.
true
A list of boolean values separated by commas.
true, false, true
A simple and widely used file format for storing tabular data, such as a spreadsheet or database. In a CSV file, each line of the file represents a single row of data, and fields within each row are separated by a delimiter, usually a comma.

id,name,price
1,Apple,0.99
2,Banana,0.59
3,Cherry,1.29
XML (Extensible Markup Language) is a markup language designed for encoding documents in a format that is both human-readable and machine-readable.
<Book>
    <Title>Example Title</Title>
    <Author>Author Name</Author>
</Book>
In a JSON, fields are represented by keys within objects, and the corresponding values can be of any JSON data type. This flexibility allows a JSON to represent structured data in a concise and readable manner, making it suitable for various applications, especially in web development and API communication.

{
  "items": [
    {
      "id": 1,
      "name": "Apple"
    },
    {
      "id": 2,
      "name": "Banana"
    },
    {
      "id": 3,
      "name": "Cherry"
    }
  ]
}
A key-value pair is a data structure commonly used in various contexts, including dictionaries, hash tables, and associative arrays. It consists of two components: a key and its corresponding value.
name = Alice
age = 30
city = Paris
Characters that separate individual fields or columns of data. The delimiter ensures that each piece of data within a row is correctly identified and separated from the others.
/
# Graph Calculations ## Overview This article outlines the more complex calculations that go on behind the graphs you see. In the [Listeners](/the-workspace/listeners), [Pipelines](/the-workspace/pipelines), and [Data sinks](/the-workspace/pipelines/data-sinks) views, you will see detailed metrics on your events and bytes in/out, represented in a graph at the top of these areas.
The line graph represents the events in/out, and the bar graph represents bytes in/on. Hover over a point on the chart to show a tooltip containing the events and bytes in for the selected time, as well as a percentage of how much increase/decrease has occurred since the previous lapse of time since the one currently selected. {% hint style="info" %} The chart in the Pipelines area is slightly different and includes some additional features. Learn more in the [Pipelines](/the-workspace/pipelines) section. {% endhint %} ## Events vs Bytes Use the **Events / Bytes** selector to choose which unit of measure you want to display in the graph. In both views, you will see the following data:
AVGThe average events/bytes per second ingested or sent by all listeners/Data Sinks in your Tenant.
MAXThe maximum number of events/bytes per second ingested or sent by all Listeners/Data Sinks in your Tenant.
MINThe minimum number of events/bytes per second ingested or sent by all Listeners/Data Sinks in your Tenant.
## Frequency slider and Stacked view Use the **Frequency** slider bar to choose how frequently you want to plot the events/bytes in the chart. By default, these graphs give an overview calculation of all the Listeners/Sinks in your Tenants. If you wish to see each Listener or Sink individually, use the **Stack** toggle. # The Time Range Selector ## Overview Throughout the entire Onum platform, you can set a period to either narrow down or extend the data shown. You can either select a predefined period or apply a custom time range. The related graph and resources will be automatically updated to display data from the chosen period. To remove a selected period, simply click the bin icon that appears next to the period to go back to the default time range (**1 hour ago**).
{% hint style="info" %} The intervals will be calculated according to the **Timezone** of your browser. Keep an eye out for future implementations, where you can manually select a timezone. {% endhint %} ## Predefined and Custom time ranges As well as predefined time intervals, you can also define a custom time range. To do it, simply select the required starting and ending dates in the calendar. ## Comparisons The interesting thing about Onum is that you can directly see how much volume you have saved compared to past ingestions, telling you what is going well and what requires further streamlining. The comparison is direct/equivalent, meaning all data shown is analyzed compared to the previously selected *equivalent* time range. For example, if the time range is **1 hour**, the calculation of differences will be carried out using the *previous one hour* before the current selection = * Range selected: **10:00-11:00** * Comparison: **09:00-10:00** Again, let´s say you now wish to view data over the last 7 days. The percentages will be calculated by measuring the volume retrospectively two weeks ago with the previous week. # Key Terminology Get to grips with these key concepts to better understand how Onum works and use it to its full potential. ### **Action** A unit of work performing a given operation on an event. *** ### **API** Application Programming Interface. A set of defined methods of communication among various components. *** ### **Cluster** Various distributors and workers can be grouped and contained within a cluster. You can have as many clusters as required per Tenant. *** ### **Data Sink** Where the data is routed after being processed by Onum. *** ### **Data source** Where the data is generated before ingesting it into Onum, e.g. application server logs, firewall logs, S3 bucket, Kafka Topic, etc. *** ### **Distributor** This service receives and processes the Listener data before sending it on to workers within a cluster. *** ### **Event** An event represents semi-structured data such as a log entry. Events can be parsed so that structured data can be generated and eventually processed by the engine. Events are composed of fields, which are referred to as **Field**. An action that produces a new field will be referred to as **outputField.** *** ### **Label** Used to sort events coming from Listeners into categories or sets that meet given filters to be used in a Pipeline. *** ### **Listener** A Listener retrieves events in a given IP address and a port, routing the data to the Pipelines so that it can be processed. *** ### **Lookup** A **lookup** refers to searching for and retrieving information from a specific source or dataset, typically based on a key or reference. *** ### **Multitenancy** Multitenancy is an architecture in which tenants share the same underlying infrastructure, including databases and application code, but their data and configurations are kept separate to ensure privacy and security. *** ### **Pipeline** A sequence of Actions connected through inputs/outputs to process a stream of data. Data comes from the Listener and eventually is routed to a Datasink. *** ### **Role** A role is assigned to a user in order to control the access they have to certain or all Onum features. This way, we can personalise the experience for each user. *** ### **Tag** Tags can be assigned to Listeners, Pipelines or Data sinks in order to classify them or make them easier to find. This is particularly useful if you have a wide database and want to avoid lengthy searching for the resources you wish to use. *** ### **Tenant** A Tenant is a domain that contains a set of data in your organization. You can use one or various tenants and grant access to as many as required. *** ### **Worker** This service runs the Pipelines, receiving data from its distributor and contained within a Cluster.\
# Home A summary of your Tenant activity {% embed url="" %} ## Overview When opening Onum, the **Home** area is the default view. Here you can see an overview of all the activity in your [Tenant](/administration/tenant-menu).
Use this view to analyze the flow of data and the change from stage to stage of the process. Here you can locate the most important contributions to your workflow at a glance. {% hint style="info" %} All data shown is analyzed compared to the previously selected time range. Use the time range selector at the top of this area to specify the periods to examine. For example, if the time range were **1 hour ago** (the default period), the calculation of differences will be carried out using the previous **one hour** before the current selection: * Range selected: **10:00-11:00** * Comparison: **09:00-10:00** To learn more about time ranges, go to [Selecting a Time Range.](/getting-started/understanding-the-essentials/the-time-range-selector) {% endhint %} ## Metrics The **Home** view shows various infographics that provide insights into your data flow. Some Listeners or Data Sinks may be excluded from these metrics if they are duplicates or reused. {% hint style="warning" %} The **Net Saved/Increased** and **Estimation** graphs will show an info tooltip if some [Data sinks](/the-workspace/pipelines/data-sinks) are excluded from these metrics. You may decide this during the Data sink creation. In those cases, you can hover over the icon to check the total metrics including all the Data sinks. {% endhint %}
Cover imageCover image (dark)
Net Saved/IncreasedHere you can see the difference (in %) of volume saved/increased in comparison to the previous period. Hover the circle icons to see the input/output volumes and see the total GB saved./files/4lAlKEvVaTmNeHenwtQS/files/SaIYslAVSN1RcRHBDrBY
ListenersView the total amount of data ingested by the Listeners in the selected time range compared to the previous, as well as the increased/decreased volume (in %)./files/JvKT9AuaT6mWUya3vvnw/files/dPHeRa3oHyFbNBtKNbPE
Data sinksYou can see at a glance the total amount of data sent out of your Tenant, as well as the difference (in %) with the previous time range selected./files/iK7fiRfRO0vdERcr4qSQ/files/nOvK7D9CawLThYcA1fAA
Data volumeThis shows the total volume of ingested data for the selected period. Notice it is the same as the input volume shown in the Net saved/increased metric. You can also see the difference (in %) with the previous time range selected./files/BlxacF30TFfsKg1kJ7Cs/files/CDOBUhIqs0Ha09Vmg8ys
EstimationThe estimated volumes ingested and sent over the next 24 hours. This is calculated using the data volume of the time period./files/GVDYL9BwuEpmBceRODFF/files/raK72iXeEGKqxzLauTPx
## Sankey diagram Each column of the Sankey diagram provides information and metrics on the key steps of your flow.
You can see how the data flows between: 1. [Listeners](/the-workspace/listeners) - each Listener in your Tenant. 2. [Clusters](/getting-started/architecture) - the Distributor/Worker group receives the Listener data and forwards it to Pipeline. 3. [Labels](/the-workspace/listeners/labels) - the operations and criteria used to filter out the data to be sent on to Pipelines. 4. [Pipelines](/the-workspace/pipelines) - the Pipelines used to obtain desired data and results. 5. [Data sinks](/the-workspace/pipelines/data-sinks) - the end destination for data having passed through **Listener › Cluster › Label › Pipeline**. Hover over a part of the diagram to see specific savings. ### Show metrics You can narrow down your analysis even further by selecting a specific node and selecting **Show metrics**. {% hint style="info" %} This option is not available for all columns. {% endhint %} ### **View details** Click a node and select **View details** to open a panel with in-depth details of the selected piece. From here, you can go on to edit the selected element. {% hint style="warning" %} This option is not available for all columns. {% endhint %} ### Hide/show columns You can choose which columns to view or hide using the eye icon next to its name. ### Add new elements You can add a new **Listener**, **Label**, **Pipeline** or **Data sink** using the plus button next to its name. You can also create all of the aforementioned elements using the **Create new** button at the top-right: # Listeners Everything starts with a good Listener {% embed url="" %} ## Overview Essentially, Onum receives any data through **Listeners**. These are logical entities created within a [Distributor](/getting-started/architecture), acting as the gateway to the Onum system. Due to this, configuring a Listener involves defining an IP address, a listening port, and a transport layer protocol, along with additional settings depending on the type of Listener specialized in the data it will receive. A **Push** type of Listener passively sources data without explicitly requesting, whereas a **Pull** type is where the user actively requests data from an external source. {% hint style="warning" %} If you are using more than one **Cluster**, it is recommended not to use a **Pull-type** Listener. You can find out the Listener type in the integration-specific articles below. {% endhint %} Click the **Listeners** tab on the left menu for a general overview of the Listeners configured in your Tenant and the events generated.
* The blue graph represents the events in. Use the buttons above the graph to switch between **Events**/**Bytes**, and the **Frequency** slider bar to choose how frequently you want to plot the events/bytes in the chart. Use the **Stack Listeners** toggle to view each individual Listener on your graph and its metrics. Learn more about this graph [in this article](/getting-started/understanding-the-essentials/graph-calculations). * Hover over a point on the chart to show a tooltip containing the events/bytes coming in for the selected time, as well as a percentage of how much increase/decrease has occurred between the previous lapse of time and the one currently selected. At the bottom, you have a list of all the Listeners in your Tenant. You can switch between the **Cards** view, which shows each Listener in a card, and the **Table** view, which displays Listeners listed in a table. Learn more about the cards and table views [in this article](/getting-started/understanding-the-essentials/cards-and-table-views). ## Narrow Down Your Data There are various ways to narrow down what you see in this view: ### **Add Filters** Add filters to narrow down the Listeners you see in the list. Click the **+ Filter** button and select the required filter type(s). You can filter by: * **Name** - Select a **Condition** (**Contains**, **Equals**, or **Matches**) and a **Value** to filter Listeners by their names. * **Version** - Filter Listeners by their version. Choose a **Condition** and a **Value** to filter by. * **Type** - Choose the Listener type(s) you want to see in the list. * **Created by** - Selecting this option opens a users dropdown where you can filter by creator. * **Updated by** - Selecting this option opens a users dropdown where you can filter by the last user to update a pipeline. The filters applied will appear as tags at the top of the view. {% hint style="warning" %} Note that you can only add one filter of each type. {% endhint %} ### **Select a Time Range** If you wish to see data for a specific time period, this is the place to click. Go to [this article](/getting-started/understanding-the-essentials/the-time-range-selector) to dive into the specifics of how the time range works. ### **Select Tags** You can choose to view only those Listeners that have been assigned the desired tags. You can create these tags in the Listener settings or from the cards view. Press the `Enter` key to confirm the tag, then **Save**. To filter by tags, click the **Tags** button, select the required tag(s) and click **Save**. ## Create a Listener Depending on your permissions, you can create a new Listener from this view. There are several ways to create a new Listener: * From the **Listeners** view, clicking the **New listener** button. * From[ the Home page](/the-workspace/home), clicking **Create new > Listener** or clicking the **+** button in the **Listeners** column of the Sankey diagram. * Hover over the **Listeners** section in the left pane and click the **Create listener** button that appears in the search window. * Within a [Pipeline](/the-workspace/pipelines). Configuring your Listener involves various steps: {% stepper %} {% step %} #### Choose your Listener type The first step is to define the Listener **Type**. Select the desired type in this window and select **Configuration**, or double-click it. Check the list of available Listener types in [this article](/the-workspace/listeners/listener-integrations). {% endstep %} {% step %} #### Configure your Listener The configuration is different for each Listener type. Check the different Listener types and how to configure them [in this section](/the-workspace/listeners/listener-integrations). {% hint style="warning" %} If your Listener is deployed in the Cloud, you will see an extra step for the network properties. {% endhint %} {% endstep %} {% step %} #### Add Labels Use Onum's labels to cut out the noise with filters and search criteria based on specific metadata. This way, you can categorize events sent on and processed in your [Pipelines](/the-workspace/pipelines). Learn more about labels [in this article](/the-workspace/listeners/labels). {% endstep %} {% endstepper %} ## Edit a Listener You can edit an existing Listener by double-clicking it in the Listeners view. You'll be directly taken to its configuration form, where you can edit any required values. Alternatively, click the ellipses in the card or table view in the **Listeners** area and select **Edit**, or click a Listener to access its details view and select **Edit listener**. # Listener Integrations Collect data in real-time, no matter the source ## Overview Although there is a fixed number of Listener types available, the integration possibilities are endless. Onum is designed to be source-agnostic, ensuring you can ingest data from virtually any product or technology. We achieve this through a strategic, two-pillar approach to our Listeners: * First, we offer a growing suite of **dedicated Listeners for specific technologies** (such as Amazon S3, Microsoft Office, and others). These provide a streamlined configuration process for popular services. * Second, and crucially, we provide **standard protocol Listeners** (including HTTP, TCP, and Syslog). This ensures that even if a product does not have a dedicated, named Listener, you can still seamlessly send data to Onum using these widely supported, industry-standard protocols. This dual model guarantees comprehensive coverage, making it clear that whether you need a highly specialized integration or simply a robust, standardized connection, Onum is ready to collect your data. You can [contact us to request a specific Listener type](/support). ## Dedicated Listeners Check the current suite of dedicated Listeners we offer in the Onum platform:
Cover image
Amazon Kinesis Data Stream ListenerIngest data streams from Amazon Kinesis/files/S1da6eoD5qqvQkP21CbU
Amazon S3 ListenerReceive data from your Amazon S3 buckets/files/wW73FuASiITHyIo7xmgb
Amazon SQS ListenerInject queue messages from Amazon SQS/files/56yJw8vC7Adh8Y5xx91F
Azure Blob Storage ListenerCollect data from a container in Azure Blob Storage/files/TV4gsDXI6CQSBqgD8W1m
Apache Kafka ListenerSend data from your Apache Kafka clusters/files/GEkBUyz7STYUNziiK8fE
Azure Event Hubs ListenerReceive messages from a hub in Azure Event Hubs/files/Y3PbyxLwcz5WcczRsWdB
Falcon LogScale Collector ListenerCollect data from your Falcon LogScale Collector/files/VlUWoeJrz6j8AiFfhIMB
Google Cloud Storage ListenerSource data from a Google Cloud Storage bucket/files/8T1EM1PQszYZmaHWuo1D
Google Pub/Sub ListenerStream data from your Google Pub/Sub subscriptions/files/qNgDLiJOtLbWkKGSiP5Z
Microsoft 365 ListenerSend content from your Microsoft 365 products/files/xSN5NhCOzhWgOMDj8KGj
## Standard Protocol Listeners Click to see how to configure each of our Listeners for standard protocols:
Cover image
Cisco NetFlow ListenerListen for NetFlow packet records/files/hutRwyJri9hOeNbfZSZ8
HTTP ListenerListen for HTTP requests/files/nKwyFKoHDkNi27jXBqjL
HTTP Pull ListenerPull JSON data from HTTP endpoints/files/hutRwyJri9hOeNbfZSZ8
OpenTelemetry ListenerProcess OpenTelemetry metrics, traces and logs/files/hutRwyJri9hOeNbfZSZ8
SNMP Trapd ListenerReceive SNMP traps from network devices/files/hutRwyJri9hOeNbfZSZ8
Syslog ListenerProcess Syslog messages/files/nKwyFKoHDkNi27jXBqjL
TCP ListenerRead data from a TCP stream of bytes/files/nKwyFKoHDkNi27jXBqjL
## Other Listeners
Cover image
Relational Databases ListenerRead data from your databases/files/jxk2RxyJcyHOq5zzoQSi
Tick ListenerEmit synthetic events on a defined schedule/files/HMhoAPmZ2oO3QHGSQxTz
# Collect data from Apache Kafka {% hint style="info" %} See the changelog of the **Apache Kafka** Listener [here](/listeners/apache-kafka-listener). {% endhint %} {% hint style="warning" %} The **Apache Kafka** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster. {% endhint %} ## Overview Onum supports integration with [Apache Kafka](https://kafka.apache.org/). Apache Kafka is a distributed, fault-tolerant, high-throughput, and scalable streaming platform. It's used for building real-time data pipelines and streaming applications. Select **Apache Kafka** from the list of Listener types and click **Configuration** to start. ## Prerequisites {% hint style="warning" %} In order to use the **Apache Kafka** Listener, you must activate the following environment variable in your distributor using docker compose: `KAFKA_LISTENER_EXECUTION_ENABLED` {% endhint %} ## Apache Kafka Setup You will need to set up a running Kafka cluster, with optional group IDs and Topics. ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Apache Kafka** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} Enter the **Bootstrap servers**. These are the host-port pairs that act as the starting point to access the full set of alive servers in the cluster. Enter your value with format `host:port` and click **Add element** to add as many elements as required. {% endstep %} {% step %} Enter the **Group ID** string, which uniquely identifies the group of consumer processes. Find this in your Kafka Cluster at **Home > Configuration > Consumer Properties**. {% endstep %} {% step %} We need to let the Listener know the **Topics** to connect to. Use `kafka-topics --bootstrap-server :9092 --describe` and write the result here. Click **Add element** to add as many topics as required. {% endstep %} {% step %} Now, you must choose an **Auto offset reset policy****\***. This policy defines the behavior when there are no committed positions available or when an offset is out of range. Choose between **Earliest**, **Latest**, or **None**. {% endstep %} {% step %} Next, you must define the **Authentication configuration**, or select **None** if no authentication is required. Choose between: {% endstep %} {% step %} * **Plain** - Enter your **Username****\*** and select your **Password****\*** from your [Secrets](/administration/global-settings/organization-settings/secrets-management) or create a new one. * **Scram** - Enter the required information: * **Username****\*** - Enter your username. * **Password****\*** - Select your password from your [Secrets](/administration/global-settings/organization-settings/secrets-management) or create a new one. * **SCRAM mechanism****\*** - Choose either **SHA-256** or **SHA-512**. * **mTLS** - Enter the required information: * **CA Certificate****\*** - Select your CA certificate from your [Secrets](/administration/global-settings/organization-settings/secrets-management) or create a new one. * **Client certificate****\*** - Select your client certificate from your [Secrets](/administration/global-settings/organization-settings/secrets-management) or create a new one. * **Client key****\*** - Select your client key from your [Secrets](/administration/global-settings/organization-settings/secrets-management) or create a new one. * **Skip verify** - Select **true** to skip or **false** to require verification. * **Server name** - Enter the name of the server to connect to. * **Minimum TLS version** - Select the required minimum version from the menu. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabelled**. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Output Ports The **Apache Kafka** Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. # Collect data from AWS products # Collect data from Amazon Kinesis Amazon Kinesis to Onum {% hint style="info" %} See the changelog of the **Amazon Kinesis** Listener [here](/listeners). {% endhint %} {% hint style="warning" %} * The **Amazon Kinesis** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster. {% endhint %} ## Overview Onum supports integration with [Amazon Kinesis Data Stream](https://aws.amazon.com//kinesis/data-streams/). Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that allows you to ingest, store, and process real-time data streams. It's designed for high-throughput, low-latency data ingestion from various sources, enabling real-time analytics and applications. Select **Amazon Kinesis** from the list of Listener types and click **Configuration** to start. ## Prerequisites {% hint style="warning" %} In order to use the **Amazon Kinesis** Listener, you must activate the following environment variable in your distributor using docker compose: `SINGLETON_LISTENER_EXECUTOR=true` {% endhint %} ## Amazon Kinesis Data Stream Setup {% stepper %} {% step %} Go to **IAM (Identity and Access Management)** to manage users, groups, roles and permissions. Under **Permissions Policies**, make sure you have assigned the policy `AmazonKinesisFullAccess` to give full access to Kinesis resources. Alternatively, if you have custom permissions, go to **Policies - Create Policy** and in the **JSON** tab, paste your custom JSON e.g. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow" "Action": [ "kinesis:CreateStream", "kinesis:DescribeStream", "kinesis:PutRecord" ], "Resource": "*" } ] } ``` {% endstep %} {% step %} Run the following command to test the configuration: ``` aws kinesis list-streams ``` If your IAM permission are correct, you'll see a list of streams. {% endstep %} {% endstepper %} ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Amazon Kinesis Data Stream** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} In the **Connection settings** section, click **Select region****\*** and choose the region of your AWS data center. Your region is displayed in the top right-hand corner of your AWS console. {% endstep %} {% step %} Click **Select access key ID****\*** and choose it from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management), or click **New secret** to generate a new one. The **Access Key ID** is found in the **IAM Dashboard** of the **AWS Management Console**. 1. In the left panel, click on **Users**. 2. Select your **IAM user**. 3. Under the **Security Credentials** tab, scroll to **Access Keys**, and you will find existing **Access Key IDs** (but not the secret access key). {% endstep %} {% step %} Click **Select secret key****\*** and choose it from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management), or click **New secret** to generate a new one. Under **Access keys**, you can see your **Access Key IDs**, but AWS **will not show the Secret Access Key**. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} {% endstep %} {% step %} You can enable a temporary external access enabling the **Assume Role** option. `AssumeRole` is an AWS STS (Security Token Service) action that allows an entity (user, service, or application) to temporarily assume an IAM role and obtain short-lived credentials. This role should have the following permissions to access Kinesis streams: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "KinesisReadAccess", "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetRecords", "kinesis:GetShardIterator", "kinesis:ListShards", "kinesis:SubscribeToShard", "kinesis:DescribeStreamSummary", "kinesis:RegisterStreamConsumer" ], "Resource": [ "" ] }, { "Sid": "KinesisListAccess", "Effect": "Allow", "Action": [ "kinesis:ListStreams", "kinesis:ListStreamConsumers" ], "Resource": "*" } ] } ``` AWS credentials and `AssumeRole` configuration can be configured specifically for Kinesis access. If credentials are not explicitly configured for Kinesis, the system will use the default AWS credentials. The configuration options are as follows: * **Role ARN****\*** - **Amazon Resource Name** used to access Kinesis resources. This is the unique identifier for the specific IAM Role that you want to assume and use (format: `arn:aws:iam::123456789012:role/KinesisReadRole`). * **External ID****\*** - Shared secret used to authenticate the usage of this role. * **Role session** - Name of the session, used to audit usage of this role (`kinesis-listener` by default) * **STS region** - If not set, it will use the Kinesis stream region. This specifies which region's STS endpoint to use when assuming the role. * **STS session duration** - How much the `AssumeRole` session will last before re-authentication. Uses Golang duration strings, like `1s`, `1m`, `1h`. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h. {% endstep %} {% step %} Enter the following information in the **Data Stream configuration** section: * **Stream name****\*** - Enter the unique identifier of your Kinesis Data Stream. To get it: 1. Go to: 2. Select **Data Streams** under **Amazon Kinesis** in the sidebar. 3. The **Stream name** will be in the first column e.g. `my-kinesis-stream-prod` * **Shard ID** - The shard is the basic unit of capacity in a Kinesis Data Stream, acting like a partition for your data stream and determining how your data is ingested, stored, and consumed. Click your Data Stream name to find your **Shard ID** in the **Shards** tab, e.g.: `shardId-000000000001` {% endstep %} {% step %} Activate the **Enable KMS Decryption** option if you want to automatically decrypt KMS-encrypted messages. If you activate it, you'll need to configure the following settings: * **Select KMS Key ID****\*** - Choose your KMS Key ID from your [Secrets](/administration/global-settings/organization-settings/secrets-management) or click **New secret** to define a new one. To find your KMS Key ID, go to the [KMS console](https://www.google.com/search?q=KMS+console\&sca_esv=fb6ec2c420ab31bb\&rlz=1C5GCCM_en\&ei=wmJvaeqqEMG0i-gP3efY4Qg\&ved=2ahUKEwjz4qXq_ZmSAxVt3QIHHRR6NToQgK4QegQIBBAB\&uact=5\&oq=FIND+MY+kms+kEY+id\&gs_lp=Egxnd3Mtd2l6LXNlcnAiEkZJTkQgTVkga21zIGtFWSBpZDIFECEYnwVIwx9QAFi4HXACeAGQAQCYAa0BoAGiE6oBBDAuMjC4AQPIAQD4AQGYAhagApgUqAIKwgIOEAAYgAQYsQMYgwEYigXCAggQABiABBixA8ICCxAAGIAEGLEDGIMBwgIOEC4YgAQYsQMY0QMYxwHCAgsQLhiABBjRAxjHAcICChAAGIAEGEMYigXCAhAQABiABBixAxhDGIMBGIoFwgIKEC4YgAQYQxiKBcICBRAAGIAEwgIQEAAYAxi0AhjqAhiPAdgBAcICDhAuGIAEGMcBGI4FGK8BwgILEC4YgAQYsQMYgwHCAggQLhiABBixA8ICERAuGIAEGLEDGNEDGIMBGMcBwgIdEC4YgAQYxwEYjgUYrwEYlwUY3AQY3gQY4ATYAQLCAg0QABiABBixAxhDGIoFwgIREC4YgAQYsQMYxwEYjgUYrwHCAgcQABiABBgTwgIIEAAYExgWGB7CAgoQABgTGBYYChgewgIFEAAY7wXCAgYQABgWGB7CAggQABiABBiiBJgDBPEFghVN0Fvc7rq6BgQIARgKugYGCAIQARgUkgcEMi4yMKAHkY4BsgcEMC4yMLgHkRTCBwYwLjcuMTXIB1KACAA\&sclient=gws-wiz-serp) and select **Customer managed keys**. You will find it in the **Key ID** column. * **Set decryption timeout** - Activate this option if you need to set a timeout for the decryption. Indicate it in the **Seconds****\*** field. * **Enable KMS encryption context** - You can set an optional set of non-secret key–value pairs that can contain additional contextual information about the data. Use the **Field****\*** and **Value**\* settings and click **Add field** to add the required pairs. {% endstep %} {% step %} In the **Advanced Configuration** section, activate the **Use compression** option if you need to compress your data. Choose the required type (**Gzip**, **Bzip2** or **Zlib**). {% endstep %} {% step %} Optionally, enter the **Custom endpoint** if you have a non-default URL that directs API requests to a specific Kinesis service endpoint. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Output Ports The **Amazon Kinesis** Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. # Collect data from Amazon S3 Amazon S3 to Onum {% hint style="info" %} See the changelog of the **Amazon S3** Listener [here](/listeners/amazon-s3-listener). {% endhint %} ## Overview **Amazon Simple Storage Service (S3)** is a fully managed object storage service. Users typically use it to store big files at a reasonable cost for long periods of time. In particular, it's commonly used as a data lake storage layer, storing files containing user events with some format/encoding/compression. Amazon S3 also supports sending notifications to an SQS queue when new files are added to some bucket. You can see a sample notification [here](https://github.com/nstrlabs/Actions/blob/main/actions/s3_listener/v1_2_0/internal/sample-notification.json). By leveraging all the above, our S3 Listener is able to react to new files being added to the bucket, get the files, and ingest their events into Onum. All that is needed is an existing SQS queue, an existing S3 bucket, and having the bucket correctly configured to send notifications to the queue. Using the **Amazon S3** Listener, you can read the following AWS content: * [**Collect AWS Application Logs**](/the-workspace/listeners/listener-integrations/collect-data-from-aws-products/collect-data-from-amazon-s3/collect-s3-application-logs) * [**Collect Bucket Content**](/the-workspace/listeners/listener-integrations/collect-data-from-aws-products/collect-data-from-amazon-s3/collect-s3-bucket-file-content) # Collect S3 Application Logs You can use the **Amazon S3** Listener to collect AWS application logs, such as: * **AWS CloudTrail** * **AWS CloudWatch** * **AWS WAF** ## Prerequisites Before configuring and starting to send data with the **Amazon S3** Listener, you need to take into consideration the following requirements: * Your Amazon user needs at least permission to use the `GetObject` operation (S3) and the `ReceiveMessage` and `DeleteMessageBatch` operations (SQS Bucket) to make this Listener work. * **Cross-Region Configurations** - Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets. * **Permissions** - Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions. * **Object Key Name Filtering** - If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded. {% hint style="warning" %} When S3 events flow through SNS before reaching SQS, enabling [**Raw Message Delivery**](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery [here](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html). {% endhint %} ## Amazon S3 Setup You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new logs are added. {% stepper %} {% step %} #### **Create an Amazon SQS Queue** * Sign in to the AWS Management Console and open the Amazon SQS console. * Choose **Create Queue** and configure the queue settings as needed. * After creating the queue, note its Amazon Resource Name (ARN), which follows this format: `arn:aws:sqs:::`. {% endstep %} {% step %} #### Modify the SQS Queue Policy to Allow S3 to Send Messages 1. In the Amazon SQS console, select your queue. 2. Navigate to the **Access Policy** tab and choose **Edit**. 3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details: ```json { "Version": "2012-10-17", "Id": "S3ToSQSPolicy", "Statement": [ { "Sid": "AllowS3Bucket", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "arn:aws:sqs:::", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:::" }, "StringEquals": { "aws:SourceAccount": "" } } } ] } ``` Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue. {% endstep %} {% step %} #### S3 Event Notification Rules for Logs Get notifications for CloudWatch, CloudTrail or WAF logs. * Open the Amazon S3 console and select the bucket you want to configure. * Go to the **Properties** tab and find the **Event notifications** section and configure according to type of log below. {% code title="Cloudwatch" %} ```json { "QueueConfigurations": [ { "Id": "CloudWatchLogNotification", "QueueArn": "arn:aws:sqs:::", "Events": ["s3:ObjectCreated:*"], "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "cloudwatch-logs/" }, { "Name": "suffix", "Value": ".log" } ] } } } ] } ``` {% endcode %} {% code title="Cloudtrail" %} ```json { "QueueConfigurations": [ { "Id": "CloudTrailLogNotification", "QueueArn": "arn:aws:sqs:::", "Events": ["s3:ObjectCreated:*"], "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "AWSLogs/" }, { "Name": "suffix", "Value": ".json.gz" } ] } } } ] } ``` {% endcode %} {% code title="WAF" %} ```json { "QueueConfigurations": [ { "Id": "WAFLogNotification", "QueueArn": "arn:aws:sqs:::", "Events": ["s3:ObjectCreated:*"], "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "aws-waf-logs/" }, { "Name": "suffix", "Value": ".gz" } ] } } } ] } ``` {% endcode %} {% endstep %} {% step %} #### Test the Configuration 1. Upload a new file to your S3 bucket. 2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly. {% endstep %} {% endstepper %} ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Amazon S3** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} In the **Objects** section, enter the required compression method used in the ingested S3 files and format of the ingested S3 files. * **Compression****\*** - This accepts the standard compression codecs (**gzip**, **zlib**, **bzip2**), **none** for no compression, and **auto** to autodetect the compression type from the file extension. * **Path pattern****\*** - This allows you to filter which S3 objects are processed based on their key (path). It accepts Go regular expressions (RE2 syntax), which are matched against the full object key after URL-decoding. Objects whose key does not match the pattern are silently skipped and their SQS notification is still acknowledged, preventing poison-pill scenarios where non-processable files (e.g. CloudTrail digest files) block the queue. This field is required and the default value is `.*` (matches everything, backward-compatible with previous versions that did not have this filter). Find some examples below:
RegExDescription
.*Matches all keys (default, backward-compatible)
^data/.*\.json(\.gz)?$Only JSON (or gzipped JSON) files under data/
^(?!.*Digest).*$Excludes CloudTrail digest files
CloudTrail/[^/]+/.*\.json\.gz$Only CloudTrail log files (not digests)
^logs/Only objects with the logs/ prefix
\.(csv|parquet)$Only CSV or Parquet files regardless of path
^[^/]+/[^/]+/year=2024/Only objects partitioned under year=2024
* **Format****\*** - This currently accepts **JSON**, **JSON lines** (a JSON object representing an event on each line), **Text** and **CSV**. If you select **JSON,** **CSV**, or **Text** more options appear:
JSON Options * **Path** - Enter the path of the JSON element you want to retrieve. The path are keys separated by dots. For example `one.two.three` will select the element in `{"one":{"two":{"three":[1,2,3]}}`. To select the root, you can leave the path empty or enter a single dot `.` (default option). {% hint style="warning" %} For CloudWatch and CloudTrail logs, you must enter the `.Records` path. {% endhint %} * **Array Unroll** - Activate this toggle if you want to generate one event for each element in the array. The element that the path points to must be a JSON array.
CSV Options * **Header Row** - Select **true** to include a header for your CSV rows. * **Delimiter****\*** - Decide between **comma**, **semicolon** and **tab**. * **Text Encoding** - Choose the scheme (or key) that maps the characters used to store and read the data in the CSV file. * **Output Format** - Choose either **CSV** or **JSON**. * **JSON Output:** * Converts CSV records to structured JSON objects * Field names are derived from header row (if present) or auto-generated (`field_0`, `field_1`, etc.) * Provides structured data for easier processing in Pipelines. * **CSV Output:** * Preserves original CSV formatting. * Each CSV record becomes a separate event containing the raw CSV line. * Useful when you want to maintain the original CSV structure. * **Trim Leading Space** - Select **true** to remove any whitespace characters that appear immediately before the first non-whitespace character in a cell. * **Lazy Quotes** - Select **true** to allow double quotes to appear in fields without strictly following the formal CSV escaping rules. * **Fields per Record** - The number of fields expected in each row (record) of your CSV file. * **Comment Character** - Use the hash symbol (`#`) to designate lines of text that should be ignored during the data parsing process.
Text Options * **Framing Method** - Select between **Non Transparent Framing (newline, zero, or none)** and **Octet counting**
{% endstep %} {% step %} Define the **Bucket** to listen from: * **Region****\*** - Find this in your **Buckets** area, next to the name. * **Name** - The [AWS bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#:~:text=A%20bucket%20is%20a%20container,the%20object%20within%20the%20bucket.) your data is stored in. This is the bucket name found in your **Buckets** area. You can fill this if you want to check that notifications come from that bucket, or leave it empty to avoid such checks. * **Authentication Type****\*** - Choose **manual** to enter your access key ID and secret access key manually in the parameters below, or **auto** to authenticate automatically. The default value is **manual**. {% hint style="warning" %} If you select the **auto** authentication and you're working on an Onum cloud deployment, please [contact Support](https://docs.onum.com/support) to request the ARN information for your Onum-managed instance. {% endhint %} * **Access key ID****\*** - Select the access key ID from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or click **New secret** to generate a new one. The **Access Key ID** is found in the **IAM Dashboard** of the **AWS Management Console**. 1. In the left panel, click on **Users**. 2. Select your **IAM user**. 3. Under the **Security Credentials** tab, scroll to **Access Keys**, and you will find existing **Access Key IDs** (but not the secret access key). * **Secret access key****\***- Select the secret access key from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or click **New secret** to generate a new one. Under **Access keys**, you can see your **Access Key IDs**, but AWS **will not show the Secret Access Key**. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one. {% endstep %} {% step %} You can enable a temporary external access enabling the **Assume Role** option. `AssumeRole` is an AWS STS (Security Token Service) action that allows an entity (user, service, or application) to temporarily assume an IAM role and obtain short-lived credentials. This role should have the following permissions to access the required S3 bucket: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "S3ReadAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetBucketNotification" ], "Resource": [ "" ] }, { "Sid": "SQSAccess", "Effect": "Allow", "Action": [ "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:DeleteMessageBatch", "sqs:GetQueueAttributes", "sqs:GetQueueUrl" ], "Resource": [ "" ] } ] } ``` AWS credentials and `AssumeRole` configuration associated can be configured individually for S3 and SQS but if SQS credentials are not configured it will use the ones for S3, `AssumeRole` configuration included. **To add a trust policy:** 1. Navigate to the **AWS IAM Console** 2. Click **Roles** in the left panel 3. Click **Create Role** 4. Under **Trusted Entity Type**, select **AWS Account** or **Another AWS Account** for cross-account access 5. Enter the **Account ID** of the entity that will assume the role 6. Click **Next** When creating the role, configure the trust policy to specify which entity can assume it. **We only support Cross-Account Access:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "CrossAccountTrustPolicy", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` **Attach the following permissions policy to the role to grant access to S3 and SQS:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "S3ReadAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetBucketNotification" ], "Resource": [ "" ] }, { "Sid": "SQSAccess", "Effect": "Allow", "Action": [ "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:DeleteMessageBatch", "sqs:GetQueueAttributes", "sqs:GetQueueUrl" ], "Resource": [ "" ] } ] } ``` **Configure AssumeRole** The configuration options are as follows: * **Role ARN****\*** - **Amazon Resource Name** used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use. * **External ID** - shared secret used to authenticate the usage of this role. * **Role Session** - name of the session, used to audit usage of this role ( `s3-listener` by default) * **STS Region** - if not set it will use the bucket or the queue region. * **STS Session Duration** - how much the `AssumeRole` session will last before reauthentication. Uses Golang duration strings, like `1s`, `1m`, `1h`. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h. {% endstep %} {% step %} Optionally, you can enter a **Service endpoint** in the **Bucket Advanced** section. Amazon S3 provides different types of service endpoints based on the region and access type. To get it: 1. Select your **bucket**. 2. Go to the **Properties** tab. 3. Under **Bucket ARN & URL**, find the S3 **endpoint URL**. {% hint style="warning" %} Proceed with caution when modifying the **Bucket Advanced** options. Default values should be enough in most cases. Amazon Service Endpoint will usually be chosen automatically, so you should not normally have to fill this up. However, in case you need to override the default access point, you can do it here. {% endhint %} {% endstep %} {% step %} In the **Queue** section, enter the following information: * **Region** - Choose the region your queue is created in from the dropdown provided. * **URL****\*** - Enter the URL of your existing Amazon SQS queue to send the data to. To find it: * Go to the **AWS Management Console**. * In the **Search Bar**, type **SQS** and click on **Simple Queue Service (SQS)**. * Click on **Queues** in the left panel. * Locate your queue from the list and click it. * The **Queue URL** will be displayed in the table under **URL**. This is the correct URL format: `https://sqs.region.localhost/awsaccountnumber/storedinenvvar` * **Authentication Type****\*** - Choose **manual** to enter your access key ID and secret access key manually in the parameters below, or **auto** to authenticate automatically. * **Event name** - Name of the S3 event that triggers the notification. If not specified, S3 will capture all events. {% endstep %} {% step %} Optionally, configure the following **Queue Advanced** options: {% hint style="warning" %} Proceed with caution when modifying the **Queue Advanced** options. Default values should be enough in most cases. {% endhint %} * **Service endpoint** - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default. * **Maximum number of messages****\*** - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `1`, and the maximum and default value is `10`. * **Visibility timeout****\*** - Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is `30s`, and the maximum value is `12h`. The default value is `1h`. * **Wait time****\***- When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is `5s`, and the maximum and default value is `20s`. {% endstep %} {% step %} Optionally, configure the following **General Advanced** options: {% hint style="warning" %} Proceed with caution when modifying the **General Advanced** options. Default values should be enough in most cases. {% endhint %} * **Event batch size****\*** - Enter a limit for the number of events allowed through per batch. The minimum value is `1`, and the maximum and default value is `1000000`. * **Minimum retry time****\*** - Set the minimum amount of time to wait before retrying. The default and minimum value is `1s`, and the maximum value is `10m`. * **Maximum retry time****\*** - Set the maximum amount of time to wait before retrying. The default value is `5m`, and the maximum value is `10m`. The minimum value is the one set in the parameter above. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabelled**. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Output Ports The **Amazon S3** Listener has only a single output port: * **Default port** - Events are sent through this port if no error occurs while processing them. # Collect S3 Bucket File Content You can send the files in your Amazon S3 buckets to Onum using the **Amazon S3** Listener. ## Prerequisites Before configuring and starting to send data with the **Amazon S3** Listener, you need to take into consideration the following requirements: * Your Amazon user needs at least permission to use the `GetObject` operation (S3) and the `ReceiveMessage` and `DeleteMessageBatch` operations (SQS Bucket) to make this Listener work. * **Cross-Region Configurations** - Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets. * **Permissions** - Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions. * **Object Key Name Filtering** - If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded. {% hint style="warning" %} When S3 events flow through SNS before reaching SQS, enabling [**Raw Message Delivery**](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery [here](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html). {% endhint %} ## Amazon S3 Setup You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new files are added. {% stepper %} {% step %} #### **Create an Amazon SQS Queue** * Sign in to the AWS Management Console and open the Amazon SQS console. * Choose **Create Queue** and configure the queue settings as needed. * After creating the queue, note its Amazon Resource Name (ARN), which follows this format: `arn:aws:sqs:::`. {% endstep %} {% step %} #### Modify the SQS Queue Policy to Allow S3 to Send Messages 1. In the Amazon SQS console, select your queue. 2. Navigate to the **Access Policy** tab and choose **Edit**. 3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details: ```json { "Version": "2012-10-17", "Id": "S3ToSQSPolicy", "Statement": [ { "Sid": "AllowS3Bucket", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "arn:aws:sqs:::", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:::" }, "StringEquals": { "aws:SourceAccount": "" } } } ] } ``` Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue. {% endstep %} {% step %} #### S3 Event Notification Rules for Files Get notified when files are added or modified using the following JSON: ```json { "QueueConfigurations": [ { "Id": "FileUploadNotification", "QueueArn": "arn:aws:sqs:::", "Events": ["s3:ObjectCreated:*"], "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "files/" } ] } } } ] } ``` 1. Open the Amazon S3 console and select the bucket you want to configure. 2. Go to the **Properties** tab and find the "Event notifications" section. 3. Click on **Create event notification**. 4. Provide a descriptive name for the event notification. 5. In the **Event types** section, enter the JSON. 6. In the **Destination** section, choose **SQS Queue** and select the queue you configured earlier. 7. Save the configuration. {% endstep %} {% step %} #### Test the Configuration 1. Upload a new file to your S3 bucket. 2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly. {% endstep %} {% endstepper %} ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Amazon S3** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} In the **Objects** section, enter the required **Compression** method used in the ingested S3 files and **Format** of the ingested S3 files. * **Compression****\*** - This accepts the standard compression codecs (**gzip**, **zlib**, **bzip2**), **none** for no compression, and **auto** to autodetect the compression type from the file extension. * **Path pattern****\*** - This allows you to filter which S3 objects are processed based on their key (path). It accepts Go regular expressions (RE2 syntax), which are matched against the full object key after URL-decoding. Objects whose key does not match the pattern are silently skipped and their SQS notification is still acknowledged, preventing poison-pill scenarios where non-processable files (e.g. CloudTrail digest files) block the queue. This field is required and the default value is `.*` (matches everything, backward-compatible with previous versions that did not have this filter). Find some examples below:
RegExDescription
.*Matches all keys (default, backward-compatible)
^data/.*\.json(\.gz)?$Only JSON (or gzipped JSON) files under data/
^(?!.*Digest).*$Excludes CloudTrail digest files
CloudTrail/[^/]+/.*\.json\.gz$Only CloudTrail log files (not digests)
^logs/Only objects with the logs/ prefix
\.(csv|parquet)$Only CSV or Parquet files regardless of path
^[^/]+/[^/]+/year=2024/Only objects partitioned under year=2024
* **Format****\*** - This currently accepts **JSON**, **JSON lines** (a JSON object representing an event on each line), **Text** and **CSV**. If you select **JSON, Text** or **CSV**, more options appear:
JSON Options * **Path** - Enter the path of the JSON element you want to retrieve. The path are keys separated by dots. For example `one.two.three` will select the element in `{"one":{"two":{"three":[1,2,3]}}`. To select the root, you can leave the path empty or enter a single dot `.` (default option). {% hint style="warning" %} For CloudWatch and CloudTrail logs, you must enter the `.Records` path. {% endhint %} * **Array Unroll** - Activate this toggle if you want to generate one event for each element in the array. The element that the path points to must be a JSON array.
CSV Options * **Header Row** - Select **true** to include a header for your CSV rows. * **Delimiter****\*** - Decide between **comma**, **semicolon** and **tab**. * **Text Encoding** - Choose the scheme (or key) that maps the characters used to store and read the data in the CSV file. * **Output Format** - Choose either **CSV** or **JSON**. * **JSON Output:** * Converts CSV records to structured JSON objects * Field names are derived from header row (if present) or auto-generated (`field_0`, `field_1`, etc.) * Provides structured data for easier processing in Pipelines. * **CSV Output:** * Preserves original CSV formatting. * Each CSV record becomes a separate event containing the raw CSV line. * Useful when you want to maintain the original CSV structure. * **Trim Leading Space** - Select **true** to remove any whitespace characters that appear immediately before the first non-whitespace character in a cell. * **Lazy Quotes** - Select **true** to allow double quotes to appear in fields without strictly following the formal CSV escaping rules. * **Fields per Record** - The number of fields expected in each row (record) of your CSV file. * **Comment Character** - Use the hash symbol (`#`) to designate lines of text that should be ignored during the data parsing process.
Text Options * **Framing Method** - Select between **Non Transparent Framing (newline, zero, or none)** and **Octet counting**
{% endstep %} {% step %} Define the **Bucket** to listen from. * **Region****\*** - Find this in your **Buckets** area, next to the name. * **Name** - The [AWS bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#:~:text=A%20bucket%20is%20a%20container,the%20object%20within%20the%20bucket.) your data is stored in. This is the bucket name found in your **Buckets** area. You can fill this if you want to check that notifications come from that bucket, or leave it empty to avoid such checks. * **Authentication Type****\***- Choose **manual** to enter your access key ID and secret access key manually in the parameters below, or **auto** to authenticate automatically. The default value is **manual**. {% hint style="warning" %} If you select the **auto** authentication and you're working on an Onum cloud deployment, please [contact Support](https://docs.onum.com/support) to request the ARN information for your Onum-managed instance. {% endhint %} * **Access key ID****\***- Select the access key ID from your [Secrets](/administration/tenant-menu) or click **New secret** to generate a new one. The **Access Key ID** is found in the **IAM Dashboard** of the **AWS Management Console**. 1. In the left panel, click on **Users**. 2. Select your **IAM user**. 3. Under the **Security Credentials** tab, scroll to **Access Keys**, and you will find existing **Access Key IDs** (but not the secret access key). * **Secret access key****\***- Select the secret access key from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or click **New secret** to generate a new one. Under **Access keys**, you can see your **Access Key IDs**, but AWS **will not show the Secret Access Key**. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one. {% endstep %} {% step %} You can enable a temporary external access enabling the **Assume Role** option. `AssumeRole` is an AWS STS (Security Token Service) action that allows an entity (user, service, or application) to temporarily assume an IAM role and obtain short-lived credentials. This role should have the following permissions to access both the S3 bucket and the configured SQS notification queue: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "S3ReadAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetBucketNotification" ], "Resource": [ "" ] }, { "Sid": "SQSAccess", "Effect": "Allow", "Action": [ "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:DeleteMessageBatch", "sqs:GetQueueAttributes", "sqs:GetQueueUrl" ], "Resource": [ "" ] } ] } ``` AWS credentials and `AssumeRole` configuration associated can be configured individually for S3 and SQS but if SQS credentials are not configured it will use the ones for S3, `AssumeRole` configuration included. **To add a trust policy:** 1. Navigate to the **AWS IAM Console** 2. Click **Roles** in the left panel 3. Click **Create Role** 4. Under **Trusted Entity Type**, select **AWS Account** or **Another AWS Account** for cross-account access 5. Enter the **Account ID** of the entity that will assume the role 6. Click **Next** When creating the role, configure the trust policy to specify which entity can assume it. **We only support Cross-Account Access:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "CrossAccountTrustPolicy", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` **Attach the following permissions policy to the role to grant access to S3 and SQS:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "S3ReadAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetBucketNotification ``` The configuration options are as follows: * **Role ARN****\*** - **Amazon Resource Name** used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use. * **External ID** - shared secret used to authenticate the usage of this role. * **Role Session** - name of the session, used to audit usage of this role ( `s3-listener` by default) * **STS Region** - if not set it will use the bucket or the queue region. * **STS Session Duration** - how much the `AssumeRole` session will last before reauthentication. Uses Golang duration strings, like `1s`, `1m`, `1h`. If not Set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h. {% endstep %} {% step %} Optionally, you can enter a **Service endpoint** in the **Bucket Advanced** section. Amazon S3 provides different types of **service endpoints** based on the **region and access type**. 1. Select your **bucket**. 2. Go to the **Properties** tab. 3. Under **Bucket ARN & URL**, find the S3 **endpoint URL**. {% hint style="warning" %} Proceed with caution when modifying the **Bucket Advanced** options. Default values should be enough in most cases. Amazon Service Endpoint will usually be chosen automatically, so you should not normally have to fill this up. However, in case you need to override the default access point, you can do it here. {% endhint %} {% endstep %} {% step %} In the **Queue** section, enter the following information: * **Region** - Choose the region your queue is created in from the dropdown provided. * **URL****\*** - Enter the URL of your existing Amazon SQS queue to send the data to. To find it: * Go to the **AWS Management Console**. * In the **Search Bar**, type **SQS** and click on **Simple Queue Service (SQS)**. * Click on **Queues** in the left panel. * Locate your queue from the list and click it. * The **Queue URL** will be displayed in the table under **URL**. This is the correct URL format: `https://sqs.region.localhost/awsaccountnumber/storedinenvvar` * **Authentication Type****\*** - Choose **manual** to enter your access key ID and secret access key manually in the parameters below, or **auto** to authenticate automatically. * **Event name** - Name of the S3 event that triggers the notification. If not specified, S3 will capture all events. {% endstep %} {% step %} Optionally, configure the following **Queue Advanced** options: {% hint style="warning" %} Proceed with caution when modifying the **Queue Advanced** options. Default values should be enough in most cases. {% endhint %} * **Service endpoint** - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default. * **Maximum number of messages****\*** - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `1`, and the maximum and default value is `10`. * **Visibility timeout****\*** - Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is `30s`, and the maximum value is `12h`. The default value is `1h`. * **Wait time****\***- When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is `5s`, and the maximum and default value is `20s`. {% endstep %} {% step %} Optionally, configure the following **General Advanced** options: {% hint style="warning" %} Proceed with caution when modifying the **General Advanced** options. Default values should be enough in most cases. {% endhint %} * **Event batch size****\***- Enter a limit for the number of events allowed through per batch. The minimum value is `1`, and the maximum and default value is `1000000`. * **Minimum retry time****\*** - Set the minimum amount of time to wait before retrying. The default and minimum value is `1s`, and the maximum value is `10m`. * **Maximum retry time****\*** - Set the maximum amount of time to wait before retrying. The default value is `5m`, and the maximum value is `10m`. The minimum value is the one set in the parameter above. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabelled**. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Output Ports The **Amazon S3** Listener has only a single output port: * **Default port** - Events are sent through this port if no error occurs while processing them. # Collect data from Amazon SQS Amazon Kinesis to Onum {% hint style="info" %} See the changelog of the **Amazon SQS** Listener [here](/listeners/amazon-sqs-listener). {% endhint %} {% hint style="warning" %} The **Amazon SQS** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster. {% endhint %} ## Overview Onum supports integration with [Amazon SQS](https://aws.amazon.com/sqs/?nc1=h_ls). Amazon Simple Queue Service (AWS SQS) is a fully managed message queuing service. Among its many features, the following ones are of special interest to our use case: * It supports both standard queues (with at-least-once, occasionally unordered delivery semantics) and FIFO queues (exactly-once and fully ordered delivery semantics). * It supports scaling through the concept of visibility timeout (a period after a consumer reads one message during which this becomes invisible to other consumers). That allows a consumer group to read from the same queue and distribute messages without duplication. So, what we have is a Listener that we can configure to **read from an existing SQS queue and inject queue messages as events into our platform**. Please note that because of the nature of the API offered to access SQS messages (HTTP-based, max 10 messages each time), this is not a high-throughput Listener. Select **Amazon SQS** from the list of Listener types and click **Configuration** to start. ## Prerequisites You will need an **IAM** User, role or group with the correct permissions to access and manage SQS. {% hint style="warning" %} When S3 events flow through SNS before reaching SQS, enabling [**Raw Message Delivery**](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery [here](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html). {% endhint %} ## Amazon SQS Setup **Go to IAM (Identity and Access Management)** to manage users, groups, roles and permissions. Under **Permissions Policies**, make sure you have assigned the policy `AmazonSQSFullAccess` to give full access to SQS resources. Alternatively, if you have custom permissions, go to **Policies > Create Policy** and in the **JSON** tab, paste your custom JSON. For example: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:CreateQueue", "sqs:GetQueueAttributes", "sqs:SendMessage" ], "Resource": "*" } ] } ``` ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Amazon SQS** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} Enter the **Region** displayed in the top right-hand corner of your AWS console. {% endstep %} {% step %} Enter the **Queue URL****\*** of your existing Amazon SQS queue, acting as the endpoint to interact with the desired queue. Use the `GetQueueUrl` command or: 1. **Go to the** AWS Management Console. 2. In the **Search Bar**, type `SQS` and click on **Simple Queue Service (SQS)**. 3. Click on **Queues** in the left panel. 4. Locate your queue from the list and click it. 5. The **Queue URL** will be displayed in the table under **URL**. This is the correct URL format: `sqs.region.localhost/awsaccountnumber/storedinenvv` {% endstep %} {% step %} Now, choose your **Authentication type****\*** between **manual** and **auto**: * **Manual** - Choose this option to manually enter your AWS credentials: * **Access key ID****\*** - Add the access key from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or create one. The **Access Key ID** is found in the **IAM Dashboard** of the **AWS Management Console**. * In the left panel, click on **Users**.Select your **IAM user**. * Under the **Security Credentials** tab, scroll to **Access Keys** and you will find existing **Access Key IDs** (but not the secret access key). * **Secret access key****\*** - Add the secret access key from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or create one. Under **Access keys**, you can see your **Access Key IDs**, but AWS **will not show the Secret Access Key**. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one. * **Auto** - Choose this option if your IAM role is attached to the host. {% endstep %} {% step %} You can enable a temporary external access enabling the **Assume Role** option. `AssumeRole` is an AWS STS (Security Token Service) action that allows an entity (user, service, or application) to temporarily assume an IAM role and obtain short-lived credentials. The configuration options are as follows: * **Role ARN****\*** - **Amazon Resource Name** used to access Kinesis resources. This is the unique identifier for the specific IAM Role that you want to assume and use. * **External ID****\*** - Shared secret used to authenticate the usage of this role. * **Role session** - Name of the session, used to audit usage of this role. * **STS region** - If not set, it will use the Kinesis stream region. This specifies which region's STS endpoint to use when assuming the role. * **STS session duration** - How much the `AssumeRole` session will last before re-authentication. Uses Golang duration strings, like `1s`, `1m`, `1h`. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h. {% endstep %} {% step %} Optionally, specify which **Message system attributes** are wanted in the response. The set of system attributes chosen by the user correspond to attributes inlined in the message/event. 1. In the **Queues** area, click on **More** or scroll down and go to the **Monitoring** tab. 2. You will see some system attributes (like deduplication and group ID). However, detailed system attributes are typically accessed via the CLI or SDKs. {% endstep %} {% step %} Optionally, configure the settings in the **Advanced** section: {% hint style="warning" %} Proceed with caution when modifying the **Advanced** options. Default values should be enough in most cases. {% endhint %} * **Service endpoint** - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default. * **Maximum number of messages****\*** - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `1`, and the maximum and default value is `10`. * **Visibility timeout****\*** - The time during which messages delivered to a consumer, but not yet acknowledged, are hidden from other consumers. Valid values go from `30s` to `12h`, and the default value is `2m`. * **Wait time****\*** - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `5`, and the maximum and default value is `10`. * **Minimum retry time****\*** - Set the minimum amount of time to wait before retrying. The default and minimum value is `1s`, and the maximum value is `10m`. * **Maximum retry time****\*** - Set the minimum amount of time to wait before retrying. The default and minimum value is `1s`, and the maximum value is `10m`. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabelled**. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} # Collect data from Cisco NetFlow Most recent version: v0.1.0 {% hint style="info" %} See the changelog of the **Cisco NetFlow** Listener [here](/listeners/cisco-netflow-listener). {% endhint %} {% hint style="warning" %} The **Cisco NetFlow** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster. {% endhint %} ## Overview Onum supports integration with [Cisco NetFlow](https://www.cisco.com/site/us/en/index.html). Cisco NetFlow is a network protocol developed by Cisco for collecting and analyzing IP network traffic data. It enables network administrators to understand traffic patterns, identify potential issues, and optimize network performance. Select **Cisco NetFlow** from the list of Listener types and click **Configuration** to start. ## Cisco NetFlow setup In order to begin listening for data, you must first: * Enable IP routing * Enable Cisco Express Forwarding (CEF) See the [Cisco Netflow configuration guide](https://www.cisco.com/c/dam/en/us/td/docs/security/stealthwatch/netflow/Cisco_NetFlow_Configuration.pdf) for help with this. ## Onum setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Cisco NetFlow** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} In the **Socket** section, enter the following: * **Transport protocol****\*** - Currently, Onum only supports the UDP protocol. * **Port****\*** - Enter the required IP port number. By default, **Cisco NetFlow** typically uses UDP port `2055` for exporting flow data. {% endstep %} {% step %} Configure the **Flow** parameters: * **Protocols to process****\*** - Select the required protocol(s) from the list: * `NetflowV5` is the most widely used version. * `NetflowV9` is more customizable than v5. * `IPFIX` is based on the IPFIX standard (IP Flow Information Export). * `sFlowv5` is another flow monitoring protocol that is typically used in high-speed networks. * **Fields to include****\*** - Select all the fields you wish to include in the output data. {% hint style="warning" %} Field selection should match the fields actually present in your data. Selecting non-existing fields will result in `null` values and may cause unexpected behavior.\ \ [Check the table below](#proposed-field-sets) for safe field sets proposals for each protocol. {% endhint %} {% endstep %} {% step %} Choose your **Access control type****\*** to selectively monitor traffic based on specific IPs: * **None** - allows all IPs. * **Whitelist** - allows certain IPs through. * **Blacklist** - blocks certain IPs from being captured or exported. {% endstep %} {% step %} Enter the **IPs** you wish to apply the access control to. Click **Add element** to add as many as required. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Proposed field sets Check the following table for recommended safe field sets per protocol (NetflowV5, NetflowV9, IPFIX and sFlowv5). We recommend using as a baseline to work from rather than selecting all available fields by default. {% hint style="warning" %} For NetflowV9 and IPFIX, since both are template-based protocols, the safe field set will always depend on the fields declared in the exporter's template. As a general rule, only select fields that you have confirmed are present in your exporter's template. Selecting fields not declared in the template will result in those fields decoding as `null` and may cause unexpected behavior during Listener recovery or restart. {% endhint %}
NetflowV5

NetflowV5 has a completely fixed format with no templates, meaning the following fields are always present and safe to use in any standard implementation:

  • SrcAddr
  • DstAddr
  • SrcPort
  • DstPort
  • Proto
  • Bytes
  • Packets
  • TimeFlowStart
  • TimeFlowEnd
  • InIf
  • OutIf
  • IPTos
  • SequenceNum
  • SamplingRate
  • SamplerAddress
  • TimeReceived
NetflowV9

NetflowV9 is dynamic and depends on the template declared by the exporter. The fields most commonly exported by standard Cisco devices and considered safe as a baseline are:

  • SrcAddr
  • DstAddr
  • SrcPort
  • DstPort
  • Proto
  • Bytes
  • Packets
  • TimeFlowStart - flowStartMilliseconds
  • TimeFlowEnd - flowEndMilliseconds
  • InIf
  • OutIf
  • IPTos
  • SequenceNum
  • SamplerAddress
  • TimeReceived


These groups of fields that should not be selected unless explicitly declared in the exporter's template:

  • MPLS
  • PPP
  • encap
  • IPv6
  • VLAN
  • MAC
  • Fragment
  • VRF
IPFIX

Like NetflowV9, IPFIX is template-based. The recommended core safe fields are practically the same:

  • SrcAddr
  • DstAddr
  • SrcPort
  • DstPort
  • Proto
  • Bytes
  • Packets
  • TimeFlowStart
  • TimeFlowEnd
  • InIf
  • OutIf
  • IPTos
  • SequenceNum
  • SamplerAddress
  • TimeReceived
sFlowv5

sFlow is a sampling protocol with a different structure to NetFlow. The most commonly available fields are:

  • SrcAddr
  • DstAddr
  • SrcPort
  • DstPort
  • Proto
  • Bytes
  • Packets
  • InIf
  • OutIf
  • TimeReceived
  • SamplingRate
  • SamplerAddress
# Collect data from Cloudflare Most recent version: v0.0.1 {% hint style="info" %} See the changelog of this Listener type [here](https://docs.onum.com/listeners/cloudflare). {% endhint %} {% hint style="warning" %} This Listener is only available in certain Tenants. {% endhint %} ## Overview Onum supports integration with [Cloudflare](https://www.cloudflare.com/). Select **Cloudflare** from the list of Listener types and click **Configuration** to start. ## Prerequisites [Contact Onum](/support) to get the required JWT token, which will be needed on the Listener setup. You can also contact us if you cannot generate the required TLS certificates. Note that these certificates must be signed by a recognized Certificate Authority (CA). Self-signed certificates are not accepted. ## Cloudflare Setup Cloudflare Logpush supports the ability to send logs to configurable HTTP endpoints. Follow [these instructions](https://developers.cloudflare.com/logs/logpush/logpush-job/enable-destinations/http/) to enable log sending before starting the configuration in Onum. ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Cloudflare** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} In the **Socket** section, enter the required **Port**. The port can be found in your Cloudflare dashboard under the **Network** tab. For custom applications, check your Cloudflare for Teams configuration under **Access >** **Applications** or **Gateway** > **Policies** for socket endpoints. {% endstep %} {% step %} In the **Authentication** section, enter the token. To find your Cloudflare authentication token: * Log in to the Cloudflare dashboard * Click on your profile icon in the top-right corner * Select **My Profile** from the dropdown menu * Navigate to the **API Tokens** tab and either: * Use an existing token (view its permissions to ensure it has appropriate access) * Create a new token by clicking **Create Token** * Use the **Global API Key** (found under **API Keys** section) for full account access Remember that tokens are only displayed once at creation time, so copy and store them securely. {% endstep %} {% step %} Open the Token **Secret** field and click **New secret** to create a new one: * Give the token a Name. * Turn off the Expiration date option. * Click **Add new value** and paste the secret corresponding to the Cloudflare token you received. * Click Save. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} {% endstep %} {% step %} You can now select the secret you just created in the **Token Secret** field. {% endstep %} {% step %} In the **TLS configuration** section, enter the required certificates (**Certificate**, **Private key** and **CA chain**). {% hint style="warning" %} Certificates must be signed by a recognized Certificate Authority (CA). Self-signed certificates are not accepted. {% endhint %} {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% endstepper %} Click **Create listener** when you're done. # Collect data from Falcon LogScale Falcon LogScale Collector to Onum {% hint style="info" %} See the changelog of the **Falcon LogScale Collector** Listener [here](/listeners/falcon-logscale-collector-listener). {% endhint %} ## Overview The following article outlines a basic data flow from **Falcon LogScale Collector** to the Onum **Falcon LogScale Collector** Listener. ## Prerequisites * In most cloud-based Onum installations, the **TLS configuration** section of the **Falcon LogScale Collector** Listener **is not visible** and you won't need to enter these values. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If your **Falcon LogScale Collector** Listener configuration requires you to manually enter these TLS certificates, you can generate them following the instructions in [this article](/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). * You'll need to know your Onum distributor URL, as it will be required in the Falcon LogScale Collector setup. In most cloud-based Onum installations, the Onum distributor URL will be displayed in the Listener details once you create it. Click your Listener in the **Listeners** area and find it under the **Address** section. If you cannot see it, [contact us](https://docs.onum.com/support/) and we'll send it to you. ## Onum setup First, you must configure a new **Falcon LogScale Collector** Listener in Onum: {% stepper %} {% step %} In Onum, go to the **Listeners** area and click **New listener**. Select the **Falcon LogScale Collector** Listener from the list. {% endstep %} {% step %} Enter a **Name** for the Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} For most cloud-based Onum installations, the **Socket** section **is not visible**, and port `443` is used by default. If you see it, enter the required port in the **Port** field. At this time, all TCP ports from `1024` to `10000` are open. {% endstep %} {% step %} Now you need to generate a token that will be used to connect Onum to your Falcon LogScale Collector instance. You can use an [online UUID generator tool](https://www.uuidgenerator.net/) to get it. {% hint style="warning" %} Note that the Falcon LogScale Collector won’t allow for token values that are just numeric. {% endhint %} Back to Onum, go to the **Authentication** section, click the **Select an** **API Key** field and select **New secret**. In the window that appears, give your secret a **Name** and turn off the **Expiration date** toggle if not needed. Then, click **Add new value** and paste the token you generated. Click **Save** when you're done. You'll later use this token in the Falcon LogScale Collector configuration. {% hint style="info" %} Learn more about Secrets in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} {% endstep %} {% step %} Now, select the token you've just created. {% endstep %} {% step %} In most cloud-based Onum installations, the **TLS configuration** section is not visible. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If you see this section, you must enter the required **Certificate**, **Private key** and **CA Chain**. Learn how to generate these self-signed certificates in [this article](/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). Once you have them, click **New secret** in each field and add the corresponding values. {% endstep %} {% step %} Now there are two possible scenarios: * If you didn't enter your TLS certificates, click **Create listener** and you'll see the **Network configuration** screen, which shows the **Address** and **Port** needed to communicate with Onum. You can also download the certificate in case you need it. {% hint style="info" %} You can access all this information in the Listener details after creation, so don't worry. {% endhint %} * If you entered the TLS certificates, you'll go directly to the next step to create the Listener labels. {% endstep %} {% step %} Finally, click **Create labels**. Create any required [labels](/the-workspace/listeners/labels) if you need to break down your data and then click **Create listener**. {% endstep %} {% endstepper %} ## Falcon LogScale Collector setup Now, access your Falcon instance and follow these steps: {% stepper %} {% step %} In the left menu, click **Data connectors > Data connections** from the left menu, then select the **Fleet management** tab. {% endstep %} {% step %} Access the relevant Falcon LogScale Collector instance's config and add the following information under the `sinks` section: * The token value you added in the **Falcon LogScale Collector** Listener setup in Onum. This will go into the `token` field of the configuration. * The Onum URL, with the following format: `https:\\`. * If you are working in a cloud tenant, you will find this URL in the Listener settings under **Address**. Click your Listener in the **Listeners** area to access its details. * Add the port you entered in the Onum configuration and include it in the `url` field of the configuration. If you are working in a cloud tenant, you can also see the **Port** in the Listener settings. {% hint style="warning" %} If you cannot get this information, contact the [Onum team](/support). {% endhint %} Check below a Falcon LogScale Collector sample config file: {% code title="FLC config file" %} ```yaml sinks: flc-to-onum: type: hec token: # Replace with generated token entered in Onum. url: https:// # Replace with Onum distributor URL & port. Must include the "https://" at the beginning. ``` {% endcode %} {% hint style="warning" %} If you manually entered the TLS certificates in the Listener configuration, you must add the following `tls` section at the end of the config file. Enter the path to the CA certificate file you generated before. Add the file in a directory that the Falcon LogScale Collector can read. ```yaml tls: caFile: "" # Replace with full file path to CA certificate. ``` If you're using Windows, you need to escape backslashes (`\`) with an extra backslash in your CA file path. {% endhint %} {% endstep %} {% step %} Click **Publish > Publish draft** to publish your FLC config. {% endstep %} {% step %} Finally, check your the **Fleet Management** page to verify the FLC status shows as **Okay**. You may find the status shows **Error** if, for example, you do not enter the right matching port you chose in Onum. {% endstep %} {% endstepper %} # Collect data from Google Cloud products # Collect data from Google Cloud Storage Most recent version: v1.0.2 {% hint style="info" %} See the changelog of the **Google Cloud Storage** Listener [here](/listeners/google-cloud-storage-listener). {% endhint %} {% hint style="warning" %} * The **Google Cloud Storage** Listener is only available in certain Tenants. [Get in touch with us](/support) if you don't see it and want to access it. * The **Google Cloud Storage** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster. {% endhint %} ## Overview Onum supports integration with [Google Cloud Storage](https://cloud.google.com/storage?hl=en). Google Cloud Storage is an online object storage service that allows users to store and retrieve data. It is a managed service, meaning Google handles the underlying infrastructure, making it scalable and reliable. GCS is designed for a variety of use cases, including storing data for web applications, big data analytics, and backups. Select **Google Cloud Storage** from the list of Listener types and click **Configuration** to start. ## Prerequisites {% hint style="warning" %} In order to use this Listener, you must activate the environment variable in your distributor using docker compose (`GOOGLE_CLOUD_STORAGE_LISTENER_EXECUTION_ENABLED`) {% endhint %} ## Google Cloud Storage Setup To source data from Google Cloud Storage you need to have a GCS bucket with data, appropriate permissions (like `Storage Admin`) to access the bucket and its objects, and the correct resource path (e.g., `gs://bucket-name/object-name`). See [the Google Cloud Storage manual](https://cloud.google.com/appengine/docs/legacy/standard/python/googlecloudstorageclient/setting-up-cloud-storage) for help. ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Google Cloud Storage** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} The Google Cloud connector uses OAuth 2.0 credentials for authentication and authorization. In the **Credentials file****\*** field, create a new [Secret](/administration/global-settings/organization-settings/secrets-management) containing these credentials or select one already created. To get it: 1. To find the **Google Cloud credentials file**, go to **Settings > Interoperability**. 2. Scroll down to the **Service Account** area. 3. You need to generate and download a **service account key** from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here. 4. To see existing Service Accounts, go to the menu in the top left and select **APIs & Services > Credentials**. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} {% endstep %} {% step %} Assign an optional **Event delimiter** to split file content into different events using a delimiter (Examples: `-`, `\n`, `\r\n`, `0x0A`...). {% endstep %} {% step %} Choose the **Compression type****\*** for your files (**None**, **Gzip**, **Bzip2** or **Auto**). {% endstep %} {% step %} If you set the **Read Bucket Once** parameter to **true**, the Listener will read the entire bucket once and stop the execution. You'll be prompted to enter the following: * **Prefix** - The optional string that acts like a folder path or directory **structure** when organizing objects within a bucket. * **Bucket****\*** - Enter the GCP bucket name. * **Start at****\*** - This will block the Listener from starting until this timestamp. The required date format is `DD/MM/YYYY HH:mm`. The specified time must be in the future and conform to the timezone where the operation is being executed. {% endstep %} {% step %} The **Project ID****\*** is a unique string with the following format: `my-project-123456`. To get it: 1. Go to the Google Cloud Console. 2. In the top left corner, click on the **project drop-down** next to the Google Cloud logo (where your current project name is shown). 3. Each project will have a **Project Name** and a **Project ID**. 4. You can also find it in the **Settings** tab on the left-hand side. {% endstep %} {% step %} Enter your **Subscription** (called **Subscription ID** in the Cloud Console). Follow these steps to get it: 1. Go to **Pub/Sub** in the **Google Cloud Console**. 2. In the top left corner, click on the **menu** and select **View all Products**. 3. Then go to **Analytics** and find **Pub/Sub**. Click it to go to Pub/Sub (you can also use the search bar and type `Pub/Sub`). 4. In the **Pub/Sub** dashboard, select the **Subscriptions** tab on the left. 5. The **Subscription ID** will be displayed in this list. {% endstep %} {% step %} In case of a failure to connect, enter the following parameters: * **Number of retries****\*** - Enter the maximum number of retries to perform in case of a failure. The minimum value is `1`, and the maximum value is `5`. The default value is `3`. * **Retry delay****\*** - Enter the number of milliseconds to wait between retries. The minimum and default value is `100`, and the maximum value is `1000`. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Output Ports The **Google Cloud Storage** Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. # Collect data from Google Pub/Sub Most recent version: v1.0.1 {% hint style="info" %} See the changelog of the **Google Pub/Sub** Listener [here](/listeners/google-pub-sub-listener). {% endhint %} {% hint style="warning" %} The **Google Pub/Sub** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster. {% endhint %} ## Overview Onum supports integration with [Google Pub/Sub](https://cloud.google.com/pubsub/docs/overview). Google Pub/Sub is an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages. Pub/Sub allows services to communicate asynchronously. Select **Google Pub/Sub** from the list of Listener types and click **Configuration** to start. ## Google Cloud Storage Setup To source data from Google Cloud Pub/Sub you need to have a Google Cloud project, appropriate roles and permissions to run Pub/Sub, and enable the Pub/Sub API. See [Google Cloud Pub/Sub documentation](https://cloud.google.com/pubsub/docs) for help on how to set these up. ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Google Pub/Sub** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} The **Project ID****\*** is a unique string with the following format: `my-project-123456`. To get it: 1. Go to the Google Cloud Console. 2. In the top left corner, click on the **project drop-down** next to the Google Cloud logo (where your current project name is shown). 3. Each project will have a **Project Name** and a **Project ID**. 4. You can also find it in the **Settings** tab on the left-hand side. {% endstep %} {% step %} Enter your **Subscription Name****\*** 1. Go to **Pub/Sub** in the **Google Cloud Console**. 2. In the top left corner, click on the **menu** and select **View all Products**. 3. Then go to **Analytics** and find **Pub/Sub**. Click it to go to Pub/Sub (you can also use the search bar and type `Pub/Sub`). 4. In the **Pub/Sub** dashboard, select the **Subscriptions** tab on the left. 5. The **Subscription Name** will be displayed in this list. {% endstep %} {% step %} The Google Cloud connector uses OAuth 2.0 credentials for authentication and authorization. Select the **Credentials File****\*** from your [Secrets](https://docs.onum.com/administration/tenant-menu) or click **New secret** to generate a new one. 1. To find the **Google Cloud credentials file**, go to **Settings > Interoperability**. 2. Scroll down to the **Service Account** area. 3. You need to generate and download a **service account key** from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here. 4. To see existing Service Accounts, go to the menu in the top left and select **APIs & Services > Credentials**. {% endstep %} {% step %} Decide whether or not to activate the bulk message option using the **Enabled****\*** field. Then, choose the required **Message Format** and enter the characters you want to use as delimiters in the **Delimiter Char Codes** field, if required. A delimiter character code refers to the numerical representation (usually in ASCII or Unicode) of a delimiter. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Output Ports The **Google Pub/Sub** Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. # Collect data using HTTP {% hint style="info" %} * See the changelog of the **HTTP** Listener type [here](/listeners/http-listener). * See the changelog of the **HTTP Standard Inbound** Listener [here](/listeners/http-standard-inbound-listener). {% endhint %} ## Overview Use the **HTTP** Listener to listen for HTTP requests. If your source does not support **mutual TLS (mTLS)** **encryption**, use the **HTTP Standard Inbound** Listener for TLS encryption without requiring mTLS. We always recommend using mTLS encryption for maximum security. The steps to set up the **HTTP Standard Inbound** Listener are the same as below, so you can follow this article. ## Important Considerations Regarding Cloud Deployments * In cloud-based Onum installations, the **TLS** **configuration** section of the **HTTP** Listener is not visible and you won't need to enter these values. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If your **HTTP** Listener configuration requires you to manually enter these TLS certificates, you can generate them following the instructions [in this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). * If you are defining this Listener in a cloud instance, Onum will automatically provide the **Port** and **TLS configuration** (mTLS for the **HTTP** Listener, TLS for **HTTP Standard Inbound** Listener). * In cloud deployments, these Listeners have an additional step in their creation process: **Network configuration**. Use these details to configure your data source to communicate with Onum. Click **Download certificate** to get the required certificate for the connection. You can also download it from the Listener details once it is created. * When configuring a Listener in a cloud tenant, the port will always be `443`. In on-prem deployments, the selected port must fall within the range of `1024` to `10000`. * In cloud deployments, endpoints are created in Onum's DNS. This process is usually fast, and Listeners are normally available immediately. However, note that this may last up to 24-48 hours, depending on your organization's DNS configuration. * Your data input must use the **Server Name Indication (SNI)** method, which means it must send its hostname in the TLS authentication process. If SNI is not used, the certificate routing will fail, and data will not be received, even if the certificate is valid. ## Onum Setup Here we will detail the steps for the **HTTP** and **HTTP Standard Inbound** Listeners**.** {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **HTTP/HTTP Standard Inbound** Listener. {% endstep %} {% step %} Enter a **Name****\*** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} For most cloud-based Onum installations, the **Socket** section is not visible, and port `443` is used by default. If you see it, enter the required port in the **Port****\*** field. At this time, all TCP ports from `1024` to `10000` are open. {% endstep %} {% step %} In most cloud-based Onum installations, the **TLS** **configuration** section is not visible. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If you see this section, you must enter the required **Certificate**, **Private key** and **CA Chain**. Learn how to generate these self-signed certificates in [this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). Once you have them, click **New secret** in each field and add the corresponding values. {% endstep %} {% step %} If your connection does not require **Authentication**, leave the **Authentication type****\*** field as **None.** Otherwise, choose the authentication type and enter the details. The options provided will vary depending on the type chosen to authenticate your API. This is the type you have selected in the API end, so it can recognize the request.
Basic Enter the following: * **Username****\*** - The user sending the request. * **Password****\*** - Choose the basic auth password from your list of Secrets or [create a new one](/administration/tenant-menu).
Bearer Enter your **Token Secret** for the API request using an existing Secret or [creating a new one](/administration/tenant-menu) if you haven't stored it in Onum yet. This grants access without needing to send credentials (like username and password) in every request. ***Example*** Let's say you have the following configuration: * **Port** - `8080` * **Authentication Type** - `Bearer` * **Bearer Token Secret** - `a-string-secret-at-least-256-bits-long` (This is the value you enter into Onum as the secret) * **Request path** - `localhost` When you listen for the HTTP request, the token will be encoded: `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWUsImlhdCI6MTUxNjIzOTAyMn0.KMUFsIDTnFmyG3nMiGM6H9FNFUROf3wh7SmqJp-QV30` This entire request will show as follows:\ `"http://localhost:8080/bearer" 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWUsImlhdCI6MTUxNjIzOTAyMn0.KMUFsIDTnFmyG3nMiGM6H9FNFUROf3wh7SmqJp-QV30'`
API Key in URL Params Enter the following: * **API Key Name****\*** - A label assigned to the API key for identification. You can find it depending on where the API key was created. * **API Key Value****\*** - API keys are usually stored in developer portals, cloud dashboards, or authentication settings. Choose the existing Secret or [create a new one](/administration/tenant-menu) if you haven't stored this key within Onum. Note that if you select this option, the HTTP Listener expects the API Key to be included in the URL, as a query parameter. For example: ``` curl --location 'http://customer.in.prod.onum.com:2250/test?My-Token=1234567890qwerty' \ --header 'Content-Type: application/json' \ --data '{"message": "hello, how are you doing? :)"}' ```
API Key in Header Enter the following: * **API Key in Header Name****\*** - A label assigned to the API key for identification. You can find it depending on where the API key was created. * **API Key in Header Value****\*** - API keys are usually stored in developer portals, cloud dashboards, or authentication settings. Choose the existing Secret or [create a new one](/administration/tenant-menu) if you haven't stored this key within Onum.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} {% endstep %} {% step %} In the **HTTP method****\*** field, choose **GET**, **POST**, or **PUT** method and the **Request** **path****\*** to the resource being requested from the server. The path starts with `/`, for example if the URL is `https://localhost/onum`, `/onum` is the path. {% endstep %} {% step %} In the **Message extraction** section, the **Strategy****\*** defines how data extraction should be performed. It is the overall methodology or approach used to extract relevant information from HTTP messages. Choose between: * **Single event with the whole request** - Choose this option if you want to include the whole request in each event. * **Single event from request path** - Choose this option if you want to include the request paths in each event. * **Single event as query string** - Choose this option if you want to include the requests with their whole query strings. * **Single event as query parameter** - Choose this option if you want to include a specific request parameter in your events. Specify the required parameter name in the **Extraction info** option (for example: `msg`) * **Single event as header** - Choose this option if you want to include a specific header in your events. Specify the required header in the **Extraction info** option (for example: `Message`) * **Single event as body (partially)** - Choose this option if you want to include a part of the request body in your events. Specify the required RegEx rule to match the required part in the **Extraction info** option (for example: `\\[BODY: (.+)\\]`) * **Single event as body (full)** - Choose this option if you want to include the whole request body in your events. Specify the required RegEx rule to match the required part in the **Extraction info** option (for example: `\\[BODY: (.+)\\]`) * **Multiple events at body with delimiter** - Choose this option if you want to include several messages in the same event separated by a delimiter. You must specify the delimiter in the **Extraction info** option. * **Multiple events at body as JSON array** - Choose this option if you want to include several messages formatted as a JSON array in your events. * **Multiple events at body as stacked JSON** - Choose this option if you want to include several messages formatted as a stacked JSON in your events. {% endstep %} {% step %} In the **General behavior** section, choose between **None** (default option), **Allow** (enter the required header keys below), or **All** (all headers will be retrieved in the `headers` field). {% endstep %} {% step %} Then, configure the following settings: * **Header keys** - Enter the required header keys in this field. Click **Add element** for each one. * **Exported headers format** - Choose the required format for your headers. The default value is **JSON**. * **Maximum message length** - Maximum characters of the message. The default value is `4096`. * **Response code** - Specify the response code to show when successful. The default value is **202 Accepted**. * **Response Content-Type** - The `Content-Type: xxx/xxx` lets the server know the expected format of the incoming message or request (**application/json** by default): * **text/plain** - The message body contains plain text. * **application/json** - The message body is formatted as JSON. * **application/xml** - The message body is formatted as XML. * **text/html** - The message body contains HTML. * **Response text** - The text that will show in case of success. {% endstep %} {% step %} **Now there are two possible scenarios:** * If you didn't enter your TLS certificates, when you click **Create listener** you'll see the **Network configuration** screen, which shows the **Address** and **Port** needed to communicate with Onum. Here you will download the certificate (see the[ steps after creation to do this](#download-certificate)). {% hint style="info" %} You can access all this information in the Listener details after creation, so don't worry. {% endhint %} * If you entered the TLS certificates, you'll go directly to the **Labels**. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% endstepper %} ## Download Certificate if you have created an **HTTP** Listener in an Onum cloud deployment, you can download the certificate from the **Listeners** view by clicking the created Listener. In the details view, select the three dots in the top right-hand corner and click **Download Certificate**. {% hint style="info" %} This .p12 does not require password to access. {% endhint %} To extract the certificates from the download, use the following command: ``` #!/bin/bash # Extract certs from certificate.p12 # Client certificate (PEM) openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out client.crt -password pass: # Client private key (PEM) openssl pkcs12 -in certificate.p12 -nocerts -nodes -out client.key -password pass: # CA chain (PEM) openssl pkcs12 -in certificate.p12 -cacerts -nokeys -out ca-chain.crt -password pass: ``` ## Output Ports The **HTTP** and **HTTP Standard Inbound** Listeners have two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. {% hint style="warning" %} The error message is provided in a free-text format and may change over time. Please consider this if performing any post-processing based on the message content. {% endhint %} # Collect data from Cloudflare {% hint style="info" %} See the changelog of the **HTTP** Listener [here](/listeners/http-listener). {% endhint %} ## Overview The following article outlines a basic data flow from **Cloudflare** to the Onum **HTTP** Listener. ## Prerequisites [Contact Onum](/support) to get the required JWT token, which will be needed on the Listener setup. You can also contact us if you cannot generate the required TLS certificates. Note that these certificates must be signed by a recognized Certificate Authority (CA). Self-signed certificates are not accepted. ## Cloudflare Setup Cloudflare Logpush supports the ability to send logs to configurable HTTP endpoints. Follow [these instructions](https://developers.cloudflare.com/logs/logpush/logpush-job/enable-destinations/http/) to enable log sending before starting the configuration in Onum. * In cloud-based Onum installations, the **TLS** configuration section of the HTTP Listener is not visible and you won't need to enter these values. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If your HTTP Listener configuration requires you to manually enter these TLS certificates, you can generate them following the instructions [in this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). * If you are defining this Listener in a cloud instance, Onum will automatically provide the **Port** and **TLS** configuration. * Cloud Listeners have an additional step in their creation process: **Network configuration**. Use these details to configure your data source to communicate with Onum. Click **Download certificate** to get the required certificate for the connection. You can also download it from the Listener details once it is created. * When configuring a Listener in a Cloud tenant, the **port** will be `443`. In on-prem, the selected port must fall within the range of `1024` to `10000`. * Cloud Listener endpoints are created in Onum's DNS. This process is usually fast, and Listeners are normally available immediately. However, note that this may last up to 24-48 hours, depending on your organization's DNS configuration. * Your data input must use the **Server Name Indication (SNI)** method, which means it must send its hostname in the TLS authentication process. If SNI is not used, the certificate routing will fail, and data will not be received, even if the certificate is valid. If your organization's software cannot meet points 2 and 3, you can use an intermediate piece of software to ensure the client-Onum connection, such as Stunnel. ## Onum Setup Here we will detail the steps for the **HTTP** Listener**.** {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **HTTP** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} For most cloud-based Onum installations, the **Socket** section is not visible, and **port** `443` is used by default. If you see it, enter the required port in the **Port** field. At this time, all TCP ports from `1024` to `10000` are open. {% endstep %} {% step %} In most cloud-based Onum installations, the **TLS** configuration section is not visible. In these setups, Onum automatically manages **TLS** certificates, eliminating the need for manual configuration. If you see this section, you must enter the required **Certificate**, **Private key** and **CA Chain.** Learn how to generate these self-signed certificates in [this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). Once you have them, click **New secret** in each field and add the corresponding values. {% endstep %} {% step %} **Now there are two possible scenarios:** If you didn't enter your **TLS** certificates, when you click **Create listener** you'll see the **Network configuration** screen, which shows the **Address** and **Port** needed to communicate with Onum. Here you will download the certificate (see the[ steps after creation to do this](#download-certificate)). {% hint style="info" %} You can access all this information in the Listener details after creation, so don't worry. {% endhint %} If you entered the TLS certificates, you'll go directly to the Labels when you eventually click **create Listener**. {% endstep %} {% step %} In the **Authentication** section, choose **Bearer** as the Authentication Type. Open the Token Secret field and click New secret to create a new one: * Give the token a **Name**. * Turn off the **Expiration** **date** option. * Click **Add new value** and paste the secret corresponding to the JWT token you received. Remember that the token will be added in the Cloudflare configuration. * Click **Save**. {% hint style="info" %} Learn more about secrets in Onum [in this article.](/administration/global-settings/organization-settings/secrets-management) {% endhint %} You can now select the secret you just created in the Token Secret field. {% endstep %} {% step %} In the **Endpoint** section, choose `POST` as the method. In the **Request path** field, enter `/` {% endstep %} {% step %} In the **Message extraction** section, choose **Single event as body (full)** in the Strategy field. {% endstep %} {% step %} In the **General behavior** section, set **Propagate headers strategy** to **Allow**. {% endstep %} {% step %} Then, configure the following settings: * `Content-Enconding` * `Content-Type` {% endstep %} {% step %} For cloud installments, copy the **DNS Address** details to configure your data source in order to communicate with Onum. This contains the IP address of the DNS (Domain Name System) server to connect to. {% hint style="warning" %} Note that you will only see this section if you're defining this Listener in a Cloud instance. {% endhint %} {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% endstepper %} Click **Create listener** when you're done. ### Download certificate Now, download the certificate from the **Listeners** view by clicking the created listener and selecting the three dots in the top right-hand corner of the menu > **Download Certificate**. {% hint style="info" %} This .p12 does not require password to access. {% endhint %} To extract the certificates from the download: ``` #!/bin/bash # Extract certs from certificate.p12 # Client certificate (PEM) openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out client.crt -password pass: # Client private key (PEM) openssl pkcs12 -in certificate.p12 -nocerts -nodes -out client.key -password pass: # CA chain (PEM) openssl pkcs12 -in certificate.p12 -cacerts -nokeys -out ca-chain.crt -password pass: ``` ### Ports The HTTP Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. {% hint style="warning" %} The error message is provided in a free-text format and may change over time. Please consider this if performing any post-processing based on the message content. {% endhint %} Click **Create listener** when you're done. # Collect data from Microsoft Defender for Cloud Apps {% hint style="info" %} See the changelog of the **HTTP** Listener [here](/listeners/http-listener). {% endhint %} ## Overview The following article outlines a basic data flow from [Microsoft Defender for Cloud Apps](https://learn.microsoft.com/en-us/defender-cloud-apps/what-is-defender-for-cloud-apps) to the Onum **HTTP** Listener. ## Prerequisites * Administrative access to the Microsoft Defender for Cloud Apps portal [Contact Onum](/support) to get the required JWT token, which will be needed on the Listener setup. You can also contact us if you cannot generate the required TLS certificates. Note that these certificates must be signed by a recognized Certificate Authority (CA). Self-signed certificates are not accepted. ## Defender for Cloud Apps Setup Microsoft Defender for Cloud Apps (MDCA) can be configured to send logs to Onum. Here's how to set it up: 1. Access [Microsoft Defender for Cloud Apps](https://portal.cloudappsecurity.com) 2. Go to **Settings** > **Security extensions** * Select **SIEM agents** tab * Click **Add SIEM agent** 3\. Set Up the SIEM Agent * Choose **Generic SIEM** as the SIEM type * Enter a name for the connection (e.g., Onum Integration) * Select the data types you want to send: * Alerts * Activities * Discovery data (if applicable) * Configure the remote SIEM server: * Protocol: HTTPS * Host: Your Onum domain (e.g., `https://[your-onum-tenant].onum.ai/api/ingest`) * Port: 443 (standard HTTPS) * URL path: Your configured path (e.g., `/ingest/mdca`) 4. Configure Authentication * Select the appropriate authentication method. In this case, we will exemplify the **bearer** method. * Specify the log format (JSON is recommended) ## Important Considerations Regarding Cloud Listeners * In cloud-based Onum installations, the **TLS** configuration section of the HTTP Listener is not visible and you won't need to enter these values. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If your HTTP Listener configuration requires you to manually enter these TLS certificates, you can generate them following the instructions [in this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). * If you are defining this Listener in a cloud instance, Onum will automatically provide the **Port** and **TLS** configuration. * Cloud Listeners have an additional step in their creation process: **Network configuration**. Use these details to configure your data source to communicate with Onum. Click **Download certificate** to get the required certificate for the connection. You can also download it from the Listener details once it is created. * When configuring a Listener in a Cloud tenant, the **port** will be `443`. In on-prem, the selected port must fall within the range of `1024` to `10000`. * Cloud Listener endpoints are created in Onum's DNS. This process is usually fast, and Listeners are normally available immediately. However, note that this may last up to 24-48 hours, depending on your organization's DNS configuration. * Your data input must use the **Server Name Indication (SNI)** method, which means it must send its hostname in the TLS authentication process. If SNI is not used, the certificate routing will fail, and data will not be received, even if the certificate is valid. If your organization's software cannot meet points 2 and 3, you can use an intermediate piece of software to ensure the client-Onum connection, such as Stunnel. ## Onum Setup Here we will detail the steps for the **HTTP** Listener**.** {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **HTTP** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} For most cloud-based Onum installations, the **Socket** section is not visible, and **port** `443` is used by default. If you see it, enter the required port in the **Port** field. At this time, all TCP ports from `1024` to `10000` are open. {% endstep %} {% step %} In most cloud-based Onum installations, the **TLS** configuration section is not visible. In these setups, Onum automatically manages **TLS** certificates, eliminating the need for manual configuration. If you see this section, you must enter the required **Certificate**, **Private key** and **CA Chain.** Learn how to generate these self-signed certificates in [this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). Once you have them, click **New secret** in each field and add the corresponding values. {% endstep %} {% step %} **Now there are two possible scenarios:** If you didn't enter your **TLS** certificates, when you click **Create listener** you'll see the **Network configuration** screen, which shows the **Address** and **Port** needed to communicate with Onum. Here you will download the certificate (see the[ steps after creation to do this](#download-certificate)). {% hint style="info" %} You can access all this information in the Listener details after creation, so don't worry. {% endhint %} If you entered the TLS certificates, you'll go directly to the Labels when you eventually click **create Listener**. {% endstep %} {% step %} In the **Authentication** section, choose **Bearer** as the Authentication Type. Open the Token Secret field and click New secret to create a new one: * Give the token a **Name**. * Turn off the **Expiration** **date** option. * Click **Add new value** and paste the secret corresponding to the JWT token you received. Remember that the token will be added in the Cloudflare configuration. * Click **Save**. {% hint style="info" %} Learn more about secrets in Onum [in this article.](/administration/global-settings/organization-settings/secrets-management) {% endhint %} You can now select the secret you just created in the Token Secret field. {% endstep %} {% step %} In the **Endpoint** section, choose `POST` as the **HTTP Method**. In the **Request path** field, you're creating an endpoint where MDCA will send data. #### Standard Format ``` /ingest/mdca

Standard Format

​
/ingest/mdca
``` {% endstep %} {% step %} In the **Message extraction** section, choose **Single event as body (full)** in the Strategy field. {% endstep %} {% step %} In the **General behavior** section, set **Propagate headers strategy** to **Allow**. {% endstep %} {% step %} Then, configure the following settings: * `Content-Enconding` * `Content-Type` {% endstep %} {% step %} For cloud installments, copy the **DNS Address** details to configure your data source in order to communicate with Onum. This contains the IP address of the DNS (Domain Name System) server to connect to. {% hint style="warning" %} Note that you will only see this section if you're defining this Listener in a Cloud instance. {% endhint %} {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% endstepper %} Click **Create listener** when you're done. ### Download certificate For cloud environments, download the certificate from the **Listeners** view by clicking the created listener and selecting the three dots in the top right-hand corner of the menu > **Download Certificate**. {% hint style="info" %} This .p12 does not require password to access. {% endhint %} To extract the certificates from the download: ``` #!/bin/bash # Extract certs from certificate.p12 # Client certificate (PEM) openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out client.crt -password pass: # Client private key (PEM) openssl pkcs12 -in certificate.p12 -nocerts -nodes -out client.key -password pass: # CA chain (PEM) openssl pkcs12 -in certificate.p12 -cacerts -nokeys -out ca-chain.crt -password pass: ``` ### Ports The HTTP Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. {% hint style="warning" %} The error message is provided in a free-text format and may change over time. Please consider this if performing any post-processing based on the message content. {% endhint %} # Collect data from Microsoft Defender for Identity {% hint style="info" %} See the changelog of the **HTTP** Listener [here](/listeners/http-listener). {% endhint %} ## Overview The following article outlines a basic data flow from [**Microsoft Defender for Identity**](https://learn.microsoft.com/en-us/defender-for-identity/what-is) to the Onum **HTTP** Listener. ## Prerequisites * Administrative access to the Microsoft 365 Defender portal [Contact Onum](/support) to get the required JWT token, which will be needed on the Listener setup. You can also contact us if you cannot generate the required TLS certificates. Note that these certificates must be signed by a recognized Certificate Authority (CA). Self-signed certificates are not accepted. ## Defender for Identity Setup Microsoft Defender for Identity (MDI) can be configured to send logs to HTTP endpoints through SIEM integration. Here's how to set it up: 1. **Access** [**the Microsoft 365 Defender portal**](https://security.microsoft.com) 2. Go to **Settings** > **Endpoints** > **Advanced features** * In the **SIEM Integration** section, enable SIEM integration. 3. **Set up the HTTP endpoint** * Select **Add SIEM connector** * Choose **Generic HTTP endpoint** as the connector type * Enter your Onum ingestion URL (typically in format: `https://[your-onum-tenant].onum.ai/api/ingest` ) * Configure the authentication method you will use to authenticate the connection in the Listener. In this case, we will exemplify the **bearer** method. * Specify the log format (JSON is recommended) 4. **Configure log types** * Select which MDI log types to forward (alerts, security events, etc.) * Set filtering options if needed ## Important Considerations Regarding Cloud Listeners * In cloud-based Onum installations, the **TLS** configuration section of the HTTP Listener is not visible and you won't need to enter these values. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If your HTTP Listener configuration requires you to manually enter these TLS certificates, you can generate them following the instructions [in this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). * If you are defining this Listener in a cloud instance, Onum will automatically provide the **Port** and **TLS** configuration. * Cloud Listeners have an additional step in their creation process: **Network configuration**. Use these details to configure your data source to communicate with Onum. Click **Download certificate** to get the required certificate for the connection. You can also download it from the Listener details once it is created. * When configuring a Listener in a Cloud tenant, the **port** will be `443`. In on-prem, the selected port must fall within the range of `1024` to `10000`. * Cloud Listener endpoints are created in Onum's DNS. This process is usually fast, and Listeners are normally available immediately. However, note that this may last up to 24-48 hours, depending on your organization's DNS configuration. * Your data input must use the **Server Name Indication (SNI)** method, which means it must send its hostname in the TLS authentication process. If SNI is not used, the certificate routing will fail, and data will not be received, even if the certificate is valid. If your organization's software cannot meet points 2 and 3, you can use an intermediate piece of software to ensure the client-Onum connection, such as Stunnel. ## Onum Setup Here we will detail the steps for the **HTTP** Listener**.** {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **HTTP** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} For most cloud-based Onum installations, the **Socket** section is not visible, and **port** `443` is used by default. If you see it, enter the required port in the **Port** field. At this time, all TCP ports from `1024` to `10000` are open. {% endstep %} {% step %} In most cloud-based Onum installations, the **TLS** configuration section is not visible. In these setups, Onum automatically manages **TLS** certificates, eliminating the need for manual configuration. If you see this section, you must enter the required **Certificate**, **Private key** and **CA Chain.** Learn how to generate these self-signed certificates in [this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). Once you have them, click **New secret** in each field and add the corresponding values. {% endstep %} {% step %} **Now there are two possible scenarios:** If you didn't enter your **TLS** certificates, when you click **Create listener** you'll see the **Network configuration** screen, which shows the **Address** and **Port** needed to communicate with Onum. Here you will download the certificate (see the[ steps after creation to do this](#download-certificate)). {% hint style="info" %} You can access all this information in the Listener details after creation, so don't worry. {% endhint %} If you entered the TLS certificates, you'll go directly to the Labels when you eventually click **create Listener**. {% endstep %} {% step %} In the **Authentication** section, choose **Bearer** as the Authentication Type. Open the Token Secret field and click New secret to create a new one: * Give the token a **Name**. * Turn off the **Expiration** **date** option. * Click **Add new value** and paste the secret corresponding to the JWT token you received. Remember that the token will be added in the Cloudflare configuration. * Click **Save**. {% hint style="info" %} Learn more about secrets in Onum [in this article.](/administration/global-settings/organization-settings/secrets-management) {% endhint %} You can now select the secret you just created in the Token Secret field. {% endstep %} {% step %} In the **Endpoint** section, choose `POST` as the **HTTP Method**. In the **Request path** field, you're creating an endpoint where MDI will send data. #### Standard Formats ``` /ingest/microsoft/defender-identity ``` ``` /api/collect/microsoft/defender-identity ``` ``` /ingest/mdi ``` ``` /webhook/security/mdi ``` {% endstep %} {% step %} In the **Message extraction** section, choose **Single event as body (full)** in the Strategy field. {% endstep %} {% step %} In the **General behavior** section, set **Propagate headers strategy** to **Allow**. {% endstep %} {% step %} Then, configure the following settings: * `Content-Enconding` * `Content-Type` {% endstep %} {% step %} For cloud installments, copy the **DNS Address** details to configure your data source in order to communicate with Onum. This contains the IP address of the DNS (Domain Name System) server to connect to. {% hint style="warning" %} Note that you will only see this section if you're defining this Listener in a Cloud instance. {% endhint %} {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% endstepper %} Click **Create listener** when you're done. ### Download certificate For cloud environments, download the certificate from the **Listeners** view by clicking the created listener and selecting the three dots in the top right-hand corner of the menu > **Download Certificate**. {% hint style="info" %} This .p12 does not require password to access. {% endhint %} To extract the certificates from the download: ``` #!/bin/bash # Extract certs from certificate.p12 # Client certificate (PEM) openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out client.crt -password pass: # Client private key (PEM) openssl pkcs12 -in certificate.p12 -nocerts -nodes -out client.key -password pass: # CA chain (PEM) openssl pkcs12 -in certificate.p12 -cacerts -nokeys -out ca-chain.crt -password pass: ``` ### Ports The HTTP Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. {% hint style="warning" %} The error message is provided in a free-text format and may change over time. Please consider this if performing any post-processing based on the message content. {% endhint %} # Collect data from Zscaler Zscaler (Nanolog Streaming Service) to Onum HTTP Listener (with TLS) {% hint style="info" %} See the changelog of the **HTTP** Listener [here](/listeners/http-listener). {% endhint %} ## Overview The following article outlines a basic data flow from **Zscaler's Nanolog Streaming Service (NSS)** to the Onum **HTTP** Listener. ## Prerequisites [Contact Onum](/support) to get the cert information needed for TLS communication, which will be needed on the Listener setup. ## Zscaler NSS Setup Identify the NSS Feeds you want to send in the [Zscaler documentation](https://help.zscaler.com/zia/documentation-knowledgebase/analytics/nss/nss-feeds/adding-nss-feeds). Configure the required ingestion setup following the steps in the documentation. #### :triangle-exclamation: **Important notes** * The **SIEM type** will be `Other`. * You must generate a **JWT token** and add it as an HTTP header. Add the word `Bearer` before the token value (`Bearer `). The corresponding secret value will be added in the Onum configuration later. {% hint style="warning" %} [Contact us](/support) if you cannot generate a JWT token. {% endhint %}
## Important Considerations Regarding Cloud Listeners * In cloud-based Onum installations, the **TLS** configuration section of the HTTP Listener is not visible and you won't need to enter these values. In these setups, Onum automatically manages TLS certificates, eliminating the need for manual configuration. If your HTTP Listener configuration requires you to manually enter these TLS certificates, you can generate them following the instructions [in this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). * If you are defining this Listener in a cloud instance, Onum will automatically provide the **Port** and **TLS** configuration. * Cloud Listeners have an additional step in their creation process: **Network configuration**. Use these details to configure your data source to communicate with Onum. Click **Download certificate** to get the required certificate for the connection. You can also download it from the Listener details once it is created. * When configuring a Listener in a Cloud tenant, the **port** will be `443`. In on-prem, the selected port must fall within the range of `1024` to `10000`. * Cloud Listener endpoints are created in Onum's DNS. This process is usually fast, and Listeners are normally available immediately. However, note that this may last up to 24-48 hours, depending on your organization's DNS configuration. * Your data input must use the **Server Name Indication (SNI)** method, which means it must send its hostname in the TLS authentication process. If SNI is not used, the certificate routing will fail, and data will not be received, even if the certificate is valid. If your organization's software cannot meet points 2 and 3, you can use an intermediate piece of software to ensure the client-Onum connection, such as Stunnel. ## Onum Setup Here we will detail the steps for the **HTTP** Listener**.** {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **HTTP** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} For most cloud-based Onum installations, the **Socket** section is not visible, and **port** `443` is used by default. If you see it, enter the required port in the **Port** field. At this time, all TCP ports from `1024` to `10000` are open. {% endstep %} {% step %} In most cloud-based Onum installations, the **TLS** configuration section is not visible. In these setups, Onum automatically manages **TLS** certificates, eliminating the need for manual configuration. If you see this section, you must enter the required **Certificate**, **Private key** and **CA Chain.** Learn how to generate these self-signed certificates in [this article](https://docs.onum.com/usecases/routing/crowdstrike-integration/self-signed-ssl-tls-certificates-creation). Once you have them, click **New secret** in each field and add the corresponding values. {% endstep %} {% step %} **Now there are two possible scenarios:** If you didn't enter your **TLS** certificates, when you click **Create listener** you'll see the **Network configuration** screen, which shows the **Address** and **Port** needed to communicate with Onum. Here you will download the certificate (see the[ steps after creation to do this](#download-certificate)). {% hint style="info" %} You can access all this information in the Listener details after creation, so don't worry. {% endhint %} If you entered the TLS certificates, you'll go directly to the Labels when you eventually click **create Listener**. {% endstep %} {% step %} In the **Authentication** section, choose **Bearer** as the Authentication Type. Open the Token Secret field and click New secret to create a new one: * Give the token a **Name**. * Turn off the **Expiration** **date** option. * Click **Add new value** and paste the secret corresponding to the JWT token you received. Remember that the token will be added in the Zscaler configuration. * Click **Save**. {% hint style="info" %} Learn more about secrets in Onum [in this article.](/administration/global-settings/organization-settings/secrets-management) {% endhint %} You can now select the secret you just created in the Token Secret field. {% endstep %} {% step %} In the **Endpoint** section, choose `POST` as the method. In the **Request path** field, enter `/` {% endstep %} {% step %} In the **Message extraction** section, choose **Multiple events at body as stacked JSON** in the **Strategy** field. You can leave the **Extraction info** field empty. {% endstep %} {% step %} In the **General behavior** section, set **Propagate headers strategy** to **None** (default option). {% endstep %} {% step %} Then, configure the following settings: * **Exported headers format** - Choose the required format for your headers. Choose **JSON** (default value). * **Maximum message length** - Maximum characters of the message. The default value is `4096`. * **Response code** - Specify the response code to show when successful. You must choose **200 OK**. {% hint style="warning" %} **Important** Note that Zscaler doesn't accept any other response than 200 OK. {% endhint %} * **Response Content-Type** - Lets the server know the expected format of the incoming message or request. In this case, choose **application/json**. * **Response text** - The text that will show in case of success. {% endstep %} {% step %} For cloud instalments, copy the **DNS Address** details to configure your data source in order to communicate with Onum. This contains the IP address of the DNS (Domain Name System) server to connect to. {% hint style="warning" %} Note that you will only see this section if you're defining this Listener in a Cloud instance. {% endhint %} {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="info" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% endstepper %} Click **Create listener** when you're done. ### Download certificate For cloud environments, download the certificate from the **Listeners** view by clicking the created listener and selecting the three dots in the top right-hand corner of the menu > **Download Certificate**. {% hint style="info" %} This .p12 does not require password to access. {% endhint %} To extract the certificates from the download: ``` #!/bin/bash # Extract certs from certificate.p12 # Client certificate (PEM) openssl pkcs12 -in certificate.p12 -clcerts -nokeys -out client.crt -password pass: # Client private key (PEM) openssl pkcs12 -in certificate.p12 -nocerts -nodes -out client.key -password pass: # CA chain (PEM) openssl pkcs12 -in certificate.p12 -cacerts -nokeys -out ca-chain.crt -password pass: ``` ### Ports The HTTP Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. {% hint style="warning" %} The error message is provided in a free-text format and may change over time. Please consider this if performing any post-processing based on the message content. {% endhint %} # Collect data from PingID PingID to Onum HTTP Listener (with TLS) {% hint style="info" %} See the changelog of the **HTTP** Listener [here](/listeners/http-listener). {% endhint %} ## Overview The following article outlines a basic data flow from **PingID** to the Onum **HTTP** Listener. ## Prerequisites [Contact Onum](/support) to get the cert information needed for TLS communication, which will be needed on the Listener setup. ## PingID Setup {% hint style="warning" %} [Contact us](/support) if you cannot generate a JWT token. {% endhint %} ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **HTTP** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} In the **Socket** section, enter the required **Port**. By default, all ports from `1024` to `10000` are open. {% endstep %} {% step %} In the **TLS configuration** section, enter the data you received from the Onum team (**Certificate**, **Private key** and **CA chain**). Choose **No client certificate** as **Client authentication method** and **TLS v.1.0** as the **Minimum TLS version**. {% endstep %} {% step %} In the **Authentication** section, choose **Bearer** as the **Authentication Type**. Open the T**oken Secret** field and click **New secret** to create a new one: * Give the token a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the JWT token you generated before. Remember that the token will be added in the PingID configuration. * Click **Save**.
{% hint style="success" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} {% endstep %} {% step %} You can now select the secret you just created in the **Token Secret** field. {% endstep %} {% step %} In the **Endpoint** section, choose **POST** as the **HTTP Method**. In the **Request path** field, enter `/` {% endstep %} {% step %} In the **Message extraction** section, choose **Multiple events at body as stacked JSON** in the **Strategy** field. You can leave the **Extraction info** field empty. {% endstep %} {% step %} In the **General behavior** section, set **Propagate headers strategy** to **None** (default option). {% endstep %} {% step %} Then, configure the following settings: * **Exported headers format** - Choose the required format for your headers. Choose **JSON** (default value). * **Maximum message length** - Maximum characters of the message. The default value is `4096`. * **Response code** - Specify the response code to show when successful. You must choose **200 OK**. * **Response Content-Type** - Lets the server know the expected format of the incoming message or request. In this case, choose **application/json**. * **Response text** - The text that will show in case of success. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**. Click **Create listener** when you're done. {% hint style="success" %} Learn more about labels in [this article](/the-workspace/listeners/labels). {% endhint %} {% endstep %} {% endstepper %} # Pull data from HTTP endpoints Most recent version: v0.0.6 {% hint style="info" %} See the changelog of this Listener type [here](/listeners/http-pull-listener). {% endhint %} {% hint style="warning" %} Note that this Listener is only available in certain Tenants. [Get in touch with us](/support) if you don't see it and want to access it. {% endhint %} ## Overview Onum supports integration with HTTP Pull. Select **HTTP Pull** from the list of Listener types and click **Configuration** to start. ## Prerequisites {% hint style="warning" %} In order to use this Listener, you must activate the following environment variable in your distributor using docker compose:`HTTP_PULL_LISTENER_ENABLED` {% endhint %} ## HTTP Pull configuration {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **HTTP Pull** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} ### Stopped {% hint style="info" %} If you do not see this feature, it has not been made available in your Tenant yet. {% endhint %} The **Stopped** toggle allows you to set the Listener as inactive in order to stop ingesting data from endpoints whilst you do not need it. {% endstep %} {% step %} Now you need to specify the **Parameters.** * Enter the **name** of the parameter to search for in the YAML below, used later as `${parameters.name}` e.g. `${parameters.domain}` * Enter the value or variable to fill in when the given parameter name has been found, e.g. `domain.com`. With the name set as `domain` and the value set as `mydomain` , the expression to execute on the YAML would be: `${parameters.domain}`, which will be automatically replaced by the variable. Add as many name/value pairs as required. **YAML Sample:** 
  url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/
  headers:
    — name: Accept
      value: application/json
    — name: Netskope—Api—Token
      value: "${secrets.netskopeApiToken}"
nextRequest:
  method: GET
  url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/
  headers:
    — name: Accept
      value: application/json
    — name: Netskope—Api—Token
      value: "${secrets.netskopeApiToken}"
{% endstep %} {% step %} Next, configure your **Secrets** * Enter the **name** of the parameter to search for in the YAML below, used later as `${secrets.name}` * Select the [Secret](#secrets) containing the connection credentials if you have added them previously, or select **New Secret** to add it. This will add this value as a variable when the field name is found in the YAML. Add as many as required. **YAML Sample:** ```yaml method: GET url: "https://${parameters.domain}/api/v2/events/data headers: — name: Accept value: application/json — name: Netskope—Api—Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/data headers: — name: Accept value: application/json — name: Netskope—Api—Token value: "${secrets.netskopeApiToken}" ``` {% endstep %} {% step %} Toggle on to configure the HTTP as a **YAML** and paste it here.
The system supports interpolated variables throughout the HTTP request building process using the syntax: `${prefix.name}` Each building block may: * Use variables depending on their role (e.g., parameters, secrets, pagination state). * Expose variables for later phases (e.g., pagination counters, temporal window bounds). {% hint style="warning" %} Not all variable types are available in every phase. Each block has access to a specific subset of variables. {% endhint %} Variables can be defined in the configuration or generated dynamically during execution. Each variable has a prefix that determines its source and scope. These are the supported prefixes: * **Parameters** - User-defined values configured manually. Available in all phases. * **Secrets** - Sensitive values such as credentials or tokens. Available in all phases. * **temporalWindow** - Automatically generated from the Temporal Window block. Available in the Enumeration and Collection phases. * **Pagination** - Values produced by the pagination mechanism (e.g., offset, cursor). Available in the Enumeration and Collection phases. * **Inputs** - Values derived from the output of the Enumeration phase. Available only in the Collection phase. If you do not have a YAML to paste, see how to manually configure the various components of a YAML in the following sections. {% endstep %} {% endstepper %} ## Desconstructing a YAML Here we will learn what each parameter of the YAML means, and how they correspond to the settings in the HTTP Pull Listener. The YAML is used for pulling alerts via an API and typically uses * A **Temporal Window** to enable the use of a time-based query window for filtering results. * **Authentication** using a token to authenticate the connection. * The first phase **(Enumeration)** enables an initial listing phase to get identifiers (e.g., alert IDs), paginating through the results. * The second phase **(Collection)** then fetches full alert details using the alert IDs from the enumeration phase. * Standard JSON response mapping is used to output the results. {% hint style="info" %} Only the Collection phase is mandatory, the rest of the fields are optional. {% endhint %} Let´s take a closer look at each phase below. *** ### Temporal window A temporal window is a defined time range used to filter or limit data retrieval in queries or API requests. It specifies the start and end time for the data you want to collect or analyze. This YAML uses a temporal window of 5 minutes, in RFC3339 format, with an offset of 10, in UTC timezone.
ParameterDescription
Duration*Add the duration that the window will remain open for.
Offset*How far back from the current time the window starts.
Time Zone*This value is usually automatically set to your current time zone. If not, select it here.
Format*Choose between Epoch or RCF3339 for the timestamp format.
Temporal Window example ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 10 tz: UTC format: RFC3339 ``` In Onum, toggle **ON** the Temporal Window selector and enter the information in the corresponding fields * **Duration****\***** -** 5m * **Offset****\***** -** 10 * **TZ****\***** -** this will set automatically according to your current timezone. * **Format****\***** -** RFC3339 So if the current UTC is 12:00, the range would be 11:50 - 11:55.
*** ### Authentication phase If your connection requires authentication, enter the credentials here.
ParameterDescription
Authentication Type*Choose the authentication type and enter the details.
#### Authentication credentials The options provided will vary depending on the type chosen to authenticate your API. This is the type you have selected in the API end, so it can recognize the request. Choose between the options below.
Basic * **Username****\***** -** the user sending the request. * **Password****\***** -** the password eg: `${secrets.password}` ```yaml withAuthentication: true authentication: type: basic basic: username: testuser password: testpass ```
API Key Enter the following: * **API Key -** API keys are usually stored in developer portals, cloud dashboards, or authentication settings. Set the a secret, eg: `${secrets.api_key}` * **Auth injection:** * **In****\***** -** Enter the incoming format of the API: Header or Query. * **Name****\***** -** The header name or parameter name where the api key will be sent. * **Prefix -** Enter a prefix if required. * **Suffix -** Enter a suffix if required. ```yaml withAuthentication: true authentication: type: apiKey apiKey: apiKey: test-api-key authInjection: name: X-API-Key in: header prefix: "Bearer" ```
Token **Token Retrieve Based Authentication** * **Request -** * **Method****\***** -** Choose between *GET* or *POST* * **URL****\*****-** Enter the URL to send the request to. * **Headers -** Add as many headers as required. * **Name** * **Value** * **Query Params -** Add as many query parameters as required. * **Name** * **Value** * **Token Path****\***** -** Enter your **Token Path** for used to retrieve an authentication token. * **Auth injection:** * **In****\***** -** Enter the incoming format of the API: Header or Query. * **Name****\***** -** A **label** assigned to the API key for identification. You can find it depending on where the API key was created. * **Prefix -** Enter a connection prefix if required. * **Suffix -** Enter a connection suffix if required. ```yaml withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded bodyType: urlEncoded bodyParams: - name: grant_type value: client_credentials - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' ``` #### Example * **Type** - Token. Token authentication is a method of authenticating API requests by using a secure token, usually passed in an HTTP header. * **Request** * **method** - `POST`Sends a POST request to obtain an access token. * **url** - `${parameters.domain}/oauth2/token`The OAuth token endpoint. `${parameters.domain}` is a placeholder for value entered in the **Parameters** section. * **headers** - these headers are key-value pairs that provide additional information to the server when making a request. * **name** - `Content-Type` * **value** - `application/x-www-form-urlencoded`Indicates that the request body is formatted as URL-encoded key-value pairs (standard for OAuth token requests). * **Body type** -`urlEncoded` Specifies the request body format is URL-encoded (like `key=value&key2=value2`). * **Body params** * **name** - `grant_type` Required by OAuth 2.0 to specify the type of grant being requested. * **value** - `client_credentials` Used for server-to-server authentication without a user. * **name** - client\_ID * **value** - `${secrets.client_id}`this is a dynamic variable pulled from the value entered in the **Secrets** setting. * **name** - client\_secret * **value** - `${secrets.client_secret}` this is a dynamic variable pulled from the value entered in the **Secrets** setting. * **Token path -** Extracts the access token from the JSON response of an authentication request. It's a JSONPath-like expression used to locate the token in the response body.
Toggle **ON** the Authentication option. * **Auth injection** - This part defines how and where to inject the authentication token (typically an access token) into the requests after it has been retrieved, for example, from an OAuth token endpoint. * **in** -`header`The token should be injected into the HTTP header of the request.This is the most common method for passing authentication tokens. * **Name -**`Authorization`The name of the header that will contain the token. Most APIs expect this to be `Authorization`. * **prefix** - The text added before the token value.`Bearer` is the standard prefix for OAuth 2.0 tokens. * **suffix** -`''`Text added after the token value. In this case, it's empty — nothing is appended.
HMAC Signs the queries using a secret key that is used by the server to authenticate and validate integrity. **Token Retrieve Based Authentication** **Request** * **Generate ID** - Toggle **ON** to generate. * **Generate Timestamp** * **Timezone****\***** -** this field is automatically-filled using your current timezone. * **Format****\***** -** the format for the timestamp syntax (Seconds, Epoch, Epoch Timestamp, RFC1123, RFC1123Z, RFC3339 or custom). Selecting **custom** opens the Go time format option, where you can write your custom syntax e.g. `2 Jan 2006 15:04:05` * **Generate content hash** * **Content hash** * **Hashing algorithm****\*** - select the [hash](/the-workspace/pipelines/actions/transformation/field-transformation/field-transformation-operations/hashing) operation to carry out on the content. * **Encoding****\*** - choose the encoding method. * **Hashing** * **Hashing algorithm****\*** - select the [hash](/the-workspace/pipelines/actions/transformation/field-transformation/field-transformation-operations/hashing) operation to carry out on the content. * **Encoding****\*** - choose the encoding method. * **Secret key****\*** - how to generate the string that will be signed. * **Data to sign****\*** - e.g. `"${request.method}\n${request.contentHash}\napplication/json\n${request.relativeUrl}\n${request.timestamp}"` * **Headers** to be added to the request (name & value). ```yaml withAuthentication: true authentication: type: hmac hmac: request: generateTimestamp: true timestamp: tz: UTC format: EpochMillis hash: secretKey: ${secrets.apiSecret} algorithm: hmac_sha256 encoding: hex dataToSign: "${secrets.apiKey}${request.body}${request.timestamp}" headers: x-logtrust-apikey: ${secrets.apiKey} x-logtrust-timestamp: ${request.timestamp} x-logtrust-sign: ${hmac.hash} ``` #### Example: Authenticate HTTP requests to Microsoft Azure using the HMAC-SHA256 scheme. [Learn how to calculate the HMAC for this API here.](https://learn.microsoft.com/es-es/azure/azure-app-configuration/rest-api-authentication-hmac) ```yaml withAuthentication: true authentication: type: hmac hmac: request: generateTimestamp: true timestamp: tz: UTC format: RFC1123 generateContentHash: true contentHash: algorithm: sha256 encoding: base64 hash: algorithm: hmac_sha256 encoding: base64 secretKey: ${secrets.secretKey} dataToSign: "${request.method}\n${request.relativeUrl}\n${request.timestamp};${request.host};${request.contentHash}" headers: - name: x-ms-date value: ${request.timestamp} - name: x-ms-content-sha256 value: ${request.contentHash} - name: Authorization value: "HMAC-SHA256 Credential=${secrets.accessKeyId}&SignedHeaders=x-ms-date;host;x-ms-content-sha256&Signature=${hmac.hash}" ``` * **Type** - HMAC. **Request Parameters** * **Generate Timestamp** * **Timezone -** UTC * **Format -** RFC1123 * **Generate Content Hash** * **Algorithm** - sha256 * **Encoding** - base64
**Hash** Base64-encoded HMACSHA256 of the *String-To-Sign*. * **Algorithm** - hmac\_sha256 * **Encoding** - base64 * **Secret** **Key** - `${secrets.secretKey}` This variable is retrieved from the **secrets** parameter. * **Data To Sign** - A canonical representation of the request with the format **HTTP\_METHOD** + '\n' + **path\_and\_query** + '\n' + **signed\_headers\_values** `${request.method}\n${request.relativeUrl}\n${request.timestamp};${request.host};${request.contentHash}`
**Headers** * **Name** - `x-ms-date` can be used when the agent cannot directly access the `Date` request header or when a proxy modifies it. If both `x-ms-date` and `Date` are provided, `x-ms-date` takes precedence. * **Value** - `${request.timestamp}` * **Name** - `x-ms-content-sha256` Base64-encoded SHA256 hash of the request body. It must be provided even if there is no body. * **Value** - `${request.contentHash}` * **Name** - `Authorization` Required by the HMAC-SHA256 scheme. * **Value** - `HMAC-SHA256 Credential=${secrets.accessKeyId}&SignedHeaders=x-ms-date;host;x-ms-content-sha256&Signature=${hmac.hash}`
#### Example 2: API HMAC Authentication for Oracle [See here for how to calculate the API HMAC in Oracle](https://docs.oracle.com/en/cloud/saas/marketing/crowdtwist-develop/Developers/HMACAuthentication.html). * **Type** - HMAC. **Request Parameters** * **Generate ID** * **Type -** uuid * **Generate Timestamp** * **Timezone -** UTC * **Format -** Epoch * **Generate Content Hash** * **Algorithm** - sha1 * **Encoding** - `base64` - The binary hash result will be encoded in **Base64** for transmission. **Hash** Base64-encoded HMACSHA256 of the *String-To-Sign*. * **Algorithm** - `hmac_sha256` * **Encoding** - `base64` * **Secret** **Key** - `${secrets.secretKey}` This variable is retrieved from the **secrets** parameter. * **Data To Sign** - `${request.method}\n${request.contentHash}\napplication/json${request.timestamp}\n${request.relativeUrl}`This is the canonical **string-to-sign**: * `${request.method}` - HTTP method (e.g., `GET`, `POST`) * `${request.contentHash}` - Base64 SHA-1 hash of the request body * `"application/json"` - Hardcoded content type * `${request.timestamp}` - Epoch UTC timestamp * `${request.relativeUrl}` - The relative path and query string The `\n` means each element is separated by a **newline**. **Headers** * **Name** - `ct-authorization` * **Value** - `CTApiV2Auth ${parameters.publicKey}:${hmac.hash}` * `CTApiV2Auth` - Authentication scheme name. * `${parameters.publicKey}` - Public key or access ID. * `${hmac.hash}` - The generated HMAC-SHA256 signature from the **hash** section. * **Name** - `ct-timestamp` * **Value** - `${request.timestamp}` the same Epoch UTC timestamp generated earlier. ```yaml withAuthentication: true authentication: type: hmac hmac: request: generateId: true idType: uuid generateTimestamp: true timestamp: tz: UTC format: Epoch generateContentHash: true contentHash: algorithm: sha1 encoding: base64 hash: algorithm: hmac_sha256 encoding: base64 secretKey: ${secrets.secretKey} dataToSign: "${request.method}\n${request.contentHash}\napplication/json${request.timestamp}\n${request.relativeUrl}" headers: - name: x-ct-authorization value: CTApiV2Auth ${parameters.publicKey}:${hmac.hash} - name: x-ct-timestamp value: ${request.timestamp} ```
Akamai EdgeGrid Authenticates using Akamai EdgeGrid endpoints. **Akamai EdgeGrid Authentication** **Request** * **Client Token****\*** * **Access Token****\*** * **Client Secret ****\*** 1. Log in to the [Akamai Control Center](https://control.akamai.com/) 2. Navigate to **Identity & Access Management > API Users** 3. Either select an existing API user or create a new one 4. Click **Create API Client** or view existing credentials 5. In the credentials view, you'll find the **Client Token** along with the **Client Secret** and **Access Token** **Advanced Configuration** * **Maximum body size in bytes or empty for whole body -** This configuration parameter defines the maximum allowed size (in bytes) for request bodies sent to the Akamai authentication endpoint. When set to a specific byte value, it limits the amount of data that will be processed, preventing potential resource exhaustion from oversized payloads. When left empty, the system will process the entire body regardless of size * **Headers added to the signature -** HTTP header fields that are incorporated into the cryptographic signature calculation for request verification. These headers become part of the signed content that Akamai uses to validate the authenticity and integrity of the request. ```yaml withAuthentication: true authentication: type: akamai akamai: clientSecret: ${secrets.clientSecret} accessToken: ${secrets.accessToken} clientToken: ${secrets.clientToken} ```
Bloodhound Authenticates using [BloodHound API.](https://bloodhound.specterops.io/integrations/bloodhound-api/working-with-api) * **Token ID****\*** * A unique identifier (usually a UUID/GUID) visible in the interface even after token creation e.g. `tkn_12a34567-89b0-12c3-d456-789012ef3456` * **Token Key****\*** * Log in to the BloodHound CE or Enterprise web interface * Navigate to **Settings** or **User Profile** (typically accessible from the top-right user menu) * Select **API Tokens** or **Access Tokens** * Either view existing tokens or create a new one by clicking **Generate Token** * Provide a name/description for the token and set appropriate permissions * After creation, the Token Key will be displayed once (copy it immediately as it won't be shown again) ```yaml withAuthentication: true authentication: type: bloodHound bloodHound: tokenId: ${secrets.tokenId} tokenKey: ${secrets.tokenKey} ```
*** ### Retry Toggle **ON** to allow for retries and to configure the specifics.
ParameterDescription
Retry Type*

  • Fixed - Retries the failed operation after a constant, fixed interval every time e.g. the same amount of time between each retry attempt

    • Interval* - enter the amount of time to wait e.g. 5s.

  • Exponential - Retries the failed operation after increasingly longer intervals to avoid overwhelming the service. The delay grows with each retry attempt.

    • Initial delay* - The starting delay before the first retry attempt to ensure there’s at least some delay before retrying to avoid immediate re-hits. For example, an initial delay of 2s equals a retry pattern of 2s, 4s, 8s, 16s, etc.
    • Maximum delay* - The maximum wait time allowed between retries to prevent the retry delay from growing indefinitely. For example, an initial delay of 2s and a maximum delay of 10s equals a delay progression of 2s, 4s, 8s, 10s, 10s, etc.
    • Increasing factor* - The multiplier used to calculate the next delay interval, determining how quickly the delay grows after each failed attempt.
Retry after response header

Used to define how long to wait before making another request e.g. HTTP 429 Too Many Requests or HTTP 503 Service Unavailable.

  • Header - Follow the header syntax for the header.

  • Format - The format for the header syntax (Seconds, Epoch, Epoch Timestamp, RFC1123, RFC1123Z, RFC3339).

    • e.g. wait 120 seconds Retry-After: 120
    • e.g. epoch timestamp Retry-After: Wed, 21 Oct 2025 07:28:00 GMT
### Retry on errors processing body Toggle **ON** to allow retry on body failures. When a response cannot be parsed, it will be retried the number of times specified in the **maximum number of retries** field. ### Throttling Use throttling to intentionally limit the rate at which the HTTP requests are sent to the API or service. **Throttling Type****\*** {% tabs %} {% tab title="Client" %} The client itself controls and limits the rate at which it sends requests.
ParameterDescription
Client type*

How to manage the rate of requests.

  • Rate - the client is restricted by the data transfer rate or request rate over time.

    • Maximum requests* - The maximum number of requests (or amount of data) to make within a specified time interval.
    • Call interval* - The sliding or fixed window of time used to calculate the rate.
    • Number of burst requests* - the number of requests that can exceed the normal rate temporarily before throttling kicks in to allow short bursts of traffic over the limit to accommodate sudden spikes without immediate blocking. e.g. if the max rate is 10 requests/sec, and burst is 5, the client could make up to 15 requests instantly, but then throttling will slow down after the burst.

  • Fixed delay - The server enforces a fixed wait time after each request before allowing the client to make the next request. Instead of limiting by rate (requests per second) or volume, it just inserts a pause/delay between requests.

    • Call interval* - The sliding or fixed window of time used to calculate the delay.
**Example** ```yaml withThrottling: true throttling: type: client client: type: rate rate: maxRequests: 42 interval: 1s burst: 123 ``` {% endtab %} {% tab title="Server" %} The server controls the rate at which it sends data.
ParameterDescription
Wait response header*

These headers inform how long to wait before retrying and how many requests remaining.

  • Header Type* - Enter the header to instruct that to do e.g. wait., Retry-After, etc.
  • Format - The format for the header syntax (Seconds, Epoch, Epoch Timestamp, RFC1123, RFC1123Z, RFC3339). e.g. wait 120 seconds Retry-After: 120 e.g. epoch timestamp Retry-After: Wed, 21 Oct 2025 07:28:00 GMT
Reset response header

Indicates when a rate limit or throttle window resets, allowing the client to resume normal activity (e.g., making more requests or pulling more data).

  • Header Type* - Enter the header to instruct that to do e.g. wait., Retry-After, etc.
  • Format - The format for the header syntax (Seconds, Epoch, Epoch Timestamp, RFC1123, RFC1123Z, RFC3339).
Remaining response header*How many requests or units of usage the puller can still make within the current time window before hitting the limit and being throttled.
**Example** ```yaml withThrottling: true throttling: type: server server: waitResponseHeader: name: Retry-After format: seconds ``` {% endtab %} {% endtabs %} *** ### Enumeration phase The enumeration phase is an optional step in data collection or API integration workflows, where the system first retrieves a list of available items (IDs, resource names, keys, etc.) before fetching detailed data about each one. Identify the available endpoints, methods, parameters, and resources exposed by the API. This performs initial data discovery to feed the collection phase and makes the results available to the Collection Phase via variable interpolation (inputs.\*). Can use: * `${parameters.xxx}` * `${secrets.xxx}` * `${temporalWindow.xxx}` (if configured) * `${pagination.xxx}` Pagination variables
ParameterDescription
Pagination Type*

Select one from the drop-down. Pagination type is the method used to split and deliver large datasets in smaller, manageable parts (pages), and how those pages can be navigated during discovery.

Each pagination method manages its own state and exposes specific variables that can be interpolated in request definitions (e.g., URL, headers, query params, body).

None

  • Description: No pagination; only a single request is issued.

    • Repeat until: No repeat to ignore, or No data to repeat the request until no data is returned.

PageNumber/PageSize

  • Description: Pages are indexed using a page number and fixed size.

  • Configuration:

    • pageSize: page size

  • Exposed Variables:

    • ${pagination.pageNumber}
    • ${pagination.pageSize}

Offset/Limit

  • Description: Uses offset and limit to fetch pages of data.

  • Configuration:

    • Limit: max quantity of records per request

  • Exposed Variables:

    • ${pagination.offset}
    • ${pagination.limit}

From/To

  • Description: Performs pagination by increasing a window using from and to values.
  • Configuration: limit: max quantity of records per request

  • Exposed Variables:

    • ${pagination.from}
    • ${pagination.to}

Web Linking (RFC 5988)

  • Description: Parses the Link header to find the rel="next" URL.
  • Exposed Variables: None

Next Link at Response Header

  • Description: Follows a link found in a response header.

  • Configuration:

    • headerName: header name that contains the next link
  • Exposed Variables: None

Next Link at Response Body

  • Description: Follows a link found in the response body.

  • Configuration:

    • nextLinkSelector: path to next link sent in response payload
  • Exposed Variables: None

Cursor

  • Description: Extracts a cursor value from each response to request the next page.

  • Configuration:

    • cursorSelector: path to the cursor sent in response payload

  • Exposed Variables:

    • ${pagination.cursor}
**Output**
ParameterDescription
Select*If your connection does not require authentication, leave as None. Otherwise, choose the authentication type and enter the details. A JSON selector expression to pick a part of the response e.g. '.data'.
FilterA JSON expression to filter the selected elements. Example: '.films | index("Tangled")'.
MapA JSON expression to transform each selected element into a new event.
Example: '{characterName: .name}'.
Output Mode*

Choose between

  • Element: emits each transformed element individually as an event.
  • Collection: emits all transformed items as a single array/collection as an event.
Enumeration example 
enumerationPhase:
  paginationType: offsetLimit
  limit: 100
  request:
    responseType: json
    method: GET
    url: ${parameters.domain}/alerts/queries/alerts/v2
    queryParams:
      - name: offset
        value: ${pagination.offset}
      - name: limit
        value: ${pagination.limit}
      - name: filter
        value: created_timestamp:>'${temporalWindow.from}'+created_timestamp:<'${temporalWindow.to}'
  output:
    select: ".resources"
    map: "."
    outputMode: collection
* **Pagination type** - `offset/Limit`Uses classic pagination with **`offset`** and **`limit`** to page through results, fetching data in batches (pages) — `limit` determines page size, `offset` determines where to start. * **Limit** - Retrieves up to **100 records per request.** This value is used in the `limit` query parameter to control batch size. * **Request** - Describes the API request that will be sent during enumeration. * **Response type** - Specifies the expected response format. Here, the system expects a `JSON` response. * **Method** - The HTTP method to use for this request. `GET` is used to retrieve data from the server. * **URL** - `${parameters.domain}` is a placeholder variable that will be replaced by the domain value you entered in the **Parameters** section.
**Query params** - These are query string parameters appended to the URL. * `${pagination.offset}`controls where to start in the dataset. Used for pagination. * `${pagination.limit}`replaced with the limit value you entered for number of records to retrieve per request (100). * Filters data to only return alerts created within a specific time window. `${temporalWindow.from}` and `${temporalWindow.to}` are dynamically filled in with RFC3339 or epoch timestamps, depending what you have configured.
**output** - Describes how to extract and interpret the results from the JSON response. * **select -** `.resources`Looks for a field named `resources` in the response JSON. This is where the array of items lives. * **map - `.`**Each item under `.resources` is returned as-is. No transformation or remapping. * **outputMode - `collection`**The result is treated as a collection (array) of individual items. Used when you expect multiple items and want to pass them along for further processing.
*** ### Collection phase The collection phase in an HTTP Puller is the part of the process where the system actively pulls or retrieves data from an external API using HTTP requests. The collection phase is mandatory. This is where the final data retrieval happens (either directly or using IDs/resources generated by an enumeration phase). The **collection phase** involves gathering actual data from an API after the enumeration phase has mapped out endpoints, parameters, and authentication methods. It supports dynamic variable resolution via the variable resolver and can use data exported from the Enumeration Phase, such as: * `${parameters.xxx}` * `${secrets.xxx}` * `${temporalWindow.xxx`} * `${inputs.xxx}` (from Enumeration Phase) * `${pagination.xxx}*` **Inputs** In collection phases, you can define variables to be used elsewhere in the configuration (for example, in URLs, query parameters, or request bodies). Each variable definition has the following fields:
ParameterDescription
NameThe variable name (used later as ${inputs.name} in the configuration).
SourceUsually "input", indicating the value comes from the enumeration phase’s output.
ExpressionA JSON expression applied to the input to extract or transform the needed value.
FormatControls how the variable is converted to a string (see Variable Formatting below). Eg: json.
### Retry Toggle **ON** to allow for retries and to configure the specifics.
ParameterDescription
Retry Type*

  • Fixed - Retries the failed operation after a constant, fixed interval every time e.g. the same amount of time between each retry attempt

    • Interval* - enter the amount of time to wait e.g. 5s.

  • Exponential - Retries the failed operation after increasingly longer intervals to avoid overwhelming the service. The delay grows with each retry attempt.

    • Initial delay* - The starting delay before the first retry attempt to ensure there’s at least some delay before retrying to avoid immediate re-hits. For example, an initial delay of 2s equals a retry pattern of 2s, 4s, 8s, 16s, etc.
    • Maximum delay* - The maximum wait time allowed between retries to prevent the retry delay from growing indefinitely. For example, an initial delay of 2s and a maximum delay of 10s equals a delay progression of 2s, 4s, 8s, 10s, 10s, etc.
    • Increasing factor* - The multiplier used to calculate the next delay interval, determining how quickly the delay grows after each failed attempt.
Retry after response header

Used to define how long to wait before making another request e.g. HTTP 429 Too Many Requests or HTTP 503 Service Unavailable.

  • Header - Follow the header syntax for the header.

  • Format - The format for the header syntax (Seconds, Epoch, Epoch Timestamp, RFC1123, RFC1123Z, RFC3339).

    • e.g. wait 120 seconds Retry-After: 120
    • e.g. epoch timestamp Retry-After: Wed, 21 Oct 2025 07:28:00 GMT
### Throttling Use throttling to intentionally limit the rate at which the HTTP requests are sent to the API or service. **Throttling Type****\*** {% tabs %} {% tab title="Client" %} The client itself controls and limits the rate at which it sends requests.
ParameterDescription
Client type*

How to manage the rate of requests.

  • Rate - the client is restricted by the data transfer rate or request rate over time.

    • Maximum requests* - The maximum number of requests (or amount of data) to make within a specified time interval.
    • Call interval* - The sliding or fixed window of time used to calculate the rate.
    • Number of burst requests* - the number of requests that can exceed the normal rate temporarily before throttling kicks in to allow short bursts of traffic over the limit to accommodate sudden spikes without immediate blocking. e.g. if the max rate is 10 requests/sec, and burst is 5, the client could make up to 15 requests instantly, but then throttling will slow down after the burst.

  • Fixed delay - The server enforces a fixed wait time after each request before allowing the client to make the next request. Instead of limiting by rate (requests per second) or volume, it just inserts a pause/delay between requests.

    • Call interval* - The sliding or fixed window of time used to calculate the delay.
{% endtab %} {% tab title="Server" %} The server controls the rate at which it sends data.
ParameterDescription
Wait response header*

These headers inform how long to wait before retrying and how many requests remaining.

  • Header Type* - Enter the header to instruct that to do e.g. wait., Retry-After, etc.
  • Format - The format for the header syntax (Seconds, Epoch, Epoch Timestamp, RFC1123, RFC1123Z, RFC3339). e.g. wait 120 seconds Retry-After: 120 e.g. epoch timestamp Retry-After: Wed, 21 Oct 2025 07:28:00 GMT
Reset response header

Indicates when a rate limit or throttle window resets, allowing the client to resume normal activity (e.g., making more requests or pulling more data).

  • Header Type* - Enter the header to instruct that to do e.g. wait., Retry-After, etc.
  • Format - The format for the header syntax (Seconds, Epoch, Epoch Timestamp, RFC1123, RFC1123Z, RFC3339).
Remaining response header*How many requests or units of usage the puller can still make within the current time window before hitting the limit and being throttled.
{% endtab %} {% endtabs %}
ParameterDescription
Pagination Type*Choose how the API organizes and delivers large sets of data across multiple pages—and how that affects the process of systematically collecting or extracting all available records.
#### Output
ParameterDescription
Select*If your connection does not require authentication, leave as None. Otherwise, choose the authentication type and enter the details. A JSON selector expression to pick a part of the response e.g. '.data'.
FilterA JSON expression to filter the selected elements. Example: '.films | index("Tangled")'.
MapA JSON expression to transform each selected element into a new event.
Example: '{characterName: .name}'.
Output Mode*

Choose between

  • Element: emits each transformed element individually as an event.
  • Collection: emits all transformed items as a single array/collection as an event.
Collection example Let´s say you have the following SIEM Integration events from Sophos. 
collectionPhase:
  paginationType: cursor
  cursorSelector: ".next_cursor"
  initialRequest:
    method: GET
    url: "${inputs.dataRegionURL}/siem/v1/events"
    headers:
      - name: Accept
        value: application/json
      - name: Accept-Encoding
        value: gzip, deflate
      - name: X-Tenant-ID
        value: "${inputs.tenantId}"
    queryParams:
      - name: from_date
        value: "${temporalWindow.from}"
    bodyParams: []
  nextRequest:
    method: GET
    url: "${inputs.dataRegionURL}/siem/v1/events"
    headers:
      - name: Accept
        value: application/json
    queryParams:
      - name: cursor
        value: "${pagination.cursor}"
    bodyParams: []
  output:
    select: ".result"
    filter: "."
    map: "."
    outputMode: element
* Pagination type - `cursor`. If you select the cursor type, you retrieve the data in chunks (pages) using a cursor token, which points to the position in the dataset where the next page of results should start. * Cursor selector - The cursor selector tells the HTTP Puller where to find the cursor value in the API response so it can be saved and used in the next request e.g. `.next_cursor` * Initial request - We fetch the first set of results, the response including the cursor token (e.g. timestamp or ID). * method - `GET` to fetch the results. * url - The URL is composed of various elements: * `https://${inputs.dataRegionURL}`- these variables are taken from the values you entered in the **Parameters** section of the HTTP Pull settings. * `/siem/v1/` -API base path — indicates you're calling version 1 of the SIEM API. * `events`- indicates the specific endpoint being accessed. `events` general category of the API (event-related). * headers - these headers are key-value pairs that provide additional information to the server when making a request. * **name** - `Accept` * **value** - `application/json` tells the server that the client expects the response to be in JSON format, a standard HTTP header used for content negotiation. * Next request - send the cursor token back to the server using a parameter (e.g., ?cursor=abc123) to get the next page of results. The server returns the next chunk of data and a new cursor. Repeat until no more data or the server returns a `has_more: false flag.`method * **Output** * select - `.result` Selects the part of the response to extract. This is a JSONPath-like expression that tells the puller where to find the list or array of items in the response. * map - `.` Maps each selected item as-is, keeping each object unchanged. It passes through each item without transforming it. If you needed to restructure or extract specific fields from each item, you would replace `.` with a field mapping (e.g., `.id`, `{ "id": .id, "name": .username }`, etc.). * output mode - `element` Controls the output format. Each item from the `select` result will be emitted individually using `element`. This is useful for event stream processing, where each object (e.g., an alert or event) is treated as a separate record. Other possible values (depending on the platform) might include `array` (emit as a batch) or `raw` (emit as-is).
## Ports The HTTP Pull Listener has two output ports: * **Default port** - Events are sent through this port if no error occurs while processing them. * **Error port** - Events are sent through this port if an error occurs while processing them. {% hint style="warning" %} The error message is provided in a free-text format and may change over time. Please consider this if performing any post-processing based on the message content. {% endhint %} ## Examples ### 1. Basic GET Puller Here's a simple example of using the HTTP Puller collector with parameters for a basic GET request. No authentication, no pagination, just pulling JSON data from an API endpoint. Keep **Config as YAML, Temporal window, Authentication** and **Enumeration phase** as `OFF.` * **Collection phase** * Pagination type - `none` Indicates that you only need one request to retrieve all data at once. * Repeat until: ***No repeat*** to ignore, or ***No data*** to repeat the request until no data is returned. * **Request** * **Response type -** `json`Tells the puller to expect a JSON response. * **Method:** `GET` Performs a basic HTTP GET request. * **URL**: Constructed from the `parameters.domain` and `parameters.path https://{{parameters.domain}}{{parameters.path}}` * **Headers**: Set standard headers and include the API key. * **Output:** * **Select:`.logs`** Tells the system where to find the list of log entries in the response. * **Output mode:** `element` each object inside `.logs` will be extracted as a separate output element e.g. ```json { "logs": [ { "timestamp": "2024-12-01T12:00:00Z", "event": "user_login" }, { "timestamp": "2024-12-01T12:05:00Z", "event": "file_upload" } ] } ```
### 2. Make an HTTP request using offset and limit pagination Instead of displaying the results in a scrollable list, we will use offset/limit pagination to fetch data in pages. * **Pagination type** - `offset/Limit` We control how many records are returned at a time (`limit`) and choose where to start each request (`offset` or `skip` parameter) * **Zero Index** - `false` * **Limit****\*** - `50` * **Request -** The request to be repeated, with `offset` and `limit` automatically incremented per iteration. * **Response type****\*** - `Json` * **Method****\*** - `GET` * **URL****\*** - `https://example.com/items` * **Query params** The API supports pagination through query parameters: * Name - `skip` * Value - `${pagination.offset}"` the number of records to skip before returning results * Name - `limit` * Value - `${pagination.limit}` uses the **limit** entered (`50`) as the maximum number of records to return in one request. ```yaml collectionPhase paginationType: "offsetLimit" limit: 50 isZeroIndex: false request: method: "GET" url: "https://example.com/items" queryParams: - name: skip value: "${pagination.offset}" name: limit value: "${pagination.limit}" ```
### 3. Enumeration + Collection with `responseBodyLink` This example defines a data extraction workflow that 1. Enumerates through a paginated API endpoint using `responseBodyLink`. 2. Filters and transforms specific data from the paginated results. 3. Collects further data based on the enumerated output using individual requests. It also uses a **temporal window** to scope or schedule the data extraction process. ```yaml # Temporal window (optional) # Generated variables: $temporalWindow.from, $temporalWindow.to temporalWindow: duration: 5m offset: 10m tz: UTC format: RFC3339 enumerationPhase: paginationType: responseBodyLink: nextLinkSelector: ".info.nextPage" request: method: "GET" url: "https://api.cyberintel.dev/iocs" headers: - name: accept value: "application/json" bodyExpression: expression: "(.data | length) == 50" output: select: '.data' filter: '.threatType == "Ransomware"' map: '._id' outputMode: "element" collectionPhase: variables: - name: id source: input expression: "." paginationType: none repeatUntilNoData: true request: method: "GET" url: "https://api.cyberintel.dev/iocs/${id}" headers: - name: accept value: "application/json" output: select: ".data" filter: "" map: "{iocName: .name}" outputMode: "element" ``` #### **Enumeration** The **enumeration** defines how to gather data in a paginated manner from the Cyber Threat Intelligence API using the `responseBodyLink` pagination strategy. * **Pagination Type -** The type is `Next Link At Response Body` * **Selector -** The next page link is found using the JSON path `".info.nextPage"` This suggests that the response will contain a field `info.nextPage` with the URL of the next page of results. For example, the response might look like: ```json { "info": { "nextPage": "https://api.cyberintel.dev/iocs?page=2" }, "data": [ ... ] } ``` * **Response type -** `JSON` * **Method -** `GET`. The HTTP method is **GET** to fetch the data. * **URL** - The initial URL for the request is `"https://api.cyberintel.dev/iocs"`, where the IOCs are listed. * **headers -** The `Accept` header specifies that the response should be in **JSON** format. **Output** * **Select -** The `.data` array from the response is selected for further processing. This array contains the actual IOC data. * **Filter -** The `filter` expression `'.threatType == "Ransomware"'` selects only those IOCs where the `threatType` is `"Ransomware"`. This is how we focus on ransomware-related indicators. * **Map -** The `map` expression `'._id'` extracts the `._id` field from each IOC that passed the filter. This results in a list of **IOC IDs** that match the ransomware threat type. * **Output Mode -** `element` indicates that each IOC ID (element) is treated as an individual item, rather than as a group or array. **Result:** After processing the pages, we will have a list of **ransomware IOC IDs**. **Collection** Once the enumeration process gathers a list of **IOC IDs** related to ransomware, the **collection** section is responsible for retrieving more detailed information for each of those IOCs. **variables -** This section defines variables used in the collection step. * **Name** - `id`: The variable `id` represents each individual IOC ID from the enumeration output. * **Source -** The `source: input` means that the IDs come from the output of the previous enumeration step. * **Expression -** `expression: "."` simply takes each item from the input (the IOC IDs). **HTTP Request for Detailed IOC Information** * **Pagination type:** The type is `"none"`, indicating no additional processing is needed before making the request. * **Response type** - `JSON`. * **Method:** The HTTP method is **GET**, to fetch detailed information about each IOC. * **Url:** The URL for each IOC is dynamic, with the IOC ID substituted in the URL (`${id}`). For example, if `id = "a1b2"`, the URL would be `https://api.cyberintel.dev/iocs/a1b2`. * **Headers:** The `Accept: "application/json"` header ensures the response is in JSON format. **Output Selection and Mapping** * **Select:** This selects the `.data` field from the response, which contains the detailed information for the IOC. * **Filter:** No additional filtering is applied. * **Map:** The map expression `"{iocName: .name}"` creates a new object with the `iocName` key, mapping it to the `.name` of the IOC from the response. * **Output Mode:** `outputMode: "element"` means each IOC’s name will be treated as an individual output item. **Result:** Each IOC name (or other information, if mapped) will be saved to a file.
### 4. Enumeration (collection output) + Collection (POST with `bodyRaw`) **Temporal window** defines a 5-minute slice of time, offset 10 minutes ago. **Enumeration** step: * Makes a paginated GET to `/posts`. * Extracts IDs from posts within the time window. * Produces a **collection of IDs**. **Collection** step: * Uses those IDs in a POST request. * Filters, maps, and outputs enriched objects (`id, title, status`). * Saves results to a file. 
# Temporal window (optional)
temporalWindow:
  duration: 5m
  offset: 10m
  tz: UTC
  format: RFC3339
enumerationPhase:
  httpRequest:
    type: "page"
    page:
      pageSize: 50
      request:
        method: "GET"
        url: "https://api.fake-rest.refine.dev/posts"
        headers:
          Accept: "application/json"
        queryParams:
          from: "${temporalWindow.from}"
          to: "${temporalWindow.to}"
          _page: "${pagination.pageNumber}"
          _per_page: "${pagination.pageSize}"
  output:
    select: '.'
    # filter: '.language == 3'
    map: '{id: .id}'
    outputMode: "collection"
collectionPhase:
  variables:
    - name: ids
      source: input
      expression: "."
      format: "json"
  httpRequest:
    type: "none"
    none:
      request:
        method: "POST"
        url: "https://api.fake-rest.refine.dev/posts"
        headers:
          Accept: "application/json"
        bodyType: "raw"
        bodyRaw: |
          {
            "ids": ${inputs.ids}
          }
  output:
    select: "."
    filter: ".id > 10"
    map: "{id: .id, title: .title, status: .status}"
    outputMode: "element"
* **Duration -** `5m` window size is 5 minutes. * **Offset -** `10m` shifts the window back 10 minutes from “now”. So if current UTC is `12:00`, the range would be `11:45 – 11:50`. * **Time zone -** `UTC` * **Format -** `RFC3339` output format for timestamps (e.g., `2025-08-20T12:00:00Z`). The variables `${temporalWindow.from}` and `${temporalWindow.to}` get auto-populated with these calculated times. **Enumeration** * **Pagination type** - `page number/page size` * **Page size:** `50` fetch 50 records per request. * **Request** * **Response type** - `JSON` * **Method** - `GET` * **URL** - `https://api.fake-rest.refine.dev/posts` * **Query Params** **1.From: "${temporalWindow\.from}"** * Inserts the **start timestamp** of the time window. `${temporalWindow.from}` is automatically computed based on your `temporalWindow` configuration e.g.\ If `now = 12:00 UTC`, `offset = 10m`, and `duration = 5m` = `temporalWindow.from = 11:45 UTC` (start) In the request, this becomes something like: ``` ?from=2025-08-20T11:45:00Z ``` **2. to: "${temporalWindow\.to}"** Inserts the **end timestamp** of the time window e.g. `temporalWindow.to = 11:50 UTC` (end). In the request, this becomes: ``` &to=2025-08-20T11:50:00Z ``` So together, `from` and `to` tell the API: > “Only give me records between 11:45 and 11:50 UTC.” **3. \_page: "${pagination.pageNumber}"** This is a built-in pagination variable. `${pagination.pageNumber}` auto-increments as the system makes repeated requests to fetch all pages e.g. First request `_page=1` Second request `_page=2` etc. This ensures you don’t just get the first batch, but all results page by page. **4. \_per\_page: "${pagination.pageSize}”** Controls how many records to fetch per page. This pulls from your earlier configuration ```yaml page: pageSize: 50 ``` So each request includes: `&_per_page=50` ``` &_per_page=50 ``` * **Select - `'.'`**selects the entire JSON response. * **Filter -** would filter only records where `.language == 3`. * **Map** - extracts only `{id: .id}` for each record. * **Output Mode -** `collection` outputs an **array of items** (instead of single elements). ```json [ {"id": 1}, {"id": 2}, {"id": 3} ] ```
**Collection (POST with BodyRaw)** * **Pagination Type** - `Next link at response body` * **Selector** - `"."` take the full collection. * **Response Type** - `json` keep it as JSON (array of IDs). * **Method -** `POST` to send data. * **URL** - `https://api.fake-rest.refine.dev/posts` * **Body Type:** `raw` freeform JSON payload. * **Body Content -** sends the IDs collected in the enumeration: `ids": ${inputs.ids}` * **Select:** `"."` take the full response. * **Filter -** `".id > 10"` only keep posts with ID greater than 10. * **Map** - reduce each record to `{id, title, status}`. * **Output Mode -** `element` output individual objects, one at a time. # Collect data from 1Password Use the **HTTP Pull** Listener to collect data from the [1Password Events API](https://www.1password.dev/events-api). We currently support the following types of data: * [Audit events](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-1password/audit-events) * [Item usages](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-1password/item-usages) * [Sign in attempts](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-1password/sign-in-attempts) # Audit events ## Overview Get a list of 1Password audit events through the [1Password Events API](https://www.1password.dev/events-api/reference/audit-events) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `1passwordkey` will reference your 1Password API key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `1passwordkey`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: "2006-01-02T15:04:05-07:00" withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".cursor" initialRequest: responseType: json method: POST url: "https://events.1password.com/api/v2/auditevents" headers: - name: Accept value: application/json - name: Content-Type value: application/json - name: Authorization value: "Bearer ${secrets.1passwordkey}" bodyType: raw bodyRaw: | { "limit": 100, "start_time": "${temporalWindow.from}", "end_time": "${temporalWindow.to}" } nextRequest: responseType: json method: POST url: "https://events.1password.com/api/v2/auditevents" headers: - name: Accept value: application/json - name: Content-Type value: application/json - name: Authorization value: "Bearer ${secrets.1passwordkey}" bodyType: raw bodyRaw: | { "cursor": "${pagination.cursor}" } output: select: ".items" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `1m` * **Offset** - `1m` * **Format** - `RFC3339` **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.cursor` * **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://events.1password.com/api/v2/auditevents` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Bearer ${secrets.1passwordkey}` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "limit": 100, "start_time": "${temporalWindow.from}", "end_time": "${temporalWindow.to}" } ``` * **Next Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://events.1password.com/api/v2/auditevents` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Bearer ${secrets.1passwordkey}` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "cursor": "${pagination.cursor}" } ``` * **Output** * **Select****\*** - `.items` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Item usages ## Overview Get a list of 1Password item usage events through the [1Password Events API](https://www.1password.dev/events-api/reference/items-usage) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `1passwordkey` will reference your 1Password API key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `1passwordkey`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: "2006-01-02T15:04:05-07:00" withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".cursor" initialRequest: responseType: json method: POST url: "https://events.1password.com/api/v2/itemusages" headers: - name: Accept value: application/json - name: Content-Type value: application/json - name: Authorization value: "Bearer ${secrets.1passwordkey}" bodyType: raw bodyRaw: | { "limit": 100, "start_time": "${temporalWindow.from}", "end_time": "${temporalWindow.to}" } nextRequest: responseType: json method: POST url: "https://events.1password.com/api/v2/itemusages" headers: - name: Accept value: application/json - name: Content-Type value: application/json - name: Authorization value: "Bearer ${secrets.1passwordkey}" bodyType: raw bodyRaw: | { "cursor": "${pagination.cursor}" } output: select: ".items" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `1m` * **Offset** - `1m` * **Format** - `RFC3339` **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.cursor` * **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://events.1password.com/api/v2/itemusages` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Bearer ${secrets.1passwordkey}` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "limit": 100, "start_time": "${temporalWindow.from}", "end_time": "${temporalWindow.to}" } ``` * **Next Request** * **Response type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://events.1password.com/api/v2/itemusages` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Bearer ${secrets.1passwordkey}` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "cursor": "${pagination.cursor}" } ``` * **Output** * **Select****\*** - `.items` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Sign in attempts ## Overview Get a list of 1Password item usage events through the [1Password Events API](https://www.1password.dev/events-api/reference/sign-in-attempt) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `1passwordkey` will reference your 1Password API key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `1passwordkey`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: "2006-01-02T15:04:05-07:00" withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".cursor" initialRequest: responseType: json method: POST url: "https://events.1password.com/api/v2/signinattempts" headers: - name: Accept value: application/json - name: Content-Type value: application/json - name: Authorization value: "Bearer ${secrets.1passwordkey}" bodyType: raw bodyRaw: | { "limit": 100, "start_time": "${temporalWindow.from}", "end_time": "${temporalWindow.to}" } nextRequest: responseType: json method: POST url: "https://events.1password.com/api/v2/signinattempts" headers: - name: Accept value: application/json - name: Content-Type value: application/json - name: Authorization value: "Bearer ${secrets.1passwordkey}" bodyType: raw bodyRaw: | { "cursor": "${pagination.cursor}" } output: select: ".items" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `1m` * **Offset** - `1m` * **Format** - `RFC3339` **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.cursor` * **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://events.1password.com/api/v2/signinattempts` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Bearer ${secrets.1passwordkey}` * **Body Type****\*** - `Raw` * **Body Content****\*** - ``` | { "limit": 100, "start_time": "${temporalWindow.from}", "end_time": "${temporalWindow.to}" } ``` * **Next Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://events.1password.com/api/v2/signinattempts` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Bearer ${secrets.1passwordkey}` * **Body Type****\*** - `Raw` * **Body Content****\*** - ``` | { "cursor": "${pagination.cursor}" } ``` * **Output** * **Select****\*** - `.items` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Abnormal Use the **HTTP Pull** Listener to collect data from the [Abnormal REST API](https://abnormalsecurity.my.site.com/knowledgebase/s/article/Abnormal-REST-API-Integration). This API is for managing threats to an organization identified by **Abnormal Security**. The organization should be integrated with Abnormal Security and enabled for real-time detection of malicious emails. This API uses bearer authentication to retrieve events through paginated API calls. {% hint style="warning" %} This API restricts the IPs that consume the data from them through an IP whitelisting mechanism. To allow your organization’s IPs, your IP must be in the Abnormal portal specific IPv4 / IPv6 address list, or a range of addresses using a CIDR block. {% endhint %} We currently support the following types of data: * [Abuse campaign](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-abnormal/abuse-campaigns) * [Audit logs](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-abnormal/audit-logs) * [Cases](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-abnormal/cases) * [Threats](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-abnormal/threats) * [Vendor cases](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-abnormal/vendor-cases) # Abuse campaigns ## Overview Get a list of submitted campaigns through the [Abnormal REST API](https://abnormalsecurity.my.site.com/knowledgebase/s/article/Abnormal-REST-API-Integration) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `abnormalToken` will reference your Abnormal bearer token. To do it, click **Add element** and enter a **Name** for the secret (in this case, `abnormalToken`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: true enumerationPhase: paginationType: "page" pageSize: 100 isZeroIndex: false request: responseType: json method: GET url: https://api.abnormalplatform.com/v1/abusecampaigns headers: - name: authorization value: Bearer ${secrets.abnormalToken} queryParams: - name: pageNumber value: "${pagination.pageNumber}" - name: pageSize value: "${pagination.pageSize}" - name: filter value: receivedTime gte ${temporalWindow.from} lte ${temporalWindow.to} output: select: "[.campaigns[].campaignId]" map: "." outputMode: element collectionPhase: variables: - source: input name: campaignId expression: "." format: "" paginationType: none request: method: GET url: https://api.abnormalplatform.com/v1/abusecampaigns/${inputs.campaignId} headers: - name: authorization value: Bearer ${secrets.abnormalToken} output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Enumeration Phase** Toggle **ON** to configure the enumeration phase. This API endpoint requires an initial request that will provide a list of alert IDs. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `page` * **Zero index****\*** - `false` * **Page Size****\*** - `100` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/abusecampaigns` * **Headers** * **Name** - `authorization` * **Value -** `Bearer ${secrets.abnormalToken}` * **Query Params** * **Name** - `pageNumber` * **Value** - `${pagination.pageNumber}` * **Name** - `pageSize` * **Value** - `${pagination.pageSize}` * **Name** - `Filter` * **Value** - `receivedTime gte ${temporalWindow.from} lte ${temporalWindow.to}` * **Output** * **Select****\*** - `[.campaigns[].campaignId]` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Name** - `campaignId` * **Expression** - `.` * **Format** - `""` * **Pagination Type****\*** - `None` * **Request** * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/abusecampaigns/${inputs.campaignId}` * **Headers** * **Name** - `Authorization` * **Value** - `Bearer ${secrets.abnormalToken}` * **Output** * **Select** - `.` * **Map** - `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Audit logs ## Overview Get a list of audit logs through the [Abnormal REST API](https://abnormalsecurity.my.site.com/knowledgebase/s/article/Abnormal-REST-API-Integration) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `abnormalToken` will reference your Abnormal bearer token. To do it, click **Add element** and enter a **Name** for the secret (in this case, `abnormalToken`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "page" pageSize: 100 isZeroIndex: false request: responseType: json method: GET url: https://api.abnormalplatform.com/v1/auditlogs headers: - name: authorization value: Bearer ${secrets.abnormalToken} queryParams: - name: pageNumber value: "${pagination.pageNumber}" - name: pageSize value: "${pagination.pageSize}" - name: filter value: timestamp gte ${temporalWindow.from} lte ${temporalWindow.to} output: select: ".auditLogs" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Collection Phase** * **Pagination Type****\*** - `PageNumber/PageSize` * **Zero Index****\*** - `false` * **Page Size****\*** - `100` * **Request** * **Response type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/auditlogs` * **Headers** * **Name** - `Authorization` * **Value** - `Bearer ${secrets.abnormalToken}` * **Query Params** * **Name** - `pageNumber` * **Value** - `"${pagination.pageNumber}"` * **Name** - `pageSize` * **Value** - `"${pagination.pageSize}"` * **Name** - `filter` * **Value** - `timestamp gte ${temporalWindow.from} lte ${temporalWindow.to}` * **Output** * **Select** - `.auditLogs` * **Map** - `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Cases ## Overview Get a list of cases through the [Abnormal REST API](https://abnormalsecurity.my.site.com/knowledgebase/s/article/Abnormal-REST-API-Integration) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `abnormalToken` will reference your Abnormal bearer token. To do it, click **Add element** and enter a **Name** for the secret (in this case, `abnormalToken`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: true enumerationPhase: paginationType: "page" pageSize: 100 isZeroIndex: false request: responseType: json method: GET url: https://api.abnormalplatform.com/v1/cases headers: - name: authorization value: Bearer ${secrets.abnormalToken} queryParams: - name: pageNumber value: "${pagination.pageNumber}" - name: pageSize value: "${pagination.pageSize}" - name: filter value: lastModifiedTime gte ${temporalWindow.from} lte ${temporalWindow.to} output: select: "[.cases[].caseId]" map: "." outputMode: element collectionPhase: variables: - source: input name: caseId expression: "." format: "" paginationType: none request: method: GET url: https://api.abnormalplatform.com/v1/cases/${inputs.caseId} headers: - name: authorization value: Bearer ${secrets.abnormalToken} output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Enumeration Phase** Toggle **ON** to configure the enumeration phase. This API endpoint requires an initial request that will provide a list of alert IDs. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `page` * **Zero index****\*** - `false` * **Page Size****\*** - `100` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/abusecampaigns` * **Headers** * **Name** - `authorization` * **Value -** `Bearer ${secrets.abnormalToken}` * **Query Params** * **Name** - `pageNumber` * **Value** - `${pagination.pageNumber}` * **Name** - `pageSize` * **Value** - `${pagination.pageSize}` * **Name** - `Filter` * **Value** - `receivedTime gte ${temporalWindow.from} lte ${temporalWindow.to}` * **Output** * **Select****\*** - `[.cases[].caseId]` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Name** - `caseId` * **Expression** - `.` * **Format** - `""` * **Pagination Type****\*** - `None` * **Request** * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/cases/${inputs.caseId}` * **Headers** * **Name** - `authorization` * **Value** - `Bearer ${secrets.abnormalToken}` * **Output** * **Select** - `.` * **Map** - `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Threats ## Overview Get a list of threats through the [Abnormal REST API](https://app.swaggerhub.com/apis/abnormal-security/abx/1.4.3#/Threats/v1_threats_retrieve) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `abnormalToken` will reference your Abnormal bearer token. To do it, click **Add element** and enter a **Name** for the secret (in this case, `abnormalToken`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: true enumerationPhase: paginationType: "page" pageSize: 100 isZeroIndex: false request: responseType: json method: GET url: https://api.abnormalplatform.com/v1/threats headers: - name: authorization value: Bearer ${secrets.abnormalToken} queryParams: - name: pageNumber value: "${pagination.pageNumber}" - name: pageSize value: "${pagination.pageSize}" - name: filter value: receivedTime gte ${temporalWindow.from} lte ${temporalWindow.to} output: select: "[.threats[].threatId]" map: "." outputMode: element collectionPhase: variables: - source: input name: threatId expression: "." format: "" paginationType: none request: method: GET url: https://api.abnormalplatform.com/v1/threats/${inputs.threatId} headers: - name: authorization value: Bearer ${secrets.abnormalToken} output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Enumeration Phase** Toggle **ON** to configure the enumeration phase. This API endpoint requires an initial request that will provide a list of alert IDs. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `page` * **Zero index****\*** - `false` * **Page Size****\*** - `100` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/threats` * **Headers** * **Name** - `authorization` * **Value -** `Bearer ${secrets.abnormalToken}` * **Query Params** * **Name** - `pageNumber` * **Value** - `${pagination.pageNumber}` * **Name** - `pageSize` * **Value** - `${pagination.pageSize}` * **Name** - `Filter` * **Value** - `receivedTime gte ${temporalWindow.from} lte ${temporalWindow.to}` * **Output** * **Select****\*** - `[.threats[].threatId]` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Name** - `threatId` * **Expression** - `.` * **Format** - `""` * **Pagination Type****\*** - `None` * **Request** * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/cases/${inputs.threatId}` * **Headers** * **Name** - `authorization` * **Value** - `Bearer ${secrets.abnormalToken}` * **Output** * **Select** - `.` * **Map** - `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Vendor cases ## Overview Get a list of vendor cases through the [Abnormal REST API](https://app.swaggerhub.com/apis/abnormal-security/abx/1.4.3#/Vendors/v1_vendor_cases_retrieve) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `abnormalToken` will reference your Abnormal bearer token. To do it, click **Add element** and enter a **Name** for the secret (in this case, `abnormalToken`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: true enumerationPhase: paginationType: "page" pageSize: 100 isZeroIndex: false request: responseType: json method: GET url: https://api.abnormalplatform.com/v1/vendor-cases headers: - name: authorization value: Bearer ${secrets.abnormalToken} queryParams: - name: pageNumber value: "${pagination.pageNumber}" - name: pageSize value: "${pagination.pageSize}" - name: filter value: lastModifiedTime gte ${temporalWindow.from} lte ${temporalWindow.to} output: select: "[.vendorCases[].vendorCaseId]" map: "." outputMode: element collectionPhase: variables: - source: input name: vendorCaseId expression: "." format: "" paginationType: none request: method: GET url: https://api.abnormalplatform.com/v1/vendor-cases/${inputs.vendorCaseId} headers: - name: authorization value: Bearer ${secrets.abnormalToken} output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Enumeration Phase** Toggle **ON** to configure the enumeration phase. This API endpoint requires an initial request that will provide a list of alert IDs. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `page` * **Zero index****\*** - `false` * **Page Size****\*** - `100` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/vendor-cases` * **Headers** * **Name** - `authorization` * **Value -** `Bearer ${secrets.abnormalToken}` * **Query Params** * **Name** - `pageNumber` * **Value** - `${pagination.pageNumber}` * **Name** - `pageSize` * **Value** - `${pagination.pageSize}` * **Name** - `Filter` * **Value** - `receivedTime gte ${temporalWindow.from} lte ${temporalWindow.to}` * **Output** * **Select****\*** - `[.vendorCases[].vendorCaseId]` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Name** - `vendorCaseId` * **Expression** - `.` * **Format** - `""` * **Pagination Type****\*** - `None` * **Request** * **Method****\*** - `GET` * **URL****\*** - `https://api.abnormalplatform.com/v1/cases/${inputs.vendorCaseId}` * **Headers** * **Name** - `authorization` * **Value** - `Bearer ${secrets.abnormalToken}` * **Output** * **Select** - `.` * **Map** - `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Absolute ## Overview Use the **HTTP Pull** Listener to collect data from the [Absolute API](https://abnormalsecurity.my.site.com/knowledgebase/s/article/Abnormal-REST-API-Integration). Currently, we support the device reporting endpoint. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `secretKey` will reference your Absolute secret key. * `TokenId` will reference your Absolute token ID. To do it, click **Add element** and enter a **Name** for the secret (in this case, `secretKey`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `TokenId`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: hmac hmac: request: generateId: false idType: uuid generateTimestamp: true timestamp: tz: UTC format: RFC1123 hash: algorithm: hmac_sha256 encoding: base64 secretKey: ${secrets.secretKey} dataToSign: "${request.method}\n${request.relativeUrl}\n${request.timestamp}\n${request.body}\n" headers: - name: x-abs-date value: ${request.timestamp} - name: Authorization value: "Absolute token=${secrets.TokenId}:${hmac.hash}" withEnumerationPhase: false collectionPhase: paginationType: "responseBodyLink" responseBodyLinkSelector: ".metadata.pagination.nextPage" request: responseType: json method: "GET" url: "https://api.absolute.com/v3/reporting/devices" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" queryParams: - name: nextPage value: "${pagination.responseBodyLink}" - name: pageSize value: "500" - name: timeZone value: "UTC" - name: lastUpdatedDateTimeUtcFromInclusive value: ${temporalWindow.from} - name: lastUpdatedDateTimeUtcTo value: ${temporalWindow.to} output: select: ".data" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. * **Type****\*** - `HMAC` * **HMAC** * **Request** * **Generate Timestamp** - Toggle **ON** * **Timestamp** * **Format** - `RFC1123` * **Hash** * **Hashing algorithm****\*** - SHA256 * **Format****\*** - `base64` * **Secret key** - `${secrets.secretKey}` * **Data to sign****\*** - `${request.method}\n${request.relativeUrl}\n${request.timestamp}\n${request.body}\n${request.method}\n${request.relativeUrl}\n${request.timestamp}\n${request.body}\n` * **Headers** * **Name** - `x-abs-date` * **Value** - `${request.timestamp}` * **Name** - `authorization` * **Value** - `Absolute token=${secrets.TokenId}:${hmac.hash}` **Collection Phase** * **Pagination Type****\*** - `Next Link at Response Body` * **Selector****\*** - `.metadata.pagination.nextPage` * **Request** * **Response type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.absolute.com/v3/reporting/devices` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Query params** * **Name** - `nextPage` * **Value** - `${pagination.responseBodyLink}` * **Name** - `pageSize` * **Value** - `500` * **Name** - `TimeZone` * **Value** - `yourtimezone` * **Name** - `lastUpdatedDateTimeUtcFromInclusive` * **Value** - `${temporalWindow.from}` * **Name** - `lastUpdatedDateTimeUtcTo` * **Value** - `${temporalWindow.to}` * **Output** * **Select****\*** - `.data` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Agari DMARC Protection Use the **HTTP Pull** Listener to collect data from the [Agari API](https://developers.agari.com/agari-platform/reference/overview). We currently support the following Agari DMARC Protection endpoints: * [Alert events](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-agari-dmarc-protection/alert-events) * [Audits](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-agari-dmarc-protection/audits) * [Domains](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-agari-dmarc-protection/domains) * [Organizations](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-agari-dmarc-protection/organizations) * [Users](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-agari-dmarc-protection/users) # Alert Events ## Overview Get a list of Agari DMARC Protection alert events through the [Agari API](https://developers.agari.com/agari-platform/reference/alert_events_index) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Agari DMARC Protection domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Agari DMARC Protection API Application ID. * `client_secret` will reference your Agari DMARC Protection Application secret key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/v1/cp/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit isZeroIndex: false limit: 200 request: responseType: json method: GET url: ${parameters.domain}/v1/cp/alert_events queryParams: - name: start_date value: ${temporalWindow.from} - name: end_date value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".alert_events" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication Phase** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/v1/cp/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `UrlEncoded` * **Body params** * **Name** - `client_id` * **Value** - `'${secrets.client_id}'` * **Name** - `client_secret` * **Value** - `'${secrets.client_secret'` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/alert_events` * **Headers** * **Name** - `start_date` * **Value** - `${temporalWindow.from}` * **Name** - `end_date` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.alert_events` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Audits ## Overview Get a list of Agari DMARC Protection audits by domain, user or organization through the [Agari API](https://developers.agari.com/agari-platform/reference/alert_events_index) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Agari DMARC Protection domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Agari DMARC Protection API Application ID. * `client_secret` will reference your Agari DMARC Protection Application secret key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: * [Audits by domain](#audits-by-domain) * [Audits by organization](#audits-by-organization) * [Audits by user](#audits-by-user) #### Audits by domain {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/v1/cp/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: true enumerationPhase: paginationType: offsetLimit isZeroIndex: false limit: 200 request: responseType: json method: GET url: ${parameters.domain}/v1/cp/domains headers: - name: Accept value: application/json queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: "[.domains[].id]" map: "." outputMode: element collectionPhase: variables: - source: input name: domainId expression: "." format: "" paginationType: none request: responseType: json method: GET url: ${parameters.domain}/v1/cp/audits headers: - name: Accept value: application/json queryParams: - name: start_date value: ${temporalWindow.from} - name: end_date value: ${temporalWindow.to} - name: object_type value: domain - name: object_id value: ${inputs.domainId} output: select: ".audits.entries" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/v1/cp/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `UrlEncoded` * **Body Params** * **Name** - `client_id` * **Value** - `'${secrets.client_id}'` * **Name** - `client_secret` * **Value** - `'${secrets.client_secret'` * **Token path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Enumeration Phase** Toggle **ON** and enter the following: * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/domains` * **Headers** * **Name** - `accept` * **Value** - `application/json` * **Query Params** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `[.domains[].id]` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Inputs** * **Name** - `domainId` * **Expression** - `.` * **Format** - `""` * **Pagination Type****\*** - `none` * **Zero Index****\*** - `false` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/audits` * **Headers** * **Name** - `accept` * **Value** - `application/json` * **Query Params** * **Name** - `start_date` * **Value** - `${temporalWindow.from}` * **Name** - `end_date` * **Value** - `${temporalWindow.to}` * **Name** - `object_type` * **Value** - `domain` * **Name** - `object_id` * **Value** - `${inputs.domainId}` * **Output** * **Select****\*** - `.audits.entries` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} #### Audits by organization {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/v1/cp/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: true enumerationPhase: paginationType: offsetLimit limit: 200 request: responseType: json method: GET url: ${parameters.domain}/v1/cp/organizations headers: - name: Accept value: application/json queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: "[.organizations[].id]" map: "." outputMode: element collectionPhase: variables: - source: input name: orgId expression: "." format: "" paginationType: none request: responseType: json method: GET url: ${parameters.domain}/v1/cp/audits headers: - name: Accept value: application/json queryParams: - name: start_date value: ${temporalWindow.from} - name: end_date value: ${temporalWindow.to} - name: object_type value: organization - name: object_id value: ${inputs.orgId} output: select: ".audits.entries" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/v1/cp/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `UrlEncoded` * **Body Params** * **Name** - `client_id` * **Value** - `'${secrets.client_id}'` * **Name** - `client_secret` * **Value** - `'${secrets.client_secret'` * **Token path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Enumeration Phase** Toggle **ON** and enter the following: * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/organizations` * **Headers** * **Name** - `accept` * **Value** - `application/json` * **Query Params** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `[.organizations[].id]` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Inputs** * **Name** - `orgId` * **Expression** - `.` * **Format** - `""` * **Pagination Type****\*** - `none` * **Zero Index****\*** - `false` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/audits` * **Headers** * **Name** - `accept` * **Value** - `application/json` * **Query Params** * **Name** - `start_date` * **Value** - `${temporalWindow.from}` * **Name** - `end_date` * **Value** - `${temporalWindow.to}` * **Name** - `object_type` * **Value** - `organization` * **Name** - `object_id` * **Value** - `${inputs.orgId}` * **Output** * **Select****\*** - `.audits.entries` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} #### Audits by user {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/v1/cp/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: true enumerationPhase: paginationType: offsetLimit limit: 200 request: responseType: json method: GET url: ${parameters.domain}/v1/cp/users headers: - name: Accept value: application/json queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: "[.users[].id]" map: "." outputMode: element collectionPhase: variables: - source: input name: userId expression: "." format: "" paginationType: none request: responseType: json method: GET url: ${parameters.domain}/v1/cp/audits headers: - name: Accept value: application/json queryParams: - name: start_date value: ${temporalWindow.from} - name: end_date value: ${temporalWindow.to} - name: object_type value: user - name: object_id value: ${inputs.userId} output: select: ".audits.entries" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/v1/cp/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `UrlEncoded` * **Body Params** * **Name** - `client_id` * **Value** - `'${secrets.client_id}'` * **Name** - `client_secret` * **Value** - `'${secrets.client_secret'` * **Token path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Enumeration Phase** Toggle **ON** and enter the following: * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/users` * **Headers** * **Name** - `accept` * **Value** - `application/json` * **Query Params** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `[.users[].id]` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Inputs** * **Name** - `userId` * **Expression** - `.` * **Format** - `""` * **Pagination Type****\*** - `none` * **Zero Index****\*** - `false` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/audits` * **Headers** * **Name** - `accept` * **Value** - `application/json` * **Query Params** * **Name** - `start_date` * **Value** - `${temporalWindow.from}` * **Name** - `end_date` * **Value** - `${temporalWindow.to}` * **Name** - `object_type` * **Value** - `user` * **Name** - `object_id` * **Value** - `${inputs.userId}` * **Output** * **Select****\*** - `.audits.entries` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Domains ## Overview Get a list of Agari DMARC Protection domains through the [Agari API](https://developers.agari.com/agari-platform/reference/alert_events_index) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Agari DMARC Protection domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Agari DMARC Protection API Application ID. * `client_secret` will reference your Agari DMARC Protection Application secret key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/v1/cp/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit isZeroIndex: false limit: 200 request: responseType: json method: GET url: ${parameters.domain}/v1/cp/domains queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication Phase** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/v1/cp/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `UrlEncoded` * **Body params** * **Name** - `client_id` * **Value** - `'${secrets.client_id}'` * **Name** - `client_secret` * **Value** - `'${secrets.client_secret'` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/domains` * **Headers** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Organizations ## Overview Get a list of Agari DMARC Protection organizations through the [Agari API](https://developers.agari.com/agari-platform/reference/alert_events_index) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Agari DMARC Protection domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Agari DMARC Protection API Application ID. * `client_secret` will reference your Agari DMARC Protection Application secret key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/v1/cp/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit isZeroIndex: false limit: 200 request: responseType: json method: GET url: ${parameters.domain}/v1/cp/organizations queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication Phase** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/v1/cp/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `UrlEncoded` * **Body params** * **Name** - `client_id` * **Value** - `'${secrets.client_id}'` * **Name** - `client_secret` * **Value** - `'${secrets.client_secret'` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/organizations` * **Headers** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Users ## Overview Get a list of Agari DMARC Protection users through the [Agari API](https://developers.agari.com/agari-platform/reference/alert_events_index) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Agari DMARC Protection domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Agari DMARC Protection API Application ID. * `client_secret` will reference your Agari DMARC Protection Application secret key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/v1/cp/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit isZeroIndex: false limit: 200 request: responseType: json method: GET url: ${parameters.domain}/v1/cp/users queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication Phase** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/v1/cp/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `UrlEncoded` * **Body params** * **Name** - `client_id` * **Value** - `'${secrets.client_id}'` * **Name** - `client_secret` * **Value** - `'${secrets.client_secret'` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/v1/cp/users` * **Headers** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Akamai Use the **HTTP Pull** Listener to collect data from the following Akamai products: * [Akamai Guardicore](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore) * [Akamai SIEM Integration](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-siem-integration)
# Collect data from Akamai Guardicore Use the **HTTP Pull** Listener to collect data from the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login). We currently support the following Akamai Guardicore endpoints: * [Agents](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/agents) * [Agent aggregators](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/agent-aggregators) * [Collectors](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/collectors) * [Connections](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/connections) * [Dashboard data](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/dashboard-data) * [Honeypots](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/honeypots) * [Incidents](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/incidents) * [Reputation logs](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/reputation-logs) # Agents ## Overview Get a list of Akamai Guardicore agents through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/agents queryParams: - name: from_time value: ${temporalWindow.from} - name: to_time value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/agents` * **Query Params** * **Name** - `from_time` * **Value** - `${temporalWindow.from}` * **Name** - `to_time` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Agent aggregators ## Overview Get a list of Akamai Guardicore agent aggregators through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/agent_aggregators queryParams: - name: from_time value: ${temporalWindow.from} - name: to_time value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/agent_aggregators` * **Query Params** * **Name** - `from_time` * **Value** - `${temporalWindow.from}` * **Name** - `to_time` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collectors ## Overview Get a list of Akamai Guardicore collectors through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/collectors queryParams: - name: from_time value: ${temporalWindow.from} - name: to_time value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/collectors` * **Query Params** * **Name** - `from_time` * **Value** - `${temporalWindow.from}` * **Name** - `to_time` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Connections ## Overview Get a list of Akamai Guardicore connections through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/connections queryParams: - name: from_time value: ${temporalWindow.from} - name: to_time value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/connections` * **Query Params** * **Name** - `from_time` * **Value** - `${temporalWindow.from}` * **Name** - `to_time` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Dashboard data ## Overview Get a list of Akamai Guardicore dashboard events through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/dashboard/security-dashboard/ queryParams: - name: time_frame value: HOUR - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/security-dashboard` * **Query Params** * **Name** - `time_frame` * **Value** - `HOUR` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Honeypots ## Overview Get a list of Akamai Guardicore honeypots through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/honeypots queryParams: - name: from_time value: ${temporalWindow.from} - name: to_time value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/honeypots` * **Query Params** * **Name** - `from_time` * **Value** - `${temporalWindow.from}` * **Name** - `to_time` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Incidents ## Overview Get a list of Akamai Guardicore incidents through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/incidents queryParams: - name: from_time value: ${temporalWindow.from} - name: to_time value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/incidents` * **Query Params** * **Name** - `from_time` * **Value** - `${temporalWindow.from}` * **Name** - `to_time` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Reputation logs ## Overview Get a list of Akamai Guardicore reputation logs through the [Akamai Guardicore Centra Web API](https://control.akamai.com/apps/auth/?SAMLRequest=fZHLTsMwEEXX%2FEXkfR41JKGjJFJRF1QCEdGIBRs0cRxqkdjB40j9fPIAqWzq3dhn5vreyQj7boDd6E76VX6Pkpx37jtNsDzkbLQaDJIi0NhLAifguHt%2BAh5EMFjjjDAd86Zzs596lUanjM7ZybmBIAyF0c6aLsAv7FEFwvQhDgOFOAmGS99hn7OP%2Bi5t6zSJk20by6bl902dirRt%2BG0sYmz4ShKN8qDJoXY54xFP%2FCjxN7zaRMC3wKP3BSt%2Ff%2FWgdKP053UL9QoRPFZV6Zcvx4p5b9LSYmICWJHNQcCibS%2BiuT4WiaSdk2DFjM1up8slA99KbHr5V8mzk1Zjl4UXOsVa%2Fd9L8QM%3D\&RelayState=Vvxm66SIRk4gzRaMRJjvcmb7tqMGF49O\&SigAlg=http%3A%2F%2Fwww.w3.org%2F2001%2F04%2Fxmldsig-more%23rsa-sha256\&Signature=NYTfLspvti1qHBeOmP7Tkz1ANStWYByfTa%2F3EWv6jm%2Bt2MMpHOi%2By95%2FWQruDLENPhmL%2FwYUVUpcks4JwUpwmDbnWGWtOGh1If9sABjyZOVo7DgCVrxJjhsF%2B65DpRtD8K5edROoL%2BnTFRW0mc%2BAmFBLiiFfekF7fVMuWvvms1seozL9BUWbyqPMpRHJSx%2FQqG4YeBRsXj12R6issVnNTGElxyhhx%2B%2FF3531jURsTHq3zNYn6q9n1J5NHNSjJ7aCNsCyPiU116ekUaNpKTca4bpK%2Bc%2F9sy7NRnN4GPe0BV0b4mzpcUerjKdmYc9w3lU%2FTJeSGnYp3HaNe9r6Pik6ow%3D%3D#/login) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai Guardicore domain name. ### Secrets You must define these credentials in Onum: * `username` will reference your Akamai Guardicore username. * `password` will reference your Akamai Guardicore password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/api/v3.0/authenticate headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.username}", "password": "${secrets.password}" } tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 1000 isZeroIndex: true request: responseType: json method: GET url: https://${parameters.domain}/api/v3.0/reputation-log queryParams: - name: from_time value: ${temporalWindow.from} - name: to_time value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".objects" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/api/v3.0/authenticate` * **Headers** * **Name** - `Content-type` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** ``` { "username": "${secrets.username}", "password": "${secrets.password}" } ``` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `true` * **Limit****\*** - `1000` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/api/v3.0/reputation-log` * **Query Params** * **Name** - `from_time` * **Value** - `${temporalWindow.from}` * **Name** - `to_time` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.objects` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Akamai SIEM Integration ## Overview Get a list of Akamai SIEM Integration security events through the [SIEM Integration API](https://techdocs.akamai.com/siem-integration/reference/api) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameters: * **Name** - `domain` * **Integration** - Enter your Akamai SIEM Integration domain name. * **Name** - `configId` * **Value** - Enter your Akamai SIEM Integration configuration ID. ### Secrets You must define these credentials in Onum: * `clientSecret` will reference your Akamai SIEM Integration Client Secret. * `accessToken` will reference your Akamai SIEM Integration Access Token. * `clientToken` will reference your Akamai SIEM Integration Client Token. To do it, click **Add element** and enter a **Name** for the secret (in this case, `clientSecret`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `accessToken` and the `clientToken`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 30m offset: 30m tz: UTC format: Epoch withAuthentication: true authentication: type: akamai akamai: clientSecret: ${secrets.clientSecret} accessToken: ${secrets.accessToken} clientToken: ${secrets.clientToken} withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: .offset initialRequest: responseType: ndjson method: "GET" url: "https://${parameters.domain}/siem/v1/configs/${parameters.configId}" queryParams: - name: from value: ${temporalWindow.from} - name: to value: ${temporalWindow.to} - name: limit value: "1000" nextRequest: responseType: ndjson method: "GET" url: "https://${parameters.domain}/siem/v1/configs/${parameters.configId}" queryParams: - name: offset value: ${pagination.cursor} - name: limit value: "1000" output: select: "." filter: ".offset == null" outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `30m` * **Offset****\*** - `30m` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Akamai EdgeGrid` * **Akamai EdgeGrid Authentication** * **Client Token****\*** - Enter your Client Token. * **Access Token****\*** - Enter your Access Token. * **Client Secret****\*** - Enter your Client Secret. **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.offset` * **Initial Request** * **Response Type****\*** - `NDJSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/siem/v1/configs/${parameters.configId}` * **Query Params** * **Name** - `from` * **Value** - `${temporalWindow.from}` * **Name** - `to` * **Value** - `${temporalWindow.to}` * **Name** - `limit` * **Value** - `1000` * **Next Request** * **Response Type****\*** - `NDJSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/siem/v1/configs/${parameters.configId}` * **Query Params** * **Name** - `offset` * **Value** - `${pagination.cursor}` * **Name** - `limit` * **Value** - `1000` * **Output** * **Select****\*** - `.` * **Filter** - `.offset == null` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Arista CloudVision ## Overview Get a list of Arista CloudVision audit events through the [CloudVision REST API](https://www.arista.io/help/articles/settings-audit-logs-rest-api) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `cvp` * **Value** - Enter your CloudVision Portal domain name. ### Secrets You must define these credentials in Onum: * `token` will reference your CloudVision token. To do it, click **Add element** and enter a **Name** for the secret (in this case, `token`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: .data[-1] initialRequest: responseType: json method: "POST" url: "https://${parameters.cvp}/cvpservice/audit/getLogs.do" headers: - name: Authorization value: Bearer ${secrets.token} - name: Content-Type value: application/json - name: Accept value: application/json bodyType: raw bodyRaw: | { "category": "USER", "startTime": ${temporalWindow.from}, "endTime": ${temporalWindow.to}, "dataSize": 1000, "objetKey": "admin" } nextRequest: responseType: json method: "POST" url: "https://${parameters.cvp}/cvpservice/audit/getLogs.do" headers: - name: Authorization value: Bearer ${secrets.token} - name: Content-Type value: application/json - name: Accept value: application/json bodyType: raw bodyRaw: | { "category": "USER", "startTime": ${temporalWindow.from}, "endTime": ${temporalWindow.to}, "dataSize": 1000, "objetKey": "admin", "lastRetrievedAudit": ${pagination.cursor} } output: select: ".data" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `5m` * **Offset** - `5m` * **Format** - `Epoch` **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.data[-1]` * **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.cvp}/cvpservice/audit/getLogs.do` * **Headers** * **Name** - `Authorization` * **Value** - `Bearer ${secrets.token}` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** - ``` { "category": "USER", "startTime": ${temporalWindow.from}, "endTime": ${temporalWindow.to}, "dataSize": 1000, "objetKey": "admin" } ``` * **Next Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.cvp}/cvpservice/audit/getLogs.do` * **Headers** * **Name** - `Authorization` * **Value** - `Bearer ${secrets.token}` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\*** - `Raw` * **Body Content****\*** - ``` { "category": "USER", "startTime": ${temporalWindow.from}, "endTime": ${temporalWindow.to}, "dataSize": 1000, "objetKey": "admin", "lastRetrievedAudit": ${pagination.cursor} } ``` * **Output** * **Select****\*** - `.data` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Armis Centrix ## Overview Get a list of Armis Centrix audit events through the [Armis Centrix API](https://docs.query.ai/docs/armis-centrix) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Armis instance name. ### Secrets You must define these credentials in Onum: * `armis_key` will reference your Armis API key. To do it, click **Add element** and enter a **Name** for the secret (in this case, `armis_key`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: "2006-01-02T15:04:05" withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/api/v1/access_token/ headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: secret_key value: ${secrets.armis_key} tokenPath: ".data.access_token" authInjection: in: header name: Authorization prefix: '' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit isZeroIndex: false limit: 200 request: responseType: json method: GET url: ${parameters.domain}/api/v1/search/ queryParams: - name: aql value: in:alerts after:${temporalWindow.from} before:${temporalWindow.to} - name: from value: ${pagination.offset} - name: length value: ${pagination.limit} output: select: ".data.results" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `5m` * **Offset** - `5m` * **Format** - `RFC3339` **Authentication** Toggle **ON** and configure these parameters: * **Type****\*** - `Token` **Token Retrieve Based Authentication** * **Request** * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/api/v1/access_token/` * **Headers** * **Name** - `Content-Type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\***** -** `URLEncoded` * **Body Params** * **Name** - `secret_key` * **Value** - `${secrets.armis_key}` * **Token path****\*** - `.data.access_token` * **Auth Injection** * **In****\*** - `Header` * **Name****\*** - `Authorization` * **Prefix** - `''` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `200` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/api/v1/search/` * **Query Params** * **Name** - `aql` * **Value** - `in:alerts after:${temporalWindow.from} before:${temporalWindow.to}` * **Name** - `from` * **Value** - `${pagination.offset}` * **Name** - `length` * **Value** - `${pagination.limit}` **Output** * **Select****\*** - `.data.results` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Atlassian Jira ## Overview Get a list of Jira issues using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `username` will reference your Jira username. * `password` will reference your Jira password. To do it, click **Add element** and enter a **Name** for the secret (in this case, `username`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `password`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle **ON** the **Config as YAML** option to enable a free text field where you can paste the following YAML: ```yaml withTemporalWindow: true temporalWindow: duration: 7m offset: 1m tz: UTC format: RFC3339 withAuthentication: true authentication: type: basic basic: username: ${secrets.username} password: ${secrets.password} withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".nextPageToken" initialRequest: method: GET responseType: json url: https://aflacinc.atlassian.net/rest/api/2/search/jql queryParams: - name: jql value: "updated>=-7m order by updated asc" - name: fields value: "updated,parent,project,resolved,resolutionDate,issuetype,summary,description,creator,assignee,reporter,created,duedate,aggregatetimespent,priority,labels,comment,status,resolution,attachment,issuekey" - name: maxResults value: 5000 headers: - name: Accept value: application/json nextRequest: method: GET responseType: json url: https://aflacinc.atlassian.net/rest/api/2/search/jql queryParams: - name: jql value: "updated>=-7m order by updated asc" - name: fields value: "updated,parent,project,resolved,resolutionDate,issuetype,summary,description,creator,assignee,reporter,created,duedate,aggregatetimespent,priority,labels,comment,status,resolution,attachment,issuekey" - name: maxResults value: 5000 - name: nextPageToken value: ${pagination.cursor} headers: - name: Accept value: application/json output: select: ".issues" outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `7m` * **Offset** - `1m` * **Format** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Basic` * **Username****\*** - Enter your Jira username. * **Password****\*** - Enter your Jira password. **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.nextPageToken` * **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://aflacinc.atlassian.net/rest/api/2/search/jql` * **Query Params** * **Name** - `jql` * **Value** - `updated>=-7m order by updated asc` * **Name** - `fields` * **Value** - `updated,parent,project,resolved,resolutionDate,issuetype,summary,description,creator,assignee,reporter,created,duedate,aggregatetimespent,priority,labels,comment,status,resolution,attachment,issuekey` * **Name** - `maxResults` * **Value** - `5000` * **Headers** - * **Name** - `Accept` * **Value** - `application/json` * **Next Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://aflacinc.atlassian.net/rest/api/2/search/jql` * **Query Params** * **Name** - `jql` * **Value** - `updated>=-7m order by updated asc` * **Name** - `fields` * **Value** - `updated,parent,project,resolved,resolutionDate,issuetype,summary,description,creator,assignee,reporter,created,duedate,aggregatetimespent,priority,labels,comment,status,resolution,attachment,issuekey` * **Name** - `maxResults` * **Value** - `5000` * **Headers** - * **Name** - `Accept` * **Value** - `application/json` * **Output** * **Select****\*** - `.issues` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Box ## Overview Use the **HTTP Pull** Listener to collect data from the [Box API](https://developer.box.com/reference/get-events). Currently, we support the events endpoint. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters N/A ### Secrets You must define these credentials in Onum: * `client_id` will reference your Box Client ID. * `client_secret` will reference your Box Client Secret. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Box API** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 30s offset: 30s tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://api.box.com/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded bodyType: urlEncoded bodyParams: - name: grant_type value: client_credentials - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: box_subject_id value: 62685208 - name: box_subject_type value: enterprise tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".next_stream_position" initialRequest: responseType: json method: GET url: "https://api.box.com/2.0/events" headers: - name: Accept value: application/json queryParams: - name: stream_type value: admin_logs - name: created_after value: ${temporalWindow.from} - name: created_before value: ${temporalWindow.to} - name: limit value: '100' # REMOVED: event_type parameter - either remove entirely or use valid Box event types # Valid types: ACCESS_GRANTED, ADMIN_LOGIN, FILE_MARKED_MALICIOUS, etc. nextRequest: responseType: json method: GET url: "https://api.box.com/2.0/events" headers: - name: Accept value: application/json queryParams: - name: stream_type value: admin_logs - name: stream_position value: '${pagination.cursor}' - name: created_after value: ${temporalWindow.from} - name: created_before value: ${temporalWindow.to} - name: limit value: '100' output: select: ".entries" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `30s` * **Offset****\*** - `30s` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. * **Type****\***** -** `token` * **Token Retrieve Baes Authentication** * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://api.box.com/oauth2/token` * **Headers** * **Name** - `Content-Type` * **Value** - `application/x-www-form-urlencodedapplication/x-www-form-urlencoded` * **Body Type****\*** - `urlEncoded` * **Body Params** * **Name** - `grant_type` * **Value** - `client_credentials` * **Name** - `Client ID` * **Value** - `${secrets.client_id}` * **Name** - `Client Secret` * **Value** - `${secrets.client_secret}` * **Name** - `box_subject_id` * **Value** - `62685208` * **Name** - `box_subject_type` * **Value** - `enterprise` * **Token path** - `.access_token` * **Auth injection** * **In****\*** - `header` * **Name****\*** - `Authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** Toggle **ON** to configure the collection phase. This API endpoint requires an initial request that will provide a list of alert ids. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.next_stream_position` * **Initial request** * **Response Type**\* - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.box.com/2.0/events` * **Headers** * **Name** - `Accept` * **Value** - `Application/JSON` * **Query Params** * **Name** - `stream_type` * **Value** - `admin_logs` * **Name** - `created_after` * **Value** - `${temporalWindow.from}` * **Name** - `created_before` * **Value** - `${temporalWindow.to}` * **Name** - `limit` * **Value** - `100` * **Next request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.box.com/2.0/events` * **Headers** * **Name** - `Accept` * **Value** - `Application/JSON` * **Query Params** * **Name** - `stream_type` * **Value** - `admin_logs` * **Name** - `stream_position` * **Value** - `${pagination.cursor}` * **Name** - `created_after` * **Value** - `${temporalWindow.from}` * **Name** - `created_before` * **Value** - `${temporalWindow.to}` * **Name** - `limit` * **Value** - `100` * **Output** * **Select****\*** - `.entries` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Cisco Umbrella ## Overview Use the **HTTP Pull** Listener to collect data from the [Cisco Umbrella API](https://developer.cisco.com/docs/cloud-security/umbrella-api-api-reference-reports-reporting-api-activity-get-activities-all/). Currently, we support the reports endpoint. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Cisco Umbrella domain name. ### Secrets You must define these credentials in Onum: * `cisco_auth` will reference your Umbrella Token Authorization API. To do it, click **Add element** and enter a **Name** for the secret (in this case, `cisco_auth`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: EpochMillis withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/auth/v2/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json - name: authorization value: ${secrets.cisco_auth} bodyType: urlEncoded bodyParams: - name: grant_type value: client_credentials tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: offsetLimit limit: 100 isZeroIndex: false request: responseType: json method: GET url: https://${parameters.domain}/reports/v2/activity queryParams: - name: from value: ${temporalWindow.from} - name: to value: ${temporalWindow.to} - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} output: select: ".data" map: "." outputMode: element retry: statusCodes: [429, 500, 502, 503, 504] type: fixed fixed: interval: 2s ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset\*** - `5m` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://${parameters.domain}/auth/v2/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `${secrets.cisco_auth}` * **Body Type****\*** - `UrlEncoded` * **Body Params** * **Name** - `grant_type` * **Value** - `client_credentials` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Retry** Toggle **ON** and configure the following parameters: * **Retry type****\*** - `Fixed` * **Interval****\*** - `2s` **Collection Phase** * **Pagination Type****\*** - `Offset/Limit` * **Zero Index****\*** - `false` * **Limit****\*** - `100` * **Request** * **Response Type** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.domain}/reports/v2/activity` * **Query Params** * **Name** - `from` * **Value** - `${temporalWindow.from}` * **Name** - `to` * **Value** - `${temporalWindow.to}` * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Output** * **Select****\*** - `.data` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from CrowdStrike Falcon NG-SIEM Use the **HTTP Pull** Listener to collect data from the [CrowdStrike Falcon API](https://developer.crowdstrike.com/api-reference/overview/). We currently support the following types of data: * ​[Alerts](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-crowdstrike-falcon-ng-siem/alerts)​ * [​Event streams​](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-crowdstrike-falcon-ng-siem/event-streams) * [​Incidents​](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-akamai/collect-data-from-akamai-guardicore/incidents)
# Alerts ## Overview Get a list of Falcon NG-SIEM alert events through the [Falcon API](https://developer.crowdstrike.com/api-reference/collections/alerts/#getqueriesalertsv2) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Falcon NG-SIEM domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Falcon NG-SIEM client ID. * `client_secret` will reference your Falcon NG-SIEM client secret. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **CrowdStrike Falcon API** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded bodyType: urlEncoded bodyParams: - name: grant_type value: client_credentials - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: true enumerationPhase: paginationType: offsetLimit limit: 100 request: responseType: json method: GET url: ${parameters.domain}/alerts/queries/alerts/v2 queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} - name: filter value: created_timestamp:>'${temporalWindow.from}'+created_timestamp:<'${temporalWindow.to}' output: select: ".resources" map: "." outputMode: collection collectionPhase: variables: - source: input name: resources expression: "." format: "json" paginationType: none request: method: POST url: ${parameters.domain}/alerts/entities/alerts/v2 headers: - name: Accept value: application/json - name: Content-Type value: application/json responseType: json bodyType: raw bodyRaw: | { "composite_ids": ${inputs.resources} } output: select: ".resources" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Request** * **Request Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/oauth2/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Body Type****\*** - `UrlEncoded` * **Body params** * **Name** - `grant_type` * **Value** - `client_credentials` * **Name** - `client_id` * **Value** - `${secrets.client_id}` * **Name** - `client_secret` * **Value** - `${secrets.client_secret}` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Enumeration Phase** Toggle **ON** to configure the enumeration phase. This API endpoint requires an initial request that will provide a list of alert ids. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `Offset/Limit` * **Zero index****\*** - `false` * **Limit****\*** - `100` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/alerts/queries/alerts/v2` * **Query Params** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Name** - `filter` * **Value** - `created_timestamp:>'${temporalWindow.from}'+created_timestamp:<'${temporalWindow.to}'` * **Output** * **Select****\*** - `.resources` * **Map** - `.` * **Output Mode****\*** - `collection` **Collection Phase** * **Pagination Type****\*** - `None` * **Request** * **Response Type****\*** - JSON * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/alerts/entities/alerts/v2` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Body type****\*** - `Raw` * **Body Content****\*** ``` { "composite_ids": ${inputs.resources} } ``` * **Output** * **Select****\*** - `.resources` * **Map -** `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Event streams ## Overview Get a list of Falcon NG-SIEM event streams through the [Falcon API](https://developer.crowdstrike.com/api-reference/collections/event-streams/#listavailablestreamsoauth2) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Falcon NG-SIEM domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Falcon NG-SIEM client ID. * `client_secret` will reference your Falcon NG-SIEM client secret. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **CrowdStrike Falcon API** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 30m offset: 0 tz: UTC format: Epoch withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded bodyType: urlEncoded bodyParams: - name: grant_type value: client_credentials - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: true enumerationPhase: paginationType: none request: responseType: json method: GET url: ${parameters.domain}/sensors/entities/datafeed/v2 queryParams: - name: appId value: my-datafeed-onum-001 output: select: ".resources[0]" map: "{dataFeedURL, sessionToken: .sessionToken.token}" outputMode: element collectionPhase: variables: - source: input name: dataFeedURL expression: ".dataFeedURL" format: '' - source: input name: sessionToken expression: ".sessionToken" format: '' paginationType: none request: method: GET url: "${inputs.dataFeedURL}" headers: - name: Accept value: application/json - name: Authorization value: "Token ${inputs.sessionToken}" queryParams: - name: appId value: my-datafeed-onum-001 - name: whence value: 2 responseType: ndjson output: select: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `30m` * **Offset****\*** - `0` * **Format****\*** - `Epoch` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Request** * **Request Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/oauth2/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Body Type****\*** - `UrlEncoded` * **Body params** * **Name** - `grant_type` * **Value** - `client_credentials` * **Name** - `client_id` * **Value** - `${secrets.client_id}` * **Name** - `client_secret` * **Value** - `${secrets.client_secret}` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Enumeration Phase** Toggle **ON** to configure the enumeration phase. This API endpoint requires an initial request that will provide a list of alert ids. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `None` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/sensors/entities/datafeed/v2` * **Query Params** * **Name** - `appId` * **Value** - `my-datafeed-onum-001` * **Output** * **Select****\*** - `.resources[0]` * **Map** - `{dataFeedURL, sessionToken: .sessionToken.token}` * **Output Mode****\*** - `element` **Collection Phase** * **Inputs** * **Name** - `dataFeedURL` * **Expression** - `.dataFeedURL` * **Format** - `''` * **Name** - `sessionToken` * **Expression** - `.sessionToken` * **Format** - `''` * **Pagination Type****\*** - `None` * **Request** * **Response Type****\*** - `NDJSON` * **Method****\*** - `GET` * **URL****\*** - `${inputs.dataFeedURL}` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Token ${inputs.sessionToken}` * **Query Params** * **Name** - `appId` * **Value** - `my-datafeed-onum-001` * **Name** - `whence` * **Value** - `2` * **Output** * **Select****\*** - `.` * **Map -** `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Incidents ## Overview Get a list of Falcon NG-SIEM incidents through the [Falcon API](https://developer.crowdstrike.com/api-reference/collections/alerts/#getqueriesalertsv2) using the **HTTP Pull** Listener. ## HTTP Pull Listener configuration In Falcon Onum, go to the **Listeners** area and click **New Listener > HTTP Pull**. Give a name to your new Listener and enter the following data: ### Parameters Add the following parameter: * **Name** - `domain` * **Value** - Enter your Falcon NG-SIEM domain name. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Falcon NG-SIEM client ID. * `client_secret` will reference your Falcon NG-SIEM client secret. To do it, click **Add element** and enter a **Name** for the secret (in this case, `client_id`). Then, click the **Value** field and select **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the **Value** field list. Repeat the process for the `client_secret`. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Setup After entering the required parameters and secrets, you can choose to manually enter the rest of configuration fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **CrowdStrike Falcon API** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: ${parameters.domain}/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded bodyType: urlEncoded bodyParams: - name: grant_type value: client_credentials - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: true enumerationPhase: paginationType: offsetLimit limit: 100 request: responseType: json method: GET url: ${parameters.domain}/incidents/queries/incidents/v1 queryParams: - name: offset value: ${pagination.offset} - name: limit value: ${pagination.limit} - name: filter value: start:>='${temporalWindow.from}'+end:<'${temporalWindow.to}' output: select: ".resources" map: "." outputMode: collection collectionPhase: variables: - source: input name: resources expression: "." format: "json" paginationType: none request: method: POST url: ${parameters.domain}/incidents/entities/incidents/GET/v1 headers: - name: Accept value: application/json - name: Content-Type value: application/json responseType: json bodyType: raw bodyRaw: | { "ids": ${inputs.resources} } output: select: ".resources" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration****\*** - `5m` * **Offset****\*** - `5m` * **Format****\*** - `RFC3339` **Authentication** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Request** * **Request Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/oauth2/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Body Type****\*** - `UrlEncoded` * **Body params** * **Name** - `grant_type` * **Value** - `client_credentials` * **Name** - `client_id` * **Value** - `${secrets.client_id}` * **Name** - `client_secret` * **Value** - `${secrets.client_secret}` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Enumeration Phase** Toggle **ON** to configure the enumeration phase. This API endpoint requires an initial request that will provide a list of alert ids. In order to get the details about that information, it will require an additional request for those details. * **Pagination Type****\*** - `Offset/Limit` * **Zero index****\*** - `false` * **Limit****\*** - `100` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `${parameters.domain}/incidents/queries/incidents/v1` * **Query Params** * **Name** - `offset` * **Value** - `${pagination.offset}` * **Name** - `limit` * **Value** - `${pagination.limit}` * **Name** - `filter` * **Value** - `start:>='${temporalWindow.from}'+end:<'${temporalWindow.to}'` * **Output** * **Select****\*** - `.resources` * **Map** - `.` * **Output Mode****\*** - `collection` **Collection Phase** * **Inputs** * **Name** - `resources` * **Expression** - `.` * **Format** - `json` * **Pagination Type****\*** - `None` * **Request** * **Response type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `${parameters.domain}/incidents/entities/incidents/GET/v1` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Body type****\*** - `Raw` * **Body Content****\*** ``` { "ids": ${inputs.resources} } ``` * **Output** * **Select****\*** - `.resources` * **Map -** `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} When you're done, click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from CyberArk ## Overview Get a list of logs from [CyberArk](https://www.cyberark.com/). ## Configuration ### Parameters N/A ### Secrets You must define these credentials in Onum: * `auth_token` will reference your CyberArk authentication token. * `apikey` will reference your CyberArk API Key. To do it, enter a **Name** for the secret and then click the **Value** field. Click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the corresponding field. Click **Add element** to add the other one. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} After entering the required parameters and secrets, you can choose to manually enter the API fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://abd4962.id.cyberark.cloud/OAuth2/Token/OnumCrowdStrike headers: - name: Accept value: application/json - name: Authorization value: Basic ${secrets.auth_token} - name: Content-Type value: application/x-www-form-urlencoded queryParams: [] bodyType: urlEncoded bodyParams: - name: grant_type value: client_credentials - name: scope value: isp.audit.events:read tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: true enumerationPhase: paginationType: none request: responseType: json method: POST url: https://cmpc.audit.cyberark.cloud/api/audits/stream/createQuery headers: - name: Accept value: application/json - name: Accept-Encoding value: gzip, deflate - name: Content-Type value: application/json - name: x-api-key value: ${secrets.apikey} bodyType: raw bodyRaw: | { "query": { "pageSize": 500, "selectedFields": [ "tenant_id", "custom_data", "arrival_timestamp", "checksum", "application_code", "audit_code", "timestamp", "user_id", "session_id", "source", "action_type", "audit_type", "component", "target", "command", "message", "username", "action", "uuid", "service_name", "cloud_roles", "cloud_workspaces", "cloud_workspaces_and_roles", "cloud_assets", "cloud_identities", "vaulted_accounts", "cloud_provider", "account_name", "target_platform", "safe", "target_account", "identity_type", "access_method", "account_id", "correlation_id" ], "filterModel": { "date": { "dateFrom": "${temporalWindow.from}", "dateTo": "${temporalWindow.to}" } } } } output: select: "." map: "." outputMode: element collectionPhase: variables: - source: input name: cursorRef expression: ".cursorRef" format: '' paginationType: cursor cursorSelector: ".paging.cursor.cursorRef" initialRequest: method: POST url: https://cmpc.audit.cyberark.cloud/api/audits/stream/results headers: - name: Accept value: application/json - name: Accept-Encoding value: gzip, deflate - name: Content-Type value: application/json - name: x-api-key value: ${secrets.apikey} bodyType: raw bodyRaw: | { "cursorRef": "${inputs.cursorRef}" } nextRequest: method: POST url: https://cmpc.audit.cyberark.cloud/api/audits/stream/results headers: - name: Accept value: application/json - name: Accept-Encoding value: gzip, deflate - name: Content-Type value: application/json - name: x-api-key value: ${secrets.apikey} bodyType: raw bodyRaw: | { "cursorRef": "${pagination.cursor}" } output: select: ".data" filter: "." map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `1m` * **Offset** - `1m` * **Format** - `RFC3339` **Authentication** Toggle **ON** and configure these parameters: * **Type****\*** - `Token` * **Token Retrieve Based Authentication** * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://abd4962.id.cyberark.cloud/OAuth2/Token/OnumCrowdStrike` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Authorization` * **Value** - `Basic ${secrets.auth_token}` * **Name** - `Content-Type` * **Value** - `application/x-www-form-urlencoded` * **Body Type****\***** -** `URLEncoded` * **Body Params** * **Name** - `grant_type` * **Value** - `client_credentials` * **Name** - `scope` * **Value** - `isp.audit.events:read` * **Token path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `Header` * **Name****\*** - `Authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Enumeration Phase** Toggle **ON** and configure these parameters: * **Pagination Type****\*** - `None` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://cmpc.audit.cyberark.cloud/api/audits/stream/createQuery` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Accept-Encoding` * **Value** - `gzip, deflate` * **Name** - `Content-Type` * **Value** - `aplication/json` * **Name** - `x-api-key` * **Value** - `${secrets.apikey}` * **Body Type****\*** - `Raw` * **Body Content** - ``` | { "query": { "pageSize": 500, "selectedFields": [ "tenant_id", "custom_data", "arrival_timestamp", "checksum", "application_code", "audit_code", "timestamp", "user_id", "session_id", "source", "action_type", "audit_type", "component", "target", "command", "message", "username", "action", "uuid", "service_name", "cloud_roles", "cloud_workspaces", "cloud_workspaces_and_roles", "cloud_assets", "cloud_identities", "vaulted_accounts", "cloud_provider", "account_name", "target_platform", "safe", "target_account", "identity_type", "access_method", "account_id", "correlation_id" ], "filterModel": { "date": { "dateFrom": "${temporalWindow.from}", "dateTo": "${temporalWindow.to}" } } } } ``` * **Output** * **Select****\*** - `.` * **Map** - `.` * **Output Mode****\*** - `element` **Collection Phase** * **Name** - `cursorRef` * **Expression** - `.cursorRef` * **Format** - `''` * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.paging.cursor.cursorRef` * **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://cmpc.audit.cyberark.cloud/api/audits/stream/results` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Accept-Encoding` * **Value** - `gzip, deflate` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `x-api-key` * **Value** - `${secrets.apikey}` * **Body Type****\*** - `Raw` * **Body Content****\*** - ``` | { "cursorRef": "${pagination.cursor}" } ``` * **Output** * **Select****\*** - `.data` * **Filter** - `.` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Doppel ## Overview Get a list of alerts from [Doppel](https://www.doppel.com/). ## Configuration ### Parameters N/A ### Secrets You must define these credentials in Onum: * `x-api-key` will reference your Doppel API Key. * `x-user-api-key` will reference your Doppel User API Key. To do it, enter a **Name** for the secret and then click the **Value** field. Click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the corresponding field. Click **Add element** to add the other one. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} After entering the required parameters and secrets, you can choose to manually enter the API fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: "2006-01-02T15:04:05" withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "page" pageSize: 200 isZeroIndex: true request: responseType: json method: "GET" url: "https://api.doppel.com/v1/alerts" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" - name: x-api-key value: ${secrets.x-api-key} - name: x-user-api-key value: ${secrets.x-user-api-key} queryParams: - name: created_after value: ${temporalWindow.from} - name: created_before value: ${temporalWindow.to} - name: page value: ${pagination.pageNumber} - name: page_size value: ${pagination.pageSize} output: select: ".data.alerts" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `5m` * **Offset** - `5m` * **Format** - `RFC3339` **Collection Phase** * **Pagination Type****\*** - `PageNumber/PageSize` * **Zero Index****\*** - `true` * **Page Size****\*** - `200` * **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://api.doppel.com/v1/alerts` * **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` * **Name** - `x-api-key` * **Value** - `${secrets.x-api-key}` * **Name** - `x-user-api-key` * **Value** - `${secrets.x-users-api-key}` * **Query Params** * **Name** - `created_after` * **Value** - `${temporalWindow.from}` * **Name** - `created_before` * **Value** - `${temporalWindow.to}` * **Name** - `page` * **Value** - `${pagination.pageNumber}` * **Name** - `page_size` * **Value** - `${pagination.pageSize}` **Output** * **Select****\*** - `.data.alerts` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Dropbox ## Overview Get a list of event streams from Dropbox. ## Configuration ### Parameters * `parameters.domain` will store the value of the API URL, excluding the endpoint paths like `/oauth2/token` or `/2/team_log/get_events`
### Secrets * `refresh_token`will reference the [Dropbox refresh token](https://pages.cs.link/prod/zscaler_block?url=https%3a%2f%2fdevelopers%2edropbox%2ecom%2foauth%2dguide\&referer=https%3a%2f%2fwww%2egoogle%2ecom%2f\&reason=Request+method+not+allowed+for+category+Personal+Use\&reasoncode=METHOD_DENIED\&timebound=1\&action=deny\&kind=category\&rule=1273379\&cat=Personal+Use\&user=sarah.bigault@crowdstrike.com\&locid=00000000\&lang=en_US\&zsq=0N0P65MHf00T5nJWPjF6sfJRsrJ46sq5HnVqJKQzsq). * `secrets.client_id` will reference the Client ID * `secrets.client_secret` will reference the Client Secret.
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Falcon API **Alerts** fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **CrowdStrike Falcon API** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded bodyType: urlEncoded bodyParams: - name: grant_type value: refresh_token - name: refresh_token value: '${secrets.refresh_token}' - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".cursor" initialRequest: method: POST url: "https://${parameters.domain}/2/team_log/get_events" headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "time": { "start_time": "${temporalWindow.from}", "end_time": "${temporalWindow.to}" } } nextRequest: method: POST url: "https://${parameters.domain}/2/team_log/get_events/continue" headers: - name: Content-Type value: application/json bodyRaw: | { "cursor": "${pagination.cursor}" } output: select: ".events" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 5 minutes (`5m`) as default, adjust based on your needs. * **Offset -** `5m` * **Format** - `RFC3339` **Authentication Phase** Toggle **ON** to configure the authentication phase. This is required to get the token to pull data using **OAuth**. * **Type****\***** -** `token` * **Request Method****\***** -** `POST` * **URL****\***** -** `${parameters.domain}/oauth2/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **BodyType****\*** - `UrlEncoded` * **Body params** * **Name -** `grant_type` * **Value -** `refresh_token` * **Name -** `refresh_token` * **Value -**`${secrets.refresh_token}` * **Name -** `client_id` * **Value -** `${secrets.client_id}` * **Name -** `client_secret` * **Value -** `${secrets.client_secret}` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `Bearer` * **Suffix** - `''` **Enumeration Phase** **OFF** **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor** - `.cursor` * **Cursor Selector -** `.cursor` * **Initial Request** * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.domain}/2/team_log/get_events` * **Headers -** * **Name** - `Content-Type` * **Value -** `application/json` * **Body Type -** `raw` * **Body Raw -** `|`\ `{`\ `"time": {`\ `"start_time": "${temporalWindow.from}",`\ `"end_time": "${temporalWindow.to}"`\ `}`\ `}` * **Next Request** * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.domain}/2/team_log/get_events/continue` * **Headers -** * **Name** - `Content-Type` * **Value -** `application/json` * **Body Type -** `raw` * **Body Raw -** `|`\ `{`\ `"cursor": "${pagination.cursor}"`\ `}` * **Output** * **Select -** `.events` * **Map -** `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} This HTTP Pull Listener now uses the business API to extract events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from FortiRecon ## Overview Where the vendor is **Fortinet**, it's product is **Fortirecon.** For Fortirecon, right now we have the following product types/endpoints: * Accounts * ACI * BP * EASM Inside each of those endpoints we have the YAML file to configure. This API endpoint returns the list of assets that have been marked as False Positive. ## Configuration ### Parameters * Domain (`organizationId`) ### Secrets * Auth Token (`fortireconAuth`) To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the **EASM** **endpoint** fields, or simply paste the desired YAML. ### Configure as YAML {% tabs %} {% tab title="Fortirecon EASM Breaches" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "page" pageSize: 100 isZeroIndex: false request: responseType: json method: GET url: https://api.fortirecon.forticloud.com/easm/${parameters.organizationId}/breaches headers: - name: Authorization value: ${secrets.fortireconAuth} queryParams: - name: page value: "${pagination.pageNumber}" - name: size value: "${pagination.pageSize}" - name: start_date value: ${temporalWindow.from} - name: end_date value: ${temporalWindow.to} output: select: ".hits" map: "." outputMode: element ``` {% endtab %} {% tab title="Fortirecon EASM Leaked Credentials" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "page" pageSize: 100 isZeroIndex: false request: responseType: json method: GET url: https://api.fortirecon.forticloud.com/easm/${parameters.organizationId}/leaked_creds headers: - name: Authorization value: ${secrets.fortireconAuth} queryParams: - name: page value: "${pagination.pageNumber}" - name: size value: "${pagination.pageSize}" - name: start_date value: ${temporalWindow.from} - name: end_date value: ${temporalWindow.to} output: select: ".hits" map: "." outputMode: element ``` {% endtab %} {% endtabs %} ### **Manually Configure** **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 5 minutes (`5m`) as default, adjust based on your needs. * **Offset -** `5m` * **Format** - `RCF3339` **Authentication Phase** **OFF** #### **Enumeration Phase** **OFF** #### **Collection Phase** * **Pagination Type****\***** -** `pageNumber/PageSize` * **Zero Index****\***** -** `false` * **Page Size****\***** -** `100` * **Request** * **Response Type****\***** -** `JSON` * **Method****\***** -** `GET` * **URL****\***** -** `https://api.fortirecon.forticloud.com/easm/${parameters.organizationId}/breaches` * **Headers -** * **Name** - `Authorization` * **Value -** `${secrets.fortireconAuth}` * **Query params** * **Name** - `page` * **Value -** `${pagination.pageNumber}` * **Name** - `Size` * **Value -** `${pagination.pageSize}` * **Name** - `start_date` * **Value -** `${temporalWindow.from}` * **Name** - `end_date` * **Value -** `${temporalWindow.to}` * **Output** * **Select -** `.hits` * **Map -** `.` * **Output Mode** - `element`
Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Greymatter API Where the vendor is **Greymatter**, we have Alert endpoints. # Alerts ## Overview Get a list of all or filtered alerts. The alerts listed are what remains after alert exclusions are applied by Netskope. ## Configuration ### Parameters * Domain (`Domain`) ### Secrets * Authorization (`greymatter_token`) refers to the[ API Token](https://docs.netskope.com/en/api-tokens-2) used to authenticate the connection to Greymatter. To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Greymatter API **Alerts** fields, or simply paste the desired YAML. ### Configure as YAML ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".data.drpAlerts.pageInfo.endCursor" initialRequest: method: POST url: "https://greymatter.myreliaquest.com/graphql" headers: - name: X-API-KEY value: ${secrets.greymatter_token} - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "query":"query drpAlerts ($after: String, $filter: DrpReportTriageItemViewFilterInput, $first: Int!, $orderBy: DRPAlertOrder) {\n drpAlerts (after: $after, filter: $filter, first: $first, orderBy: $orderBy) {\n edges {\n cursor\n node {\n active\n alertFingerprint\n classification\n closedSource\n createdAt\n domain\n hosts\n id\n migrated\n rejectedByRuleIds\n removedAt\n riskFactorKeys\n riskType\n severity\n shortCode\n sourceGroup\n sourceRef\n sourceUpdated\n sourceUris\n subTitle\n thumbnailUri\n title\n updatedAt\n uri\n }\n }\n pageInfo {\n endCursor\n hasNextPage\n hasPreviousPage\n startCursor\n }\n totalCount\n }\n}", "variables": { "after": "T18w", "filter": { "changed": "PT5M#UTC" }, "first": 1000, "orderBy": { "direction": "ASC", "orderBy": "CREATED_AT" } } } nextRequest: method: POST url: "https://greymatter.myreliaquest.com/graphql" headers: - name: X-API-KEY value: ${secrets.greymatter_token} - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "query": "query drpAlerts ($after: String, $filter: DrpReportTriageItemViewFilterInput, $first: Int!, $orderBy: DRPAlertOrder) {\n drpAlerts (after: $after, filter: $filter, first: $first, orderBy: $orderBy) {\n edges {\n cursor\n node {\n active\n alertFingerprint\n classification\n closedSource\n createdAt\n domain\n hosts\n id\n migrated\n rejectedByRuleIds\n removedAt\n riskFactorKeys\n riskType\n severity\n shortCode\n sourceGroup\n sourceRef\n sourceUpdated\n sourceUris\n subTitle\n thumbnailUri\n title\n updatedAt\n uri\n }\n }\n pageInfo {\n endCursor\n hasNextPage\n hasPreviousPage\n startCursor\n }\n totalCount\n }\n}", "variables": { "after": "${pagination.cursor}", "filter": { "changed": "PT5M#UTC" }, "first": 1000, "orderBy": { "direction": "ASC", "orderBy": "CREATED_AT" } } } output: select: ".data.drpAlerts.edges" map: "." outputMode: element ``` ### **Manually Configure** **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 5 minutes (`5m`) as default, adjust based on your needs. * **Offset -** `5m` * **Format** - `RFC3339` **Authentication Phase** **OFF** #### **Enumeration Phase** **OFF** #### **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor Selector****\***** -** `.data.drpAlerts.pageInfo.endCursor` * **Initial Request** * **Method****\***** -** `POST` * **URL****\***** -** `https://greymatter.myreliaquest.com/graphql` * **Headers -** * **Name** - `X-API-KEY` * **Value -** `${secrets.greymatter_token}` * **Name** - `Content-Type` * **Value -** `application/json` * **Body type-** `raw` * **Body raw** - `|`\ `{`\ `"query":"query drpAlerts ($after: String, $filter: DrpReportTriageItemViewFilterInput, $first: Int!, $orderBy: DRPAlertOrder) {\n drpAlerts (after: $after, filter: $filter, first: $first, orderBy: $orderBy) {\n edges {\n cursor\n node {\n active\n alertFingerprint\n classification\n closedSource\n createdAt\n domain\n hosts\n id\n migrated\n rejectedByRuleIds\n removedAt\n riskFactorKeys\n riskType\n severity\n shortCode\n sourceGroup\n sourceRef\n sourceUpdated\n sourceUris\n subTitle\n thumbnailUri\n title\n updatedAt\n uri\n }\n }\n pageInfo {\n endCursor\n hasNextPage\n hasPreviousPage\n startCursor\n }\n totalCount\n }\n}",`\ `"variables": {`\ `"after": "T18w",`\ `"filter": {`\ `"changed": "PT5M#UTC"`\ `},`\ `"first": 1000,`\ `"orderBy": {`\ `"direction": "ASC",`\ `"orderBy": "CREATED_AT"`\ `}`\ `}`\ `}` * **Next Request** * **Method****\***** -** `POST` * **URL****\***** -** `https://greymatter.myreliaquest.com/graphql` * **Headers -** * **Name** - `X-API-KEY` * **Value -** `${secrets.greymatter_token}` * **Name** - `Content-Type` * **Value -** `application/json` * **Body type-** `raw` * **Body raw** - `|`\ `{`\ `"query": "query drpAlerts ($after: String, $filter: DrpReportTriageItemViewFilterInput, $first: Int!, $orderBy: DRPAlertOrder) {\n drpAlerts (after: $after, filter: $filter, first: $first, orderBy: $orderBy) {\n edges {\n cursor\n node {\n active\n alertFingerprint\n classification\n closedSource\n createdAt\n domain\n hosts\n id\n migrated\n rejectedByRuleIds\n removedAt\n riskFactorKeys\n riskType\n severity\n shortCode\n sourceGroup\n sourceRef\n sourceUpdated\n sourceUris\n subTitle\n thumbnailUri\n title\n updatedAt\n uri\n }\n }\n pageInfo {\n endCursor\n hasNextPage\n hasPreviousPage\n startCursor\n }\n totalCount\n }\n}",`\ `"variables": {`\ `"after": "${pagination.cursor}",`\ `"filter": {`\ `"changed": "PT5M#UTC"`\ `},`\ `"first": 1000,`\ `"orderBy": {`\ `"direction": "ASC",`\ `"orderBy": "CREATED_AT"`\ `}`\ `}`\ `}` * **Output** * **Select -** `.data.drpAlerts.edges` * **Map -** `.` * **Output Mode** - `element` Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Hoxhunt ## Overview Get a list of call records using the [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/). ## HTTP Pull Listener configuration ### Parameters N/A ### Secrets You must define these credentials in Onum: * `hoxhuntToken` will reference your Hoxhunt token. To do it, enter a **Name** for the secret and then click the **Value** field. Click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the corresponding field. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} ### Configuration After entering the required parameters and secrets, you can choose to manually enter the API fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".data.incidents.pageInfo.endCursor" initialRequest: method: POST url: "https://app.hoxhunt.com/graphql-external" headers: - name: Authorization value: "Authtoken ${secrets.hoxhuntToken}" bodyType: raw bodyRaw: | { "query": "query($startDate: DateTime, $endDate: DateTime) {\n incidents(\n first: 100, \n filter: {\n createdAt: {\n gte: $startDate,\n lte: $endDate\n }\n }\n ) {\n edges {\n node {\n id\n createdAt\n updatedAt\n status\n severity\n reportedBy\n description\n }\n }\n pageInfo {\n hasNextPage\n endCursor\n }\n }\n}", "variables": "{\n \"startDate\": \"${temporalWindow.from}\",\n \"endDate\": \"${temporalWindow.to}\"\n}" } nextRequest: method: POST url: "https://app.hoxhunt.com/graphql-external" headers: - name: Authorization value: "Authtoken ${secrets.hoxhuntToken}" bodyType: raw bodyRaw: | { "query": "query($cursor: String, $startDate: DateTime, $endDate: DateTime) {\n incidents(\n first: 100, \n after: $cursor,\n filter: {\n createdAt: {\n gte: $startDate,\n lte: $endDate\n }\n }\n ) {\n edges {\n node {\n id\n createdAt\n updatedAt\n status\n severity\n reportedBy\n description\n }\n }\n pageInfo {\n hasNextPage\n endCursor\n }\n }\n}", "variables": "{\n \"cursor\": \"${pagination.cursor}\"\n \"startDate\": \"${temporalWindow.from}\",\n \"endDate\": \"${temporalWindow.to}\"\n}" } output: select: ".data.incidents" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `5m` * **Offset** - `5m` * **Format** - `RFC3339` **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.data.incidents.pageInfo.endCursor` **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://app.hoxhunt.com/graphql-external` * **Headers** * **Name** - `Authorization` * **Value** - `Authtoken ${secrets.hoxhuntToken}` * **Body Type****\*** - `Raw` * **Body Content****\*** - ``` | { "query": "query($startDate: DateTime, $endDate: DateTime) {\n incidents(\n first: 100, \n filter: {\n createdAt: {\n gte: $startDate,\n lte: $endDate\n }\n }\n ) {\n edges {\n node {\n id\n createdAt\n updatedAt\n status\n severity\n reportedBy\n description\n }\n }\n pageInfo {\n hasNextPage\n endCursor\n }\n }\n}", "variables": "{\n \"startDate\": \"${temporalWindow.from}\",\n \"endDate\": \"${temporalWindow.to}\"\n}" } ``` **Next Request** * **Response Type****\*** - `JSON` * **Method****\*** - `POST` * **URL****\*** - `https://app.hoxhunt.com/graphql-external` * **Headers** * **Name** - `Authorization` * **Value** - `Authtoken ${secrets.hoxhuntToken}` * **Body Type****\*** - `Raw` * **Body Content****\*** - ``` | { "query": "query($cursor: String, $startDate: DateTime, $endDate: DateTime) {\n incidents(\n first: 100, \n after: $cursor,\n filter: {\n createdAt: {\n gte: $startDate,\n lte: $endDate\n }\n }\n ) {\n edges {\n node {\n id\n createdAt\n updatedAt\n status\n severity\n reportedBy\n description\n }\n }\n pageInfo {\n hasNextPage\n endCursor\n }\n }\n}", "variables": "{\n \"cursor\": \"${pagination.cursor}\"\n \"startDate\": \"${temporalWindow.from}\",\n \"endDate\": \"${temporalWindow.to}\"\n}" } ``` **Output** * **Select****\*** - `.data.incidents` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Microsoft Where the vendor is **Microsoft**, right now we have the following product types/endpoints: * [Microsoft Graph API](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-microsoft/microsoft-graph-api) * [Monitor API](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-microsoft/azure-monitor-api) Inside each of those endpoints we have the YAML file to configure. # Microsoft Graph API # Call records ## Overview Get a list of call records using the [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/). ## Configuration ### Parameters Add the following parameter: * **Name** - `tenant_id` * **Value** - Enter your Microsoft tenant ID. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Microsoft Graph client ID. * `client_secret` will reference your Microsoft Graph client secret. To do it, enter a **Name** for the secret and then click the **Value** field. Click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the corresponding field. Click **Add element** to add the other one. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} After entering the required parameters and secrets, you can choose to manually enter the API fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://login.microsoftonline.com/${parameters.tenant_id}/oauth2/v2.0/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: scope value: https://graph.microsoft.com/.default - name: grant_type value: client_credentials tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: "responseBodyLink" responseBodyLinkSelector: .["@odata.context"] request: responseType: json method: "GET" url: "https://graph.microsoft.com/v1.0/communications/callRecords" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" queryParams: - name: $filter value: startDateTime ge ${temporalWindow.from} and startDateTime lt ${temporalWindow.to} output: select: ".value" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `5m` * **Offset** - `5m` * **Format** - `RFC3339` **Authentication** Toggle **ON** and configure these parameters: * **Type****\*** - `Token` **Token Retrieve Based Authentication** * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://login.microsoftonline.com/${parameters.tenant_id}/oauth2/v2.0/token` * **Headers** * **Name** - `Content-Type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\***** -** `URLEncoded` * **Body Params** * **Name** - `client_id` * **Value** - `${secrets.client_id}` * **Name** - `client_secret` * **Value** - `${secrets.client_secret}` * **Name** - `scope` * **Value** - `https://graph.microsoft.com/.default` * **Name** - `grant_type` * **Value** - `client_credentials` * **Token path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `Header` * **Name****\*** - `Authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Next Link at Response Body` * **Selector****\*** - `.["@odata.context"]` **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://graph.microsoft.com/v1.0/communications/callRecords` **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` **Query Params** * **Name** - `$filter` * **Value** - `startDateTime ge ${temporalWindow.from} and startDateTime lt ${temporalWindow.to}` **Output** * **Select****\*** - `.value` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Message traces ## Overview Get a list of message traces using the [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/). ## Configuration ### Parameters Add the following parameter: * **Name** - `tenant_id` * **Value** - Enter your Microsoft tenant ID. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Microsoft Graph client ID. * `client_secret` will reference your Microsoft Graph client secret. To do it, enter a **Name** for the secret and then click the **Value** field. Click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the corresponding field. Click **Add element** to add the other one. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} After entering the required parameters and secrets, you can choose to manually enter the API fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://login.microsoftonline.com/${parameters.tenant_id}/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: resource value: https://graph.microsoft.com - name: grant_type value: client_credentials tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: "responseBodyLink" responseBodyLinkSelector: .["@odata.nextLink"] request: responseType: json method: "GET" url: "https://graph.microsoft.com/v1.0/admin/exchange/tracing/messageTraces" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" queryParams: - name: $filter value: receivedDateTime ge ${temporalWindow.from} and receivedDateTime le ${temporalWindow.to} - name: $top value: 5000 output: select: ".value" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `5m` * **Offset** - `5m` * **Format** - `RFC3339` **Authentication** Toggle **ON** and configure these parameters: * **Type****\*** - `Token` **Token Retrieve Based Authentication** * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://login.microsoftonline.com/${parameters.tenant_id}/oauth2/v2.0/token` * **Headers** * **Name** - `Content-Type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\***** -** `URLEncoded` * **Body Params** * **Name** - `client_id` * **Value** - `${secrets.client_id}` * **Name** - `client_secret` * **Value** - `${secrets.client_secret}` * **Name** - `resource` * **Value** - `https://graph.microsoft.com` * **Name** - `grant_type` * **Value** - `client_credentials` * **Token path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `Header` * **Name****\*** - `Authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `Next Link at Response Body` * **Selector****\*** - `.["@odata.nextLink"]` **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://graph.microsoft.com/v1.0/admin/exchange/tracing/messageTraces` **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` **Query Params** * **Name** - `$filter` * **Value** - `receivedDateTime ge ${temporalWindow.from} and receivedDateTime le ${temporalWindow.to}` * **Name** - `$top` * **Value** - `5000` **Output** * **Select****\*** - `.value` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Azure Monitor API ## Overview Get a list of call records using the [Azure Monitor API](https://learn.microsoft.com/en-us/rest/api/monitor/). ## Configuration ### Parameters Add the following parameter: * **Name** - `tenant_id` * **Value** - Enter your Microsoft tenant ID. ### Secrets You must define these credentials in Onum: * `client_id` will reference your Microsoft Azure client ID. * `client_secret` will reference your Microsoft Azure client secret. To do it, enter a **Name** for the secret and then click the **Value** field. Click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the corresponding field. Click **Add element** to add the other one. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} After entering the required parameters and secrets, you can choose to manually enter the API fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 15m offset: 15m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://login.microsoftonline.com/${secrets.tenant_id}/oauth2/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: resource value: https://management.azure.com - name: grant_type value: client_credentials tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: "none" request: responseType: json method: "GET" url: "https://management.azure.com/subscriptions/${parameters.subscription_id}/providers/Microsoft.Insights/eventtypes/management/values" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" queryParams: - name: $filter value: eventTimestamp ge ${temporalWindow.from} and eventTimestamp le ${temporalWindow.to} - name: api-version value: 2015-04-01 output: select: ".value" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `15m` * **Offset** - `15m` * **Format** - `RFC3339` **Authentication** Toggle **ON** and configure these parameters: * **Type****\*** - `Token` **Token Retrieve Based Authentication** * **Request** * **Method****\*** - `POST` * **URL****\*** - `https://login.microsoftonline.com/${secrets.tenant_id}/oauth2/v2.0/token` * **Headers** * **Name** - `Content-Type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **Body Type****\***** -** `URLEncoded` * **Body Params** * **Name** - `client_id` * **Value** - `${secrets.client_id}` * **Name** - `client_secret` * **Value** - `${secrets.client_secret}` * **Name** - `resource` * **Value** - `https://management.azure.com` * **Name** - `grant_type` * **Value** - `client_credentials` * **Token path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `Header` * **Name****\*** - `Authorization` * **Prefix** - `'Bearer '` * **Suffix** - `''` **Collection Phase** * **Pagination Type****\*** - `None` **Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://management.azure.com/subscriptions/${parameters.subscription_id}/providers/Microsoft.Insights/eventtypes/management/values` **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `Content-Type` * **Value** - `application/json` **Query Params** * **Name** - `$filter` * **Value** - `eventTimestamp ge ${temporalWindow.from} and eventTimestamp le ${temporalWindow.to}` * **Name** - `api-version` * **Value** - `2015-04-01` **Output** * **Select****\*** - `.value` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Mimecast Where the vendor is **Mimecast**, we have the following product types/endpoints: * [Audit](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-mimecast/audit) * [Security](https://docs.onum.com/~/revisions/computed_1XUbIXX4i0OVBRVvl6U4_8027d6f0fd5d06c0294669b38a24187237e4a4cc/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-mimecast/security) * [Threats](https://docs.onum.com/~/revisions/computed_1XUbIXX4i0OVBRVvl6U4_8027d6f0fd5d06c0294669b38a24187237e4a4cc/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-mimecast/threats) Inside each of those endpoints we have the YAML file to configure. # Audit ## Overview Returns the audit events matching the request. #### Pre-requisites In order to successfully use this endpoint, the role assigned to the app must have at least the following level of application permissions granted: Account | Logs | Read. The page size should be provided in the meta pagination, otherwise the default is 10. ## Configuration ### Parameters * Domain (`domain`) ### Secrets * **Client Id** (`client_id`) * **Client Secret** (`client_secret`) Open the **Secret** fields and click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: grant_type value: 'client_credentials' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".meta.pagination.next" initialRequest: method: POST url: "https://${parameters.domain}/api/audit/get-audit-events" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "startDateTime":"${temporalWindow.from}", "endDateTime":"${temporalWindow.to}" } ], "meta": { "pagination": { "pageSize": 500, } } } nextRequest: method: POST url: "https://${parameters.domain}/api/audit/get-audit-events" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "startDateTime":"${temporalWindow.from}", "endDateTime":"${temporalWindow.to}" } ], "meta": { "pagination": { "pageToken": "${pagination.cursor}", "pageSize": 500, } } } output: select: ".data" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 1 minute * **Offset -** initial offset should be `1m` * **Format** - `RFC3339` **Authentication Phase** Toggle **ON** to set the Authentication settings. * **Type****\***** -** `token` * **Request Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.domain}/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **BodyType****\*** - `UrlEncoded` * **Body params** * **Name -** `client_id` * **Value -**`${secrets.client_id}` * **Name -** `client_secret` * **Value -**`${secrets.client_secret}` * **Name -** `grant type` * **Value -** `client_credentials` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `Bearer` * **Suffix** - `''` **Enumeration Phase** **OFF** **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor Selector****\***** -** `.meta.pagination.next` * **Initial Request** * **Method****\***** -** `GET` * **URL****\***** -** `https://${parameters.domain}/api/audit/get-audit-events` * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Content-Type` * **Value -** `application/json` * **Body** - `raw` * **raw** `|`\ `{`\ `"data": [`\ `{`\ `"startDateTime":"${temporalWindow.from}",`\ `"endDateTime":"${temporalWindow.to}"`\ `}`\ `],`\ `"meta": {`\ `"pagination": {`\ `"pageSize": 500,`\ `}`\ `}`\ `}` * **Next Request** * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.domain}/api/audit/get-audit-events` * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Content-Type` * **Value -** `application/json` * **Body** - `raw` * **raw** `|`\ `{`\ `"data": [`\ `{`\ `"startDateTime":"${temporalWindow.from}",`\ `"endDateTime":"${temporalWindow.to}"`\ `}`\ `],`\ `"meta": {`\ `"pagination": {`\ `"pageToken": "${pagination.cursor}",`\ `"pageSize": 500,`\ `}`\ `}`\ `}` * **Output** * **Select -** `.data` * **Map -** `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} This HTTP Pull Listener now uses the data export API to extract audit events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Security ## Overview Returns the audit events matching security requests. ## Configuration ### Parameters * Domain (`domain`) ### Secrets * **Client Id** (`client_id`) * **Client Secret** (`client_secret`) Open the **Secret** fields and click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the fields, or simply paste the given YAML. ### Configure as YAML {% tabs %} {% tab title="Attachments" %} ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: grant_type value: 'client_credentials' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".meta.pagination.next" initialRequest: method: POST url: "https://${parameters.domain}/api/ttp/attachment/get-logs" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "to":"${temporalWindow.to}", "from":"${temporalWindow.from}" } ], "meta": { "pagination": { "pageSize": 500, } } } nextRequest: method: POST url: "https://${parameters.domain}/api/ttp/attachment/get-logs" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "to":"${temporalWindow.to}", "from":"${temporalWindow.from}" } ], "meta": { "pagination": { "pageToken": ${pagination.cursor}, "pageSize": 500, } } } output: select: ".data[0].clickLogs" map: "." outputMode: element ``` {% endtab %} {% tab title="Logs" %} ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: grant_type value: 'client_credentials' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".meta.pagination.next" initialRequest: method: POST url: "https://${parameters.domain}/api/dlp/get-logs" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "to":"${temporalWindow.to}", "from":"${temporalWindow.from}", "oldestFirst": false } ], "meta": { "pagination": { "pageSize": 500, } } } nextRequest: method: POST url: "https://${parameters.domain}/api/dlp/get-logs" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "to":"${temporalWindow.to}", "from":"${temporalWindow.from}", "oldestFirst": false } ], "meta": { "pagination": { "pageToken": ${pagination.cursor}, "pageSize": 500, } } } output: select: ".data[0].dlpLogs" map: "." outputMode: element ``` {% endtab %} {% tab title="URLs" %} ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: grant_type value: 'client_credentials' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".meta.pagination.next" initialRequest: method: POST url: "https://${parameters.domain}/api/ttp/url/get-logs" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "to":"${temporalWindow.to}", "from":"${temporalWindow.from}" } ], "meta": { "pagination": { "pageSize": 500, } } } nextRequest: method: POST url: "https://${parameters.domain}/api/ttp/url/get-logs" headers: - name: Accept value: application/json - name: Content-Type value: application/json bodyRaw: | { "data": [ { "to":"${temporalWindow.to}", "from":"${temporalWindow.from}" } ], "meta": { "pagination": { "pageToken": ${pagination.cursor}, "pageSize": 500, } } } output: select: ".data[0].clickLogs" map: "." outputMode: element ``` {% endtab %} {% endtabs %} ### **Manually configure** All parameters are the same, except the URLs in the request fields, which you change according to the type Attachment - `https://${parameters.domain}/api/ttp/attachment/get-logs` Logs - `https://${parameters.domain}/api/dlp/get-logs` URLs - `https://${parameters.domain}/api/ttp/url/get-logs` **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 1 minute * **Offset -** initial offset should be `1m` * **Format** - `RFC3339` **Authentication Phase** Toggle **ON** to set the Authentication settings. * **Type****\***** -** `token` * **Request Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.domain}/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **BodyType****\*** - `UrlEncoded` * **Body params** * **Name -** `client_id` * **Value -**`${secrets.client_id}` * **Name -** `client_secret` * **Value -**`${secrets.client_secret}` * **Name -** `grant type` * **Value -** `client_credentials` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `Bearer` * **Suffix** - `''` **Enumeration Phase** **OFF** **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor Selector****\***** -** `.meta.pagination.next` * **Initial Request** * **Method****\***** -** `POST` * **URL****\***** - INSERT URL HERE** * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Content-Type` * **Value -** `application/json` * **Body** - `raw` * **raw** `|`\ `{`\ `"data": [`\ `{`\ `"to":"${temporalWindow.to}",`\ `"from":"${temporalWindow.from}"`\ `}`\ `],`\ `"meta": {`\ `"pagination": {`\ `"pageSize": 500,`\ `}`\ `}`\ `}` * **Next Request** * **Method****\***** -** `POST` * **URL****\***** - INSERT URL HERE** * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Content-Type` * **Value -** `application/json` * **Body** - `raw` * **raw** `|`\ `{`\ `"data": [`\ `{`\ `"to":"${temporalWindow.to}",`\ `"from":"${temporalWindow.from}"`\ `}`\ `],`\ `"meta": {`\ `"pagination": {`\ `"pageToken": ${pagination.cursor},`\ `"pageSize": 500,`\ `}`\ `}`\ `}` * **Output** * **Select -** `.data[0].clickLogs` * **Map -** `.` * **Output Mode** - `element` This HTTP Pull Listener now uses the data export API to extract audit events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Threats ## Overview Returns the audit events matching security requests. ## Configuration ### Parameters * Domain (`domain`) ### Secrets * **Client Id** (`client_id`) * **Client Secret** (`client_secret`) Open the **Secret** fields and click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the fields, or simply paste the given YAML. ### Configure as YAML {% tabs %} {% tab title="SIEM events" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: grant_type value: 'client_credentials' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: .["@nextPage"] initialRequest: method: GET url: "https://${parameters.domain}/siem/v1/events/cg" headers: - name: Accept value: application/json - name: Content-Type value: application/json queryParams: - name: dateRangeStartsAt value: ${temporalWindow.from} - name: dateRangeEndsAt value: ${temporalWindow.to} - name: pageSize value: 100 nextRequest: method: GET url: "https://${parameters.domain}/siem/v1/events/cg" headers: - name: Accept value: application/json - name: Content-Type value: application/json queryParams: - name: dateRangeStartsAt value: ${temporalWindow.from} - name: dateRangeEndsAt value: ${temporalWindow.to} - name: pageSize value: 100 - name: nextPage value: ${pagination.cursor} output: select: ".value" map: "." outputMode: element ``` {% endtab %} {% tab title="Events" %} ```yaml withTemporalWindow: true temporalWindow: duration: 1m offset: 1m tz: UTC format: 2006-01-02T15:04:05Z withAuthentication: true authentication: type: token token: request: method: POST url: https://${parameters.domain}/oauth/token headers: - name: Content-Type value: application/x-www-form-urlencoded - name: Accept value: application/json bodyType: urlEncoded bodyParams: - name: client_id value: '${secrets.client_id}' - name: client_secret value: '${secrets.client_secret}' - name: grant_type value: 'client_credentials' tokenPath: ".access_token" authInjection: in: header name: Authorization prefix: 'Bearer ' suffix: '' withEnumerationPhase: false collectionPhase: paginationType: cursor cursorSelector: ".meta.nextPage" initialRequest: method: GET url: "https://${parameters.domain}/threats/v1/events" headers: - name: Accept value: application/json - name: Content-Type value: application/json queryParams: - name: timestampRangeStartsAt value: ${temporalWindow.from} - name: timestampRangeEndsAt value: ${temporalWindow.to} - name: pageSize value: 50 nextRequest: method: GET url: "https://${parameters.domain}/threats/v1/events" headers: - name: Accept value: application/json - name: Content-Type value: application/json queryParams: - name: timestampRangeStartsAt value: ${temporalWindow.from} - name: timestampRangeEndsAt value: ${temporalWindow.to} - name: pageSize value: 50 - name: pageToken value: ${pagination.cursor} output: select: ".value" map: "." outputMode: element ``` {% endtab %} {% endtabs %} ### **Manually configure** All parameters are the same, except the URLs in the request fields, which you change according to the type Events - `https://${parameters.domain}/threats/v1/events` SIEM Events - `https://${parameters.domain}/siem/v1/events/cg` **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 1 minute * **Offset -** initial offset should be `1m` * **Format** - `2006-01-02T15:04:05Z` **Authentication Phase** Toggle **ON** to set the Authentication settings. * **Type****\***** -** `token` * **Request Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.domain}/oauth/token` * **Headers** * **Name** - `Content-type` * **Value** - `application/x-www-form-urlencoded` * **Name** - `Accept` * **Value** - `application/json` * **BodyType****\*** - `UrlEncoded` * **Body params** * **Name -** `client_id` * **Value -**`${secrets.client_id}` * **Name -** `client_secret` * **Value -**`${secrets.client_secret}` * **Name -** `grant type` * **Value -** `client_credentials` * **Token Path****\*** - `.access_token` * **Auth Injection** * **In****\*** - `header` * **Name****\*** - `authorization` * **Prefix** - `Bearer` * **Suffix** - `''` **Enumeration Phase** **OFF** **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor Selector****\***** -** `.meta.nextPage` * **Initial Request** * **Method****\***** -** `GET` * **URL****\***** - INSERT URL HERE** * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Content-Type` * **Value -** `application/json` * **Query Params** * **Name** - `timestampRangeStartsAt` * **Value -** `${temporalWindow.from}` * **Name** - `timestampRangeEndsAt` * **Value -** `${temporalWindow.to}` * **Name** - `pageSize` * **Value -** `50` * **Next Request** * **Method****\***** -** `GET` * **URL****\***** - INSERT URL HERE** * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Content-Type` * **Value -** `application/json` * **Query Params** * **Name** - `timestampRangeStartsAt` * **Value -** `${temporalWindow.from}` * **Name** - `timestampRangeEndsAt` * **Value -** `${temporalWindow.to}` * **Name** - `pageSize` * **Value -** `50` * **Name** - `pageToken` * **Value -** `${pagination.cursor}` * **Output** * **Select -** `.value` * **Map -** `.` * **Output Mode** - `element` This HTTP Pull Listener now uses the data export API to extract audit events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Netskope Where the vendor is **Netskope**, we have the following product types/endpoints: * [Alerts](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-netskope/alert-endpoints) * [Events](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-netskope/event) Inside each of those endpoints we have the YAML file to configure. # Alert Endpoints ## Overview Get a list of all or filtered alerts. The alerts listed are what remains after alert exclusions are applied by Netskope. ## Configuration ### Parameters * Domain (`netskopeDomain`) * Index (`netskopeIndex`) - The index parameter in the Netskope API for Data Export is used to: * Uniquely identify an export session. * Prevent multiple API consumers from overlapping their collections. * Allow incremental paging without losing events. ### Secrets * `NetskopeApiToken` refers to the[ API Token](https://docs.netskope.com/en/api-tokens-2) used to authenticate the connection to Netskope.
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Netskope API **Alerts** fields, or simply paste the desired YAML. ### Configure as YAML {% tabs %} {% tab title="Compromised Credential" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/compromisedcredential?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/compromisedcredential?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="CTEP" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/ctep?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/ctep?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="DLP" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/dlp?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/dlp?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Malsite" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/malsite?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/malsite?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Malware" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/malware?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/malware?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Policy" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/policy?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/policy?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Quarantine" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/quarantine?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/quarantine?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Remediation" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/remediation?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/remediation?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Security Assessment" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/securityassessment?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/securityassessment?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="UBA" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/uba?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/uba?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Watchlist" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/watchlist?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/alerts/watchlist?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% endtabs %} ### **Manually Configure** **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 5 minutes (`5m`) as default, adjust based on your needs. * **Offset -** `5m` * **Format** - `Epoch` **Authentication Phase** **OFF** #### **Enumeration Phase** **OFF** #### **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor Selector****\***** -** `.timestamp_hwm` * **Initial Request** * **Method****\***** -** `GET` * **URL****\***** -** `https://${parameters.domain}/api/v2/events/dataexport/alerts/`***`INSERT NAME FROM YAMLS ABOVE`***` ``?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}` * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Netskope-Api-Token` * **Value -** `${secrets.netskopeApiToken}` **Next Request** * **Method****\***** -** `GET` * **URL****\***** -** `https://${parameters.domain}/api/v2/events/dataexport/alerts/`***`INSERT NAME`***`?operation=next&index=${parameters.netskopeIndex}` * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Netskope-Api-Token` * **Value -** `${secrets.netskopeApiToken}` * **Output** * **Select -** `.result` * **Map -** `.` * **Output Mode** - `element`
Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Event ## Overview Get a list of all or filtered events. ## Configuration ### Parameters * Domain (`netskopeDomain`) * Index (`netskopeIndex`) - The index parameter in the Netskope API for Data Export is used to: * Uniquely identify an export session. * Prevent multiple API consumers from overlapping their collections. * Allow incremental paging without losing events. ### Secrets * `NetskopeApiToken` refers to the[ API Token](https://docs.netskope.com/en/api-tokens-2) used to authenticate the connection to Netskope.
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Netskope API **Alerts** fields, or simply paste the desired YAML. ### Configure as YAML {% tabs %} {% tab title="Alert" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/alert?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/alert?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Application" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/application?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/application?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Audit" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/audit?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/audit?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Incident" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/incident?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/incident?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Infrastructure" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/infrastructure?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/infrastructure?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Network" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/network?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/network?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% tab title="Page" %} ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".timestamp_hwm" initialRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/page?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" nextRequest: method: GET url: "https://${parameters.domain}/api/v2/events/dataexport/events/page?operation=next&index=${parameters.netskopeIndex}" headers: - name: Accept value: application/json - name: Netskope-Api-Token value: "${secrets.netskopeApiToken}" output: select: ".result" map: "." outputMode: element ``` {% endtab %} {% endtabs %} ### **Manually Configure** **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 5 minutes (`5m`) as default, adjust based on your needs. * **Offset -** `5m` * **Format** - `Epoch` **Authentication Phase** **OFF** #### **Enumeration Phase** **OFF** #### **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor Selector****\***** -** `.timestamp_hwm` * **Initial Request** * **Method****\***** -** `GET` * **URL****\***** -** `https://${parameters.domain}/api/v2/events/dataexport/events/`***`INSERT NAME FROM YAML`***`?index=${parameters.netskopeIndex}&operation=${temporalWindow.from}` * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Netskope-Api-Token` * **Value -** `${secrets.netskopeApiToken}` **Next Request** * **Method****\***** -** `GET` * **URL****\***** -** `https://${parameters.domain}/api/v2/events/dataexport/events/`***`INSERT NAME`***`?operation=next&index=${parameters.netskopeIndex}` * **Headers -** * **Name** - `Accept` * **Value -** `application/json` * **Name** - `Netskope-Api-Token` * **Value -** `${secrets.netskopeApiToken}` * **Body type****\***** -** there is no required body type because the parameters are included in the URL. However, these fields are mandatory, so select `raw` and enter the `{}` placeholder. * **Output** * **Select -** `.result` * **Map -** `.` * **Output Mode** - `element`
Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from OKTA ## Overview Get system logs using the OKTA API. ## Configuration ### Parameters * `parameters.mydomain` will store the value of the API URL, excluding the endpoint paths like or `/api/v1/logs`
### Secrets * Auth Token (`OktaAuthorization`)
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the OKTA System Log fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: "2006-01-02T15:04:05" withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "webLinking" limit: 1000 request: responseType: json method: "GET" url: "https://${parameters.mydomain}/api/v1/logs" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" - name: Authorization value: "SSWS ${secrets.OktaAuthorization}" queryParams: - name: since value: "${temporalWindow.from}" - name: until value: "${temporalWindow.to}" output: select: "." map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 5 minutes (`5m`) as default, adjust based on your needs. * **Offset -** `5m` * **Format** - `2006-01-02T15:04:05` **Authentication Phase** Off #### **Enumeration Phase** Off #### **Collection Phase** * **Pagination Type****\***** -** `webLinking` * **Zero index****\***** -** false * **Limit****\***** -** 100 * **Request** * **Response Type****\***** -** `JSON` * **Method****\***** -** `GET` * **URL****\***** -** `https://${parameters.myDomain}/api/v1/logs` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Name** - Authorization * **Value** - `SSWS ${secrets.OktaAuthorization}` * **Query Params** * **Name** - since * **Value** - `${temporalWindow.from}` * **Name** - Content-Type * **Value** -`${temporalWindow.to}` * **Output** * **Select -** `.` * **Map -** `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} This HTTP Pull Listener now uses the data export API to extract events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Palo Alto products # Collect data from Cortex XDR Integrate with API Logs from the Cortex Platform using the[ **HTTP Pull**](broken://pages/bZ07iBY7MKqfhB9A0NAf) Listener using the data Integration API. # Incident Management - Alerts ## Overview Get a list of all or filtered alerts. The alerts listed are what remains after alert exclusions are applied by Cortex XDR. * Response is concatenated using AND condition (OR is not supported). * Maximum result set size is 100. * Offset is the zero-based number of alerts from the start of the result set. The response indicates whether an PAN NGFW type alert contains a PCAP triggering packet. Use the Retrieve PCAP Packet API to retrieve a list of alert IDs and their associated PCAP data. Required license: Cortex XDR Prevent, Cortex XDR Pro per Endpoint, or Cortex XDR Pro per GB. ## Configuration ### Parameters **Name** - domain **Value** - `CortexXdrDomain` ### Secrets * `CortexXDRAuthorization` will reference the Cortex XDR Authorization token. * `CortexXDRAuthId` will reference the [Cortex XDR Authorization ID](https://docs-cortex.paloaltonetworks.com/r/Cortex-XDR-Platform-APIs/Get-your-Cortex-XDR-API-key-ID?contentId=GwbuuBI0cJvEhgDAelFiMw). To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Cortex incident Management fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Cortex XDR API** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "fromTo" limit: 100 request: method: "POST" url: "https://${parameters.CortexXdrDomain}/public_api/v1/alerts/get_alerts" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" - name: Authorization value: "${secrets.CortexXdrAuthorization}" - name: x-xdr-auth-id value: ${secrets.CortexXdrAuthId} bodyType: raw bodyRaw: | { "request_data": { "search_from": ${pagination.from}, "search_to": ${pagination.to}, "filters": [ { "field": "creation_time", "operator": "gte", "value": ${temporalWindow.from} }, { "field": "creation_time", "operator": "lte", "value": ${temporalWindow.to} } ] } } output: select: ".reply.alerts" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected.
**Authentication Phase** Off #### **Enumeration Phase** Off #### **Collection Phase** * **Pagination Type****\***** -** `fromTo` * **Zero index****\***** -** false * **Limit****\***** -** 100 * **Request** * **Response Type****\***** -** JSON * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.CortexXdrDomain}/public_api/v1/alerts/get_alerts` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Name** - Authorization * **Value** - `${secrets.CortexXdrAuthorization`} * **Name** - x-xdr-auth-id * **Value** - `${secrets.CortexXdrAuthId}` * **Body type****\***** -** raw * **Body content****\*** - `{`\ `"request_data": {`\ `"search_from": ${pagination.from},`\ `"search_to": ${pagination.to},`\ `"filters": [`\ `{`\ `"field": "creation_time",`\ `"operator": "gte",`\ `"value": ${temporalWindow.from}`\ `},`\ `{`\ `"field": "creation_time",`\ `"operator": "lte",`\ `"value": ${temporalWindow.to}`\ `}`\ `]`\ `}`\ `}` * **Output** * **Select -** `.reply.alerts` * **Map -** `.` * **Output Mode** - `element`
{% endtab %} {% endtabs %} This HTTP Pull Listener now uses the data export API to extract events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Incident Management - Multi Alerts ## Overview Get a list of alerts with multiple events. * The response is concatenated using AND condition (OR is not supported). * The maximum result set size is 100. * Offset is the zero-based number of alerts from the start of the result set. Cortex XDR displays in the API response whether a PAN NGFW type alert contains a PCAP triggering packet. Use the Retrieve PCAP Packet API to retrieve a list of alert IDs and their associated PCAP data. Required license: Cortex XDR Prevent, Cortex XDR Pro per Endpoint, or Cortex XDR Pro per GB. ## Configuration ### Parameters **Name** - domain **Value** - `CortexXdrDomain` ### Secrets * `CortexXDRAuthorization` will reference the Cortex XDR Authorization token. * `CortexXDRAuthId` will reference the [Cortex XDR Authorization ID](https://docs-cortex.paloaltonetworks.com/r/Cortex-XDR-Platform-APIs/Get-your-Cortex-XDR-API-key-ID?contentId=GwbuuBI0cJvEhgDAelFiMw).
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Cortex incident Management fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Cortex XDR multi alerts** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "fromTo" limit: 100 request: responseType: json method: "POST" url: "https://${parameters.CortexXdrDomain}/public_api/v2/alerts/get_alerts_multi_events" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" - name: Authorization value: "${secrets.CortexXdrAuthorization}" - name: x-xdr-auth-id value: ${secrets.CortexXdrAuthId} bodyType: raw bodyRaw: | { "request_data": { "search_from": ${pagination.from}, "search_to": ${pagination.to}, "filters": [ { "field": "creation_time", "operator": "lte", "value": ${temporalWindow.to} } ] } } output: select: ".reply.alerts" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected.
**Authentication Phase** Off #### **Enumeration Phase** Off #### **Collection Phase** * **Pagination Type****\***** -** `fromTo` * **Zero index****\***** -** false * **Limit****\***** -** 100 * **Request** * **Response Type****\***** -** JSON * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.CortexXdrDomain}/public_api/v1/alerts/get_alerts` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Name** - Authorization * **Value** - `${secrets.CortexXdrAuthorization}` * **Name** - x-xdr-auth-id * **Value** - `${secrets.CortexXdrAuthId}` * **Body type****\***** -** raw * **Body content****\*** - `{`\ `"request_data": {`\ `"search_from": ${pagination.from},`\ `"search_to": ${pagination.to},`\ `"filters": [`\ `{`\ `"field": "creation_time",`\ `"operator": "lte",`\ `"value": ${temporalWindow.to}`\ `}`\ `]`\ `}`\ `}` * **Output** * **Select -** `.reply.alerts` * **Map -** `.` * **Output Mode** - `element`
{% endtab %} {% endtabs %} This HTTP Pull Listener now uses the data export API to extract events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Incident Management - Incidents ## Overview Get a list of incidents filtered by a list of incident IDs, modification time, or creation time. This includes all incident types and severities, including correlation-generated incidents. * The response is concatenated using AND condition (OR is not supported). * The maximum result set size is >100. * Offset is the zero-based number of incidents from the start of the result set. ## Configuration ### Parameters **Name** - domain **Value** - `CortexXdrDomain` ### Secrets * `CortexXDRAuthorization` will reference the Cortex XDR Authorization token. * `CortexXDRAuthId` will reference the [Cortex XDR Authorization ID](https://docs-cortex.paloaltonetworks.com/r/Cortex-XDR-Platform-APIs/Get-your-Cortex-XDR-API-key-ID?contentId=GwbuuBI0cJvEhgDAelFiMw).
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Cortex incident Management fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Cortex XDR multi alerts** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "fromTo" limit: 100 request: method: "POST" url: "https://${parameters.CortexXdrDomain}/public_api/v1/incidents/get_incidents" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" - name: Authorization value: "${secrets.CortexXdrAuthorization}" - name: x-xdr-auth-id value: ${secrets.CortexXdrAuthId} bodyType: raw bodyRaw: | { "request_data": { "search_from": ${pagination.from}, "search_to": ${pagination.to}, "filters": [ { "field": "creation_time", "operator": "gte", "value": ${temporalWindow.from}000 }, { "field": "creation_time", "operator": "lte", "value": ${temporalWindow.to}000 } ] } } output: select: ".reply.incidents" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected.
**Authentication Phase** Off #### **Enumeration Phase** Off #### **Collection Phase** * **Pagination Type****\***** -** `fromTo` * **Zero index****\***** -** false * **Limit****\***** -** 100 * **Request** * **Response Type****\***** -** JSON * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.CortexXdrDomain}/public_api/v1/alerts/get_alerts` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Name** - Authorization * **Value** - `${secrets.CortexXdrAuthorization}` * **Name** - x-xdr-auth-id * **Value** - `${secrets.CortexXdrAuthId}` * **Body type****\***** -** raw * **Body content****\*** - `{`\ `"request_data": {`\ `"search_from": ${pagination.from},`\ `"search_to": ${pagination.to},`\ `"filters": [`\ `{`\ `"field": "creation_time",`\ `"operator": "gte",`\ `"value": ${temporalWindow.from}000`\ `},`\ `{`\ `"field": "creation_time",`\ `"operator": "lte",`\ `"value": ${temporalWindow.to}000`\ `}`\ `]`\ `}`\ `}` * **Output** * **Select -** `.reply.alerts` * **Map -** `.` * **Output Mode** - `element`
{% endtab %} {% endtabs %} This HTTP Pull Listener now uses the data export API to extract events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Incident Management - Incidents Extradata ## Overview Get the extradata associated to all the incidents within a time range defined by the time window. * The response is concatenated using AND condition (OR is not supported). * The maximum result set size is >100. * Offset is the zero-based number of incidents from the start of the result set. ## Configuration ### Parameters **Name** - domain **Value** - `CortexXdrDomain` ### Secrets * `CortexXDRAuthorization` will reference the Cortex XDR Authorization token. * `CortexXDRAuthId` will reference the [Cortex XDR Authorization ID](https://docs-cortex.paloaltonetworks.com/r/Cortex-XDR-Platform-APIs/Get-your-Cortex-XDR-API-key-ID?contentId=GwbuuBI0cJvEhgDAelFiMw).
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Cortex incident Management fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Cortex XDR multi alerts** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: false withEnumerationPhase: true enumerationPhase: paginationType: "fromTo" limit: 100 request: responseType: json method: "POST" url: "https://${parameters.CortexXdrDomain}/public_api/v1/incidents/get_incidents" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" - name: Authorization value: "${secrets.CortexXdrAuthorization}" - name: x-xdr-auth-id value: ${secrets.CortexXdrAuthId} bodyType: raw bodyRaw: | { "request_data": { "search_from": ${pagination.from}, "search_to": ${pagination.to}, "filters": [ { "field": "creation_time", "operator": "gte", "value": ${temporalWindow.from}000 }, { "field": "creation_time", "operator": "lte", "value": ${temporalWindow.to}000 } ] } } output: select: '.reply.incidents' map: "." outputMode: element collectionPhase: variables: - source: input name: incident_id expression: ".incident_id" default: "0" paginationType: none request: responseType: json method: "POST" url: "https://${parameters.CortexXdrDomain}/public_api/v1/incidents/get_incident_extra_data" headers: - name: Accept value: "application/json" - name: Content-Type value: "application/json" - name: Authorization value: "${secrets.CortexXdrAuthorization}" - name: x-xdr-auth-id value: ${secrets.CortexXdrAuthId} bodyType: raw bodyRaw: | { "request_data":{ "incident_id":"${inputs.incident_id}", "alerts_limit":100 } } output: select: ".reply" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected.
**Authentication Phase** Off #### **Enumeration Phase** * **Pagination Type****\***** -** `fromTo` * **Zero index****\***** -** false * **Limit****\***** -** 100 * **Request** * **Response Type****\***** -** JSON * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.CortexXdrDomain}/public_api/v1/incidents/get_incidents` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Name** - Authorization * **Value** - `${secrets.CortexXdrAuthorization}` * **Name** - x-xdr-auth-id * **Value** - `${secrets.CortexXdrAuthId}` * **Body type****\***** -** raw * **Body content****\*** - \ `{`\ `"request_data": {`\ `"search_from": ${pagination.from},`\ `"search_to": ${pagination.to},`\ `"filters": [`\ `{`\ `"field": "creation_time",`\ `"operator": "gte",`\ `"value": ${temporalWindow.from}000`\ `},`\ `{`\ `"field": "creation_time",`\ `"operator": "lte",`\ `"value": ${temporalWindow.to}000`\ `}`\ `]`\ `}`\ `}` **Output** * **Select -** `.reply.incidents` * **Map -** `.` * **Output Mode** - `element` #### **Collection Phase** * **Source -** `input` * **Name -** `incident_id` * **Expression -** `.incident_id` * **Format** - `JSON` * **Pagination Type****\***** -** `None` * **Request** * **Response Type****\***** -** JSON * **Method****\***** -** `POST` * **URL****\***** -** `https://${parameters.CortexXdrDomain}/public_api/v1/incidents/get_incident_extra_data` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Name** - Authorization * **Value** - `${secrets.CortexXdrAuthorization}` * **Name** - x-xdr-auth-id * **Value** - `${secrets.CortexXdrAuthId}` * **Body type****\***** -** raw * **Body content****\*** -`{ "request_data":{ "incident_id":"${inputs.incident_id}", "alerts_limit":100 } }` * **Output** * **Select -** `.reply` * **Map -** `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} This HTTP Pull Listener now uses the data export API to extract events. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Prisma Cloud ## Overview Get a list of all audit logs. Retrieves paginated audit logs based on the provided filter criteria. ## Configuration ### Parameters **Name** - Domain **Value** - `PrismaCloudEndpoint` ### Secrets * `PrismaCloudAccessKeyId` corresponds to the [authorization Access Key](https://docs.prismacloud.io/en/enterprise-edition/content-collections/get-started/access-keys) ID number. * `PrismaCloudAccessKeySecret` corresponds to the Access Key itself.
To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**.
{% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required parameters and secrets, you can choose to manually enter the Cortex incident Management fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Cortex XDR API** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: Epoch withAuthentication: true authentication: type: "token" token: request: method: POST url: "${parameters.PrismaCloudEndpoint}/login" headers: - name: Content-Type value: application/json bodyType: raw bodyRaw: | { "username": "${secrets.PrismaCloudAccessKeyId}", "password": "${secrets.PrismaCloudAccessKeySecret}" } responseType: json tokenPath: ".token" authInjection: name: "Authorization" in: "header" prefix: "Bearer " withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".nextPageToken" initialRequest: method: POST url: "${parameters.PrismaCloudEndpoint}/audit/api/v1/log" headers: - name: Accept value: application/json - name: Content-Type value: application/json responseType: json bodyType: raw bodyRaw: | { "timeRange": { "type": "absolute", "value": { "startTime": ${temporalWindow.from}, "endTime": ${temporalWindow.to}, } } } nextRequest: method: POST url: "{parameters.PrismaCloudEndpoint}/audit/api/v1/log" headers: - name: Accept value: application/json - name: Content-Type value: application/json responseType: json bodyType: raw bodyRaw: | { "timeRange": { "type": "absolute", "value": { "startTime": ${temporalWindow.from}, "endTime": ${temporalWindow.to}, } }, "nextPageToken": ${pagination.cursor} } output: select: ".value" map: "." outputMode: element ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected.
**Authentication Phase** * **Type** - `token` * **Request** - * **Method** - `POST` * **URL** - `${parameters.PrismaCloudEndpoint}/login` * **Headers** * **Name -** Content-Type * **Value** - `application/json` * **Body Type -** `raw` * **Body Raw -** `|`\ `{`\ `"username": "${secrets.PrismaCloudAccessKeyId}",`\ `"password": "${secrets.PrismaCloudAccessKeySecret}"`\ `}` * **Response Type -** `json` * **Token Path** - `.token` * **Auth injection** * **Name** - `Authorization` * **In** - `header` * **Prefix** - `Bearer` #### **Enumeration Phase** Off #### **Collection Phase** * **Pagination Type****\***** -** `cursor` * **Cursor****\***** -** `.nextPageToken` * **Cursor Selector -** `.nextPageToken` * **Initial Request** * **Method****\***** -** `POST` * **URL****\***** -** `${parameters.PrismaCloudEndpoint}/audit/api/v1/log` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Response Type****\*** - `json` * **Body type****\***** -** raw * **Body content****\*** - `{`\ `"timeRange": {`\ `"type": "absolute",`\ `"value": {`\ `"startTime": ${temporalWindow.from},`\ `"endTime": ${temporalWindow.to},`\ `}`\ `}`\ `}` * **Next Request** * **Method****\***** -** `POST` * **URL****\***** -** `${parameters.PrismaCloudEndpoint}/audit/api/v1/log` * **Headers** * **Name** - Accept * **Value** - `application/json` * **Name** - Content-Type * **Value** - `application/json` * **Response Type****\*** - `json` * **Body type****\***** -** raw * **Body content****\*** - |\ `{`\ `"timeRange": {`\ `"type": "absolute",`\ `"value": {`\ `"startTime": ${temporalWindow.from},`\ `"endTime": ${temporalWindow.to},`\ `}`\ `},`\ `"nextPageToken": ${pagination.cursor}`\ `}` * **Output** * **Select -** `.value` * **Map -** `.` * **Output Mode** - `element` {% endtab %} {% endtabs %} This HTTP Pull Listener now uses the data export API to extract audit logs. Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from PingIdentity ## Overview Get a list of logs from [PingOne](https://docs.pingidentity.com/pingone/introduction_to_pingone/p1_introduction.html), the cloud-based Identity as a Service (IDaaS) platform developed by Ping Identity. ## Configuration ### Parameters Add the following parameter: * **Name** - `tenantEnvFqdn` * **Value** - Enter your tenant FQDN. ### Secrets You must define these credentials in Onum: * `x-api-key` will reference your Ping Identity API Key. * `x-api-secret` will reference your Ping Identity API Secret. To do it, enter a **Name** for the secret and then click the **Value** field. Click **New secret** to create a new one: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. You can now select the secret you just created in the corresponding field. Click **Add element** to add the other one. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} After entering the required parameters and secrets, you can choose to manually enter the API fields, or simply paste the given YAML: {% tabs %} {% tab title="Config as YAML" %} Toggle this **ON** to enable a free text field where you can paste your **Tenable** YAML. ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: false withEnumerationPhase: false collectionPhase: paginationType: "cursor" cursorSelector: ".pagedResultsCookie" initialRequest: method: "GET" url: "https://${parameters.tenantEnvFqdn}/monitoring/logs" headers: - name: "Accept" value: "application/json" - name: "x-api-key" value: "${secrets.x-api-key}" - name: "x-api-secret" value: "${secrets.x-api-secret}" queryParams: - name: "source" value: "am-everything,idm-access, idm-activity, idm-authentication, idm-config,idm-recon,idm-sync" - name: "beginTime" value: "${temporalWindow.from}" - name: "endTime" value: "${temporalWindow.to}" nextRequest: method: "GET" url: "https://${parameters.tenantEnvFqdn}/monitoring/logs" headers: - name: "Accept" value: "application/json" - name: "x-api-key" value: "${secrets.x-api-key}" - name: "x-api-secret" value: "${secrets.x-api-secret}" queryParams: - name: "source" value: "am-everything,idm-access, idm-activity, idm-authentication, idm-config,idm-recon,idm-sync" - name: "beginTime" value: "${temporalWindow.from}" - name: "endTime" value: "${temporalWindow.to}" - name: "pagedResultsCookie" value: "${pagination.cursor}" output: select: ".result" map: "." outputMode: "element" ``` {% endtab %} {% tab title="Manually configure" %} **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration** - `5m` * **Offset** - `5m` * **Format** - `RFC3339` **Collection Phase** * **Pagination Type****\*** - `Cursor` * **Cursor Selector****\*** - `.pagedResultsCookie` **Initial Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.tenantEnvFqdn}/monitoring/logs` **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `x-api-key` * **Value** - `${secrets.x-api-key}` * **Name** - `x-api-secret` * **Value** - `${secrets.x-api-secret}` **Query Params** * **Name** - `source` * **Value** - `am-everything,idm-access, idm-activity, idm-authentication, idm-config,idm-recon,idm-sync` * **Name** - `beginTime` * **Value** - `${temporalWindow.from}` * **Name** - `endTime` * **Value** - `${temporalWindow.to}` **Next Request** * **Response Type****\*** - `JSON` * **Method****\*** - `GET` * **URL****\*** - `https://${parameters.tenantEnvFqdn}/monitoring/logs` **Headers** * **Name** - `Accept` * **Value** - `application/json` * **Name** - `x-api-key` * **Value** - `${secrets.x-api-key}` * **Name** - `x-api-secret` * **Value** - `${secrets.x-api-secret}` **Query Params** * **Name** - `source` * **Value** - `am-everything,idm-access, idm-activity, idm-authentication, idm-config,idm-recon,idm-sync` * **Name** - `beginTime` * **Value** - `${temporalWindow.from}` * **Name** - `endTime` * **Value** - `${temporalWindow.to}` * **Name** - `pagedResultsCookie` * **Value** - `${pagination.cursor}` **Output** * **Select****\*** - `.result` * **Map** - `.` * **Output Mode****\*** - `element` {% endtab %} {% endtabs %} Click **Create labels** to move on to the next step and define the required [Labels](/the-workspace/listeners/labels) if needed. # Collect data from Proofpoint TAP Where the vendor is **Proofpoint TAP,** it's product is **SIEM API.** For SIEM API, right now we have the following product types/endpoints: * [Clicks blocked](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-proofpoint-tap/clicks-blocked) * [Clicks permitted](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-proofpoint-tap/clicks-permitted) * [Messages blocked](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-proofpoint-tap/messages-blocked) * [Messages delivered](/the-workspace/listeners/listener-integrations/pull-data-from-http-endpoints/collect-data-from-proofpoint-tap/messages-delivered) Inside each of those endpoints we have the YAML file to configure. See below for **All** endpoints. ## Overview Proofpoint TAP SIEM API allows integration with SIEM solutions by giving administrators the ability to periodically download detailed information about several types of TAP events in a SIEM-compatible, vendor-neutral format. Currently, the following event types are exposed: * Blocked or permitted clicks to threats recognized by URL Defense * Blocked or delivered messages that contain threats recognized by URL Defense or Attachment Defense We have those different endpoints that will return an object with a key, that will contain an array of objects. In each of those specific templates we should define that key name. If we use the `/siem/all` endpoint we will get all the options included, each of those, with it's own array. ## Configuration ### Parameters No parameters needed ### Secrets These secrets will correspond to the username and password fields in the authentication phase. * **Username** (Value: `pp_sp`) `${secrets.pp_sp}` * **Password** (Value: `pp_secret) ${secrets.pp_secret}` To add a Secret, open the **Secret** fields and click **New secret**: * Give the secret a **Name**. * Turn off the **Expiration date** option. * Click **Add new value** and paste the secret corresponding to the value. * Click **Save**. {% hint style="info" %} Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management). {% endhint %} You can now select the secret you just created in the corresponding fields. After entering the required secrets, you can choose to manually enter the fields, or simply paste the desired YAML. ### Configure as YAML ```yaml withTemporalWindow: true temporalWindow: duration: 5m offset: 5m tz: UTC format: RFC3339 withAuthentication: true authentication: type: basic basic: username: "${secrets.pp_sp}" password: "${secrets.pp_secret}" withEnumerationPhase: false collectionPhase: paginationType: none request: method: GET url: https://tap-api-v2.proofpoint.com/v2/siem/all headers: - name: Accept value: text/plain queryParams: - name: format value: json - name: interval value: ${temporalWindow.from}/${temporalWindow.to} output: select: ".clicksBlocked" filter: "." map: "." outputMode: element retry: statusCodes: [429, 500, 502, 503, 504] type: fixed fixed: interval: 5m ``` ### **Manually Configure** #### **Temporal Window** Toggle **ON** to add a temporal window for events. This repeatedly shifts the time window over which data is collected. * **Duration -** 5 minutes (`5m`) as default, adjust based on your needs. * **Offset -** `5m` * **Format** - `RFC3339` #### **Authentication Phase** Toggle **ON** to configure the authentication parameters * **Type****\***** -** `basic` * **Username****\***** -** `${secrets.pp_sp}` * **Password****\***** -** `${secrets.pp_secret}` #### **Retry** * **Retry type -** `Fixed` * **Interval** **-** `5m` #### **Enumeration Phase** **OFF** #### **Collection Phase** * **Pagination Type****\***** -** `none` * **Request** * **Method****\***** -** `GET` * **URL****\***** -** `https://tap-api-v2.proofpoint.com/v2/siem/all` * **Headers** * **Name -** `interval` * **Format** - `${temporalWindow.from}/${temporalWindow.to}` * **Name** - `format` * **Value -** `json` * **Output** * **Select -** `.clicksBlocked` * **Map -** `.` * **Filter -** `.` * **Output Mode** - `element`
--- [Next Page](/llms-full.txt/1)