Using HTTP Pull as a collector

HTTP Puller integrations are parametrized and organized by vendor, categorized by product/API.

Inside each endpoint, you will find a yaml configuration. This configuration is used in the Onum HTTP Puller action in order to start feeding that information into the platform. Check the articles under this section to learn more about configurations specific to each vendor.

Desconstructing a YAML

Here we will learn what each parameter of the YAML means, and how they correspond to the settings in the HTTP Pull Listener.

The YAML is used for pulling alerts via an API and typically uses

• A Temporal Window to enable the use of a time-based query window for filtering results.

• Authentication using a token to authenticate the connection.

• The first phase (Enumeration) enables an initial listing phase to get identifiers (e.g., alert IDs), paginating through the results.

• The second phase (Collection) then fetches full alert details using the alert IDs from the enumeration phase.

• Standard JSON response mapping is used to output the results.

Let´s take a closer look at each phase below.

Temporal window

A temporal window is a defined time range used to filter or limit data retrieval in queries or API requests. It specifies the start and end time for the data you want to collect or analyze. This YAML uses a temporal window of 5 minutes, in RFC3339 format, with an offset of 0, in UTC timezone.

Example

withTemporalWindow: true
temporalWindow:
  duration: 5m
  offset: 0
  tz: UTC
  format: RFC3339

In Onum, toggle ON the Temporal Window selector and enter the information in the corresponding fields

• Duration* - 5m

• Offset* - 0s

• TZ* - this will set automatically according to your current timezone.

• Format* - RFC3339

Authentication phase

If your connection requires authentication, enter the credentials here.

Authentication credentials

The options provided will vary depending on the type chosen to authenticate your API. This is the type you have selected in the API end, so it can recognize the request.

Choose between the options below.

Example

withAuthentication: true
authentication:
  type: token
  token:
    request:
      method: POST
      url: ${parameters.domain}/oauth2/token
      headers:
        - name: Content-Type
          value: application/x-www-form-urlencoded
      bodyType: urlEncoded
      bodyParams:
        - name: grant_type
          value: client_credentials
        - name: client_id
          value: '${secrets.client_id}'
        - name: client_secret
          value: '${secrets.client_secret}'
    tokenPath: ".access_token"
    authInjection:
      in: header
      name: Authorization
      prefix: 'Bearer '
      suffix: ''

Toggle ON the Authentication option.

• Type - Token. Token authentication is a method of authenticating API requests by using a secure token, usually passed in an HTTP header.

• Request

  • method - POSTSends a POST request to obtain an access token.

  • url - ${parameters.domain}/oauth2/tokenThe OAuth token endpoint. ${parameters.domain} is a placeholder for value entered in the Parameters section.

  • headers - these headers are key-value pairs that provide additional information to the server when making a request.

    • name - Content-Type

    • value - application/x-www-form-urlencodedIndicates that the request body is formatted as URL-encoded key-value pairs (standard for OAuth token requests).

  • Body type -urlEncoded Specifies the request body format is URL-encoded (like key=value&key2=value2).

    • Body params

      • name - grant_type Required by OAuth 2.0 to specify the type of grant being requested.

      • value - client_credentials Used for server-to-server authentication without a user.

      • name - client_ID

      • value - ${secrets.client_id}this is a dynamic variable pulled from the value entered in the Secrets setting.

      • name - client_secret

      • value - ${secrets.client_secret} this is a dynamic variable pulled from the value entered in the Secrets setting.

  • Token path - Extracts the access token from the JSON response of an authentication request. It's a JSONPath-like expression used to locate the token in the response body.

• Auth injection - This part defines how and where to inject the authentication token (typically an access token) into the requests after it has been retrieved, for example, from an OAuth token endpoint.

  • in -headerThe token should be injected into the HTTP header of the request.This is the most common method for passing authentication tokens.

  • Name -AuthorizationThe name of the header that will contain the token. Most APIs expect this to be Authorization.

  • prefix - The text added before the token value.Bearer is the standard prefix for OAuth 2.0 tokens.

  • suffix -''Text added after the token value. In this case, it's empty — nothing is appended.

Enumeration phase

The enumeration phase is an optional step in data collection or API integration workflows, where the system first retrieves a list of available items (IDs, resource names, keys, etc.) before fetching detailed data about each one.

Identify the available endpoints, methods, parameters, and resources exposed by the API. This performs initial data discovery to feed the collection phase and makes the results available to the Collection Phase via variable interpolation (inputs.*).

Can use:

• ${parameters.xxx}

• ${secrets.xxx}

• ${temporalWindow.xxx} (if configured)

• ${pagination.xxx} Pagination variables

Output

Example

enumerationPhase:
  paginationType: offsetLimit
  limit: 100
  request:
    responseType: json
    method: GET
    url: ${parameters.domain}/alerts/queries/alerts/v2
    queryParams:
      - name: offset
        value: ${pagination.offset}
      - name: limit
        value: ${pagination.limit}
      - name: filter
        value: created_timestamp:>'${temporalWindow.from}'+created_timestamp:<'${temporalWindow.to}'
  output:
    select: ".resources"
    map: "."
    outputMode: collection

• Pagination type - offset/LimitUses classic pagination with offset and limit to page through results, fetching data in batches (pages) — limit determines page size, offset determines where to start.

• Limit - Retrieves up to 100 records per request. This value is used in the limit query parameter to control batch size.

• Request - Describes the API request that will be sent during enumeration.

  • Response type - Specifies the expected response format. Here, the system expects a JSON response.

  • Method - The HTTP method to use for this request. GET is used to retrieve data from the server.

  • URL - ${parameters.domain} is a placeholder variable that will be replaced by the domain value you entered in the Parameters section.

• Query params - These are query string parameters appended to the URL.

  • ${pagination.offset}controls where to start in the dataset. Used for pagination.

  • ${pagination.limit}replaced with the limit value you entered for number of records to retrieve per request (100).

  • Filters data to only return alerts created within a specific time window. ${temporalWindow.from} and ${temporalWindow.to} are dynamically filled in with RFC3339 or epoch timestamps, depending what you have configured.

• output - Describes how to extract and interpret the results from the JSON response.

  • select - .resourcesLooks for a field named resources in the response JSON. This is where the array of items lives.

  • map - .Each item under .resources is returned as-is. No transformation or remapping.

  • outputMode - collectionThe result is treated as a collection (array) of individual items. Used when you expect multiple items and want to pass them along for further processing.

Collection phase

The collection phase in an HTTP Puller is the part of the process where the system actively pulls or retrieves data from an external API using HTTP requests.

The collection phase is mandatory. This is where the final data retrieval happens (either directly or using IDs/resources generated by an enumeration phase).

The collection phase involves gathering actual data from an API after the enumeration phase has mapped out endpoints, parameters, and authentication methods. It supports dynamic variable resolution via the variable resolver and can use data exported from the Enumeration Phase, such as:

• ${parameters.xxx}

• ${secrets.xxx}

• ${temporalWindow.xxx}

• ${inputs.xxx} (from Enumeration Phase)

• ${pagination.xxx}*

Inputs

In collection phases, you can define variables to be used elsewhere in the configuration (for example, in URLs, query parameters, or request bodies). Each variable definition has the following fields:

Output

Example

Let´s say you have the following SIEM Integration events from Sophos.

collectionPhase:
  paginationType: cursor
  cursorSelector: ".next_cursor"
  initialRequest:
    method: GET
    url: "${inputs.dataRegionURL}/siem/v1/events"
    headers:
      - name: Accept
        value: application/json
      - name: Accept-Encoding
        value: gzip, deflate
      - name: X-Tenant-ID
        value: "${inputs.tenantId}"
    queryParams:
      - name: from_date
        value: "${temporalWindow.from}"
    bodyParams: []
  nextRequest:
    method: GET
    url: "${inputs.dataRegionURL}/siem/v1/events"
    headers:
      - name: Accept
        value: application/json
    queryParams:
      - name: cursor
        value: "${pagination.cursor}"
    bodyParams: []
  output:
    select: ".result"
    filter: "."
    map: "."
    outputMode: element

• Pagination type - cursor. If you select the cursor type, you retrieve the data in chunks (pages) using a cursor token, which points to the position in the dataset where the next page of results should start.

  • Cursor selector - The cursor selector tells the HTTP Puller where to find the cursor value in the API response so it can be saved and used in the next request e.g. .next_cursor

• Initial request - We fetch the first set of results, the response including the cursor token (e.g. timestamp or ID).

  • method - GET to fetch the results.

  • url - The URL is composed of various elements:

    • https://${inputs.dataRegionURL}- these variables are taken from the values you entered in the Parameters section of the HTTP Pull settings.

    • /siem/v1/ -API base path — indicates you're calling version 1 of the SIEM API.

    • events- indicates the specific endpoint being accessed. events general category of the API (event-related).

• headers - these headers are key-value pairs that provide additional information to the server when making a request.

  • name - Accept

  • value - application/json tells the server that the client expects the response to be in JSON format, a standard HTTP header used for content negotiation.

• Next request - send the cursor token back to the server using a parameter (e.g., ?cursor=abc123) to get the next page of results. The server returns the next chunk of data and a new cursor.

  Repeat until no more data or the server returns a has_more: false flag.method

• Output

  • select - .result Selects the part of the response to extract. This is a JSONPath-like expression that tells the puller where to find the list or array of items in the response.

  • map - . Maps each selected item as-is, keeping each object unchanged. It passes through each item without transforming it. If you needed to restructure or extract specific fields from each item, you would replace . with a field mapping (e.g., .id, { "id": .id, "name": .username }, etc.).

  • output mode - element Controls the output format. Each item from the select result will be emitted individually using element. This is useful for event stream processing, where each object (e.g., an alert or event) is treated as a separate record. Other possible values (depending on the platform) might include array (emit as a batch) or raw (emit as-is).

Examples

1. Basic GET Puller

Here's a simple example of using the HTTP Puller collector with parameters for a basic GET request. No authentication, no pagination, just pulling JSON data from an API endpoint. Keep Config as YAML, Temporal window, Authentication and Enumeration phase as OFF

• Collection phase -

  • Pagination type - none Indicates that you only need one request to retrieve all data at once.

  • Request

    • Response type - jsonTells the puller to expect a JSON response.

    • Method: GET Performs a basic HTTP GET request.

    • URL: Constructed from the parameters.domain and parameters.path.

  • Headers: Set standard headers and include the API key.

  • Output:

    • Select:.logs Tells the system where to find the list of log entries in the response.

    • Output mode: element each object inside .logs will be extracted as a separate output element e.g.

      Copy

      {
        "logs": [
          { "timestamp": "2024-12-01T12:00:00Z", "event": "user_login" },
          { "timestamp": "2024-12-01T12:05:00Z", "event": "file_upload" }
        ]
      }

Integrations

Read the following articles to learn how to use the HTTP Pull listener to integrate data with other providers.