WebsiteBlogLogin

HTTP Pull

Most recent version: v0.0.1

See the changelog of this Listener type here.

Note that this Listener is only available in certain Tenants. Get in touch with us if you don't see it and want to access it.

Overview

Onum supports integration with HTTP Pull. Select HTTP Pull from the list of Listener types and click Configuration to start.

Configuration

Now you need to specify how and where to collect the data and how to establish an HTTP connection.

Metadata

Enter the basic information for the new Listener.

Parameter
Description

Name*

Enter a name for the new Listener.

Description

Optionally, enter a description for the Listener.

Tags

Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

Cloud Listeners

Note that you won't see the Socket and TLS configuration sections in the creation form if you're defining this Listener in a Cloud instance, as these are already provided by Onum. Learn more about Cloud Listeners in this article.

Parameters

Parameter
Description

Name

Enter the name of the parameter to search for in the YAML below, used later as ${parameters.name} e.g. ${parameters.domain}.

Value

Enter the value or variable to fill in when the given parameter name has been found, e.g. “domain.com”. With the name set as “domain” and the value set as “domain.com” , the expression to execute on the YAML would be: ${parameters.domain}., which will be automatically replaced by the variable. Add as many name and value pairs as required.

Secrets

Parameter
Description

Name

Enter the name of the parameter to search for in the YAML below, used later as ${secrets.name}.

Value

Select the Secret containing the connection credentials if you have added them previously, or select New Secret to add it. This will add this value as the variable when the field name is found in the YAML. Add as many as required.

Config as YAML

Toggle ON to configure the HTTP as a YAML and paste it here.

The system supports interpolated variables throughout the HTTP request building process using the syntax:

${prefix.name} Each building block may:

  • Use variables depending on its role (e.g., parameters, secrets, pagination state).

  • Expose variables for later phases (e.g., pagination counters, temporal window bounds).

Not all variable types are available in every phase. Each block has access to a specific subset of variables.

Variables can be defined in the configuration or generated dynamically during execution. Each variable has a prefix that determines its source and scope.

Supported Prefixes:

Parameters
Secrets
temporalWindow
Pagination
Inputs

User-defined values configured manually.

Available in all phases.

Sensitive values such as credentials or tokens.

Available in all phases

Automatically generated from the Temporal Window block.

Available in Enumeration and Collection phases.

Values produced by the pagination mechanism (e.g., offset, cursor).

Available in Enumeration and Collection phases.

Values derived from the output of the Enumeration phase.

Available only in the Collection phase.

Temporal Window

Toggle ON to add a temporal window for events. This repeatedly shifts the time window over which data is collected.

Parameter
Description

Duration*

Add the duration in milliseconds that the window will remain open for.

Offset*

How far behind the current time the window should end (e.g., 5m behind "now").

Time Zone*

This value is usually automatically set to your current time zone. If not, select it here.

Format*

Choose between Epoch or RCF3339 for the timestamp format.

Authentication

If your connection requires authentication, enter the credentials here.

Parameter
Description

Authentication Type*

Choose the authentication type and enter the details.

Authentication credentials

The options provided will vary depending on the type chosen to authenticate your API. This is the type you have selected in the API end, so it can recognize the request.

Choose between the options below.

Basic

  • Username* - the user sending the request.

  • Password* - the password eg: ${secrets.password}

API Key

Enter the following:

  • API Key - API keys are usually stored in developer portals, cloud dashboards, or authentication settings. Set the a secret, eg: ${secrets.api_key}

  • Auth injection:

    • In* - Enter the incoming format of the API: Header or Query.

    • Name* - A label assigned to the API key for identification. You can find it depending on where the API key was created.

    • Prefix - Enter a connection prefix if required.

    • Suffix - Enter a connection suffix if required.

Token

Token Retrieve Based Authentication

  • Request -

    • Method* - Choose between GET or POST

    • URL*- Enter the URL to send the request to.

  • Headers - Add as many headers as required.

    • Name

    • Value

  • Query Params - Add as many query parameters as required.

    • Name

    • Value

  • Token Path* - Enter your Token Path for used to retrieve an authentication token.

  • Auth injection:

    • In* - Enter the incoming format of the API: Header or Query.

    • Name* - A label assigned to the API key for identification. You can find it depending on where the API key was created.

    • Prefix - Enter a connection prefix if required.

    • Suffix - Enter a connection suffix if required.

Enumeration Phase

Identify the available endpoints, methods, parameters, and resources exposed by the API. This performs initial data discovery to feed the collection phase and makes the results available to the Collection Phase via variable interpolation (inputs.*).

Can use:

  • ${parameters.xxx}

  • ${secrets.xxx}

  • ${temporalWindow.xxx} (if configured)

  • ${pagination.xxx*} Pagination variables

Parameter
Description

Pagination Type*

Select one from the drop-down. Pagination type is the method used to split and deliver large datasets in smaller, manageable parts (pages), and how those pages can be navigated during discovery.

Each pagination method manages its own state and exposes specific variables that can be interpolated in request definitions (e.g., URL, headers, query params, body).

None

  • Description: No pagination; only a single request is issued.

  • Exposed Variables: None

PageNumber/PageSize

  • Description: Pages are indexed using a page number and fixed size.

  • Configuration:

    • pageSize: page size

  • Exposed Variables:

    • ${pagination.pageNumber}

    • ${pagination.pageSize}

Offset/Limit

  • Description: Uses offset and limit to fetch pages of data.

  • Configuration:

    • Limit: max quantity of records per request

  • Exposed Variables:

    • ${pagination.offset}

    • ${pagination.limit}

From/To

  • Description: Performs pagination by increasing a window using from and to values.

  • Configuration: limit: max quantity of records per request

  • Exposed Variables:

    • ${pagination.from}

    • ${pagination.to}

Web Linking (RFC 5988)

  • Description: Parses the Link header to find the rel="next" URL.

  • Exposed Variables: None

Next Link at Response Header

  • Description: Follows a link found in a response header.

  • Configuration:

    • headerName: header name that contains the next link

  • Exposed Variables: None

Next Link at Response Body

  • Description: Follows a link found in the response body.

  • Configuration:

    • nextLinkSelector: path to next link sent in response payload

  • Exposed Variables: None

Cursor

  • Description: Extracts a cursor value from each response to request the next page.

  • Configuration:

    • cursorSelector: path to the cursor sent in response payload

  • Exposed Variables:

    • ${pagination.cursor}

Output

Parameter
Description

Select*

If your connection does not require authentication, leave as None. Otherwise, choose the authentication type and enter the details. A JSON selector expression to pick a part of the response e.g. '.data'.

Filter

A JSON expression to filter the selected elements. Example: '.films | index("Tangled")'.

Map

A JSON expression to transform each selected element into a new event. Example: '{characterName: .name}'.

Output Mode*

Choose between

  • Element: emits each transformed element individually as an event.

  • Collection: emits all transformed items as a single array/collection as an event.

Collection Phase*

The collection phase is mandatory. This is where the final data retrieval happens (either directly or using IDs/resources generated by an enumeration phase).

The collection phase involves gathering actual data from an API after the enumeration phase has mapped out endpoints, parameters, and authentication methods. It supports dynamic variable resolution via the variable resolver and can use data exported from the Enumeration Phase, such as:

  • ${parameters.xxx}

  • ${secrets.xxx}

  • ${temporalWindow.xxx}

  • ${inputs.xxx} (from Enumeration Phase)

  • ${pagination.xxx}*

Inputs

In collection phases, you can define variables to be used elsewhere in the configuration (for example, in URLs, query parameters, or request bodies). Each variable definition has the following fields:

Parameter
Description

Name

The variable name (used later as ${inputs.name} in the configuration).

Source

Usually "input", indicating the value comes from the enumeration phase’s output.

Expression

A JSON expression applied to the input to extract or transform the needed value.

Format

Controls how the variable is converted to a string (see Variable Formatting below). Eg: json.

Parameter
Description

Pagination Type*

Choose how the API organizes and delivers large sets of data across multiple pages—and how that affects the process of systematically collecting or extracting all available records.

Output

Parameter
Description

Select*

If your connection does not require authentication, leave as None. Otherwise, choose the authentication type and enter the details. A JSON selector expression to pick a part of the response e.g. '.data'.

Filter

A JSON expression to filter the selected elements. Example: '.films | index("Tangled")'.

Map

A JSON expression to transform each selected element into a new event. Example: '{characterName: .name}'.

Output Mode*

Choose between

  • Element: emits each transformed element individually as an event.

  • Collection: emits all transformed items as a single array/collection as an event.

Click Create labels to move on to the next step and define the required Labels if needed.

Ports

The HTTP Pull sink has two output ports:

  • Default port - Events are sent through this port if no error occurs while processing them.

  • Error port - Events are sent through this port if an error occurs while processing them.

The error message is provided in a free-text format and may change over time. Please consider this if performing any post-processing based on the message content.

Integrations

HTTP Puller integrations are parametrized and organized by vendor, categorized by product/API.

Inside each endpoint, you will find a yaml configuration. This configuration is used in the Onum HTTP Puller action in order to start feeding that information into the platform. Check the articles under this section to learn more.

PreviousHTTPNextPalo Alto

Last updated

Was this helpful?