HTTP Pull
Most recent version: v0.0.1
Last updated
Was this helpful?
Most recent version: v0.0.1
Last updated
Was this helpful?
Note that this Listener is only available in certain Tenants. Get in touch with us if you don't see it and want to access it.
Onum supports integration with HTTP Pull. Select HTTP Pull from the list of Listener types and click Configuration to start.
Now you need to specify how and where to collect the data and how to establish an HTTP connection.
Enter the basic information for the new Listener.
Name*
Enter a name for the new Listener.
Description
Optionally, enter a description for the Listener.
Tags
Add tags to easily identify your Listener. Hit the Enter key after you define each tag.
Cloud Listeners
Note that you won't see the Socket and TLS configuration sections in the creation form if you're defining this Listener in a Cloud instance, as these are already provided by Onum. Learn more about Cloud Listeners in this article.
Name
Enter the name of the parameter to search for in the YAML below, used later as ${parameters.name} e.g. ${parameters.domain}.
Value
Enter the value or variable to fill in when the given parameter name has been found, e.g. “domain.com”. With the name set as “domain” and the value set as “domain.com” , the expression to execute on the YAML would be: ${parameters.domain}., which will be automatically replaced by the variable. Add as many name and value pairs as required.
Name
Enter the name of the parameter to search for in the YAML below, used later as ${secrets.name}.
Value
Config as YAML
Toggle ON to configure the HTTP as a YAML and paste it here.
The system supports interpolated variables throughout the HTTP request building process using the syntax:
${prefix.name} Each building block may:
Use variables depending on its role (e.g., parameters, secrets, pagination state).
Expose variables for later phases (e.g., pagination counters, temporal window bounds).
Not all variable types are available in every phase. Each block has access to a specific subset of variables.
Variables can be defined in the configuration or generated dynamically during execution. Each variable has a prefix that determines its source and scope.
Supported Prefixes:
User-defined values configured manually.
Available in all phases.
Sensitive values such as credentials or tokens.
Available in all phases
Automatically generated from the Temporal Window block.
Available in Enumeration and Collection phases.
Values produced by the pagination mechanism (e.g., offset, cursor).
Available in Enumeration and Collection phases.
Values derived from the output of the Enumeration phase.
Available only in the Collection phase.
Temporal Window
Toggle ON to add a temporal window for events. This repeatedly shifts the time window over which data is collected.
Duration*
Add the duration in milliseconds that the window will remain open for.
Offset*
How far behind the current time the window should end (e.g., 5m behind "now").
Time Zone*
This value is usually automatically set to your current time zone. If not, select it here.
Format*
Choose between Epoch or RCF3339 for the timestamp format.
Authentication
If your connection requires authentication, enter the credentials here.
Authentication Type*
Choose the authentication type and enter the details.
The options provided will vary depending on the type chosen to authenticate your API. This is the type you have selected in the API end, so it can recognize the request.
Choose between the options below.
Identify the available endpoints, methods, parameters, and resources exposed by the API. This performs initial data discovery to feed the collection phase and makes the results available to the Collection Phase via variable interpolation (inputs.*).
Can use:
${parameters.xxx}
${secrets.xxx}
${temporalWindow.xxx} (if configured)
${pagination.xxx*} Pagination variables
Pagination Type*
Select one from the drop-down. Pagination type is the method used to split and deliver large datasets in smaller, manageable parts (pages), and how those pages can be navigated during discovery.
Each pagination method manages its own state and exposes specific variables that can be interpolated in request definitions (e.g., URL, headers, query params, body).
None
Description: No pagination; only a single request is issued.
Exposed Variables: None
PageNumber/PageSize
Description: Pages are indexed using a page number and fixed size.
Configuration:
pageSize: page size
Exposed Variables:
${pagination.pageNumber}
${pagination.pageSize}
Offset/Limit
Description: Uses offset and limit to fetch pages of data.
Configuration:
Limit: max quantity of records per request
Exposed Variables:
${pagination.offset}
${pagination.limit}
From/To
Description: Performs pagination by increasing a window using from and to values.
Configuration: limit: max quantity of records per request
Exposed Variables:
${pagination.from}
${pagination.to}
Web Linking (RFC 5988)
Description: Parses the Link header to find the rel="next" URL.
Exposed Variables: None
Next Link at Response Header
Description: Follows a link found in a response header.
Configuration:
headerName: header name that contains the next link
Exposed Variables: None
Next Link at Response Body
Description: Follows a link found in the response body.
Configuration:
nextLinkSelector: path to next link sent in response payload
Exposed Variables: None
Cursor
Description: Extracts a cursor value from each response to request the next page.
Configuration:
cursorSelector: path to the cursor sent in response payload
Exposed Variables:
${pagination.cursor}
Output
Select*
If your connection does not require authentication, leave as None. Otherwise, choose the authentication type and enter the details. A JSON selector expression to pick a part of the response e.g. '.data'.
Filter
A JSON expression to filter the selected elements. Example: '.films | index("Tangled")'.
Map
A JSON expression to transform each selected element into a new event.
Example: '{characterName: .name}'.
Output Mode*
Choose between
Element: emits each transformed element individually as an event.
Collection: emits all transformed items as a single array/collection as an event.
The collection phase is mandatory. This is where the final data retrieval happens (either directly or using IDs/resources generated by an enumeration phase).
The collection phase involves gathering actual data from an API after the enumeration phase has mapped out endpoints, parameters, and authentication methods. It supports dynamic variable resolution via the variable resolver and can use data exported from the Enumeration Phase, such as:
${parameters.xxx}
${secrets.xxx}
${temporalWindow.xxx}
${inputs.xxx} (from Enumeration Phase)
${pagination.xxx}*
Inputs
In collection phases, you can define variables to be used elsewhere in the configuration (for example, in URLs, query parameters, or request bodies). Each variable definition has the following fields:
Name
The variable name (used later as ${inputs.name} in the configuration).
Source
Usually "input", indicating the value comes from the enumeration phase’s output.
Expression
A JSON expression applied to the input to extract or transform the needed value.
Format
Controls how the variable is converted to a string (see Variable Formatting below). Eg: json.
Pagination Type*
Choose how the API organizes and delivers large sets of data across multiple pages—and how that affects the process of systematically collecting or extracting all available records.
Select*
If your connection does not require authentication, leave as None. Otherwise, choose the authentication type and enter the details. A JSON selector expression to pick a part of the response e.g. '.data'.
Filter
A JSON expression to filter the selected elements. Example: '.films | index("Tangled")'.
Map
A JSON expression to transform each selected element into a new event.
Example: '{characterName: .name}'.
Output Mode*
Choose between
Element: emits each transformed element individually as an event.
Collection: emits all transformed items as a single array/collection as an event.
Click Create labels to move on to the next step and define the required Labels if needed.
The HTTP Pull Listener supports the following integrations:
Select the containing the connection credentials if you have added them previously, or select New Secret to add it. This will add this value as the variable when the field name is found in the YAML. Add as many as required.
