Pipelines
A Pipeline is Onum's way of streamlining your data
Last updated
Was this helpful?
A Pipeline is Onum's way of streamlining your data
Last updated
Was this helpful?
Use Pipelines to transform your events and build a data flow linking Actions from Listeners and to Data sinks.
Select the Pipelines tab at the left menu to visualize all your Pipelines in one place. Here's what you will find all the actions you can perform in this area:
The graph at the top plots the data volume going through your Pipelines. The purple line graph represents the events in, and the blue one represents the events going out. Use the buttons above the graph to switch between Events/Bytes, and the Frequency slider bar to choose how frequently you want to plot the events/bytes in the chart.
At the bottom, you will find a list of all the Pipelines in your tenant. You can switch between the Cards view, which shows each Pipeline in a card, and the Table view, which displays Pipelines listed in a table. Learn more about the cards and table views in this article.
There are various ways to narrow down what you see in this view, both the Pipeline list and the informative graphs. To do it, use the options at the top of this view:
Add filters to narrow down the Pipelines you see in the list. Click the + Add filter button and select the required filter type(s). You can filter by:
Name: Select a Condition (Contains, Equals, or Matches) and a Value to filter Pipelines by their names.
Status: Choose the status(es) you want to filter by: Draft, Running, and/or Stopped. You'll only see Pipelines with the selected status(es).
Created by: Filter for the creator of the Pipeline in the window that appears.
Updated by: Filter for users to see the Pipeline they last updated.
The filters applied will appear as tags at the top of the view.
Note that you can only add one filter of each type.
If you wish to see data for a specific time period, this is the place to click. Go to Selecting a Time Range to dive into the specifics of how the time range works.
You can choose to view only those Pipelines that have been assigned the desired tags. You can create these tags in the Pipeline settings or from the cards view. Press the Enter key to confirm the tag, then Save.
To filter by tags, click the + Tags button and select the required tag(s).
Below the filters, you will see 3 metrics informing you about various components in your Pipelines.
Note that these metrics are affected by the time range selected.
View the events per second (EPS) ingested by all Listeners in your Pipelines for the selected time range, as well as the difference in percentage compared to the previous lapse.
View the events per second (EPS) sent by all Data Sinks in your Pipelines for the selected time range, as well as the difference in percentage compared to the previous.
See the overall data volume processed by all Pipelines for the selected time range, and the difference in percentage with the previous.
Select between In and Out to see the volume received or sent by your Pipelines for the selected time range. The line graph represents the Events and the bar graph represents Bytes.
Hover over a point on the chart to show a tooltip containing the Events and Bytes in/out for the selected time, as well as a percentage of how much increase/decrease has occurred since the previous lapse of time since the one currently selected.
You can also analyze a different time range directly on the graph. To do it, click a starting date in the map and drag the frame that appears until the required ending date. The time range above will be also updated.
At the bottom, you have a list of all the Pipelines in your tenant.
Use the Group by drop-down menu at the right area to select a criterion to organize your Pipelines in different groups (Status or None). You can also use the search icon to look for specific Pipelines by name.
Use the bottoms at the left of this area to display the Pipelines as Cards or listed in a Table:
In this view, Pipelines are displayed as cards that display useful information. Click a card to open the Pipeline detail view, or double-click it to access it.
This is the information you can check on each card:
The percentage at the top left corner indicates the amount of data that goes out of the Pipeline compared to the total incoming events, so you can check how data is optimized at a glance. Hover over it to see the in/out data in bytes and the estimation over the next 24 hours.
You can also see the status of the Pipeline (Running, Draft, or Stopped).
Next to the status, you can check the Pipeline current version.
Click the Add tag button to define tags for the Pipeline. To assign a new tag, simply type the name you wish to assign, make sure to press Enter, and then select the Save button. If the Pipeline has tags defined already, you'll see the number of tags next to the tag icon.
Click the ellipses in the right-hand corner of the card to reveal the options to Edit, Copy ID, or Remove it.
In this view, Pipelines are displayed in a table, where each row represents a Pipeline. Click a row to open the Pipeline detail view, or double-click it to access it.
Click the cog icon at the top left corner to rearrange the column order, hide columns, or pin them. You can click Reset to recover the default configuration.
Click a Pipeline to open its settings in the right-hand pane. Here you can see Pipeline versions and edit the Pipeline. Click the ellipses in the top right to Copy ID or Duplicate / Remove it.
The details pane is split into three tabs showing the Pipeline at different statuses:
Running
This is the main tab, where you can see details of the Pipeline versions that are currently running.
Select the drop-down next to the Pipeline version name to see which clusters the Pipeline is currently running in.
Draft
Check the details of the draft versions of your Pipeline.
Stopped
Check the details of the version of your Pipeline that are currently stopped.
Once you have located the Pipeline to work with, click Edit Pipeline to open it.
If you wish to use a Pipeline just like the one you are currently working on, click the ellipses in the Card or Table view and select Duplicate, or from the Configuration pane.
Depending on your permissions, you can create a new Pipeline from this view. There are several ways to create a new Pipeline:
From the Pipelines view
From the Home page
This will open the new Pipeline, ready to be built.
Give your Pipeline a name and add optional Tags to identify it. You can also assign a Version in the top-right.
Keep reading to learn how to build a Pipeline from this view.
See Building a Pipeline to learn step by step.
