LogoLogo
WebsiteBlogLogin
  • Onum Docs
  • Use Cases
  • Videos
  • Release Notes
  • Welcome
  • Getting Started
    • About Onum
    • Architecture
    • Deployment
    • Getting Started with Onum
    • Understanding The Essentials
      • Cards and Table Views
      • Data Types
      • Graph Calculations
      • The Time Range Selector
    • Key Terminology
  • THE WORKSPACE
    • Home
    • Listeners
      • Cloud Listeners
      • Listener Integrations
        • Amazon SQS
        • Amazon S3
        • Apache Kafka
        • Azure Event Hubs
        • Cisco NetFlow
        • Google Pub/Sub
        • HTTP
        • HTTP Pull
        • Microsoft 365
        • OpenTelemetry
        • Syslog
        • TCP
      • Labels
    • Pipelines
      • Building a Pipeline
        • AI Assistant
          • AI Pipeline Assistant
          • AI Action Assistant
      • Listeners
      • Actions
        • Advanced
          • Anonymizer
          • Bring Your Own Code
          • Field Generator
          • For Each
          • Google DLP
          • HTTP Request
          • Redis
        • Aggregation
          • Accumulator
          • Group By
        • AI
          • Amazon GenAI
          • BLIP-2
          • Cog
          • Google GenAI
          • Llama
          • Replicate
        • Detection
          • Sigma Rules
        • Enrichment
          • Lookup
        • Filtering
          • Conditional
          • Sampling
        • Formatting
          • Message Builder
        • Transformation
          • Field Transformation
            • Field Transformation Operations
              • Arithmetic / Logic
                • Divide Operation
                • Median
                • Multiply Operation
                • Subtract Operation
                • Sum Operation
              • Code tidy
                • JSON Minify
              • Control characters
                • Escape String
                • Unescape String
              • Conversion
                • Convert Area
                • Convert Data Units
                • Convert Distance
                • Convert Mass
                • Convert Speed
                • List to String
                • String to List
              • Data format
                • From Base
                • From Base64
                • From Hex
                • To Base
                • To Base64
                • To Hex
              • Date / Time
                • From Unix Timestamp
                • To Timestamp
                • To Unix Timestamp
                • Translate Datetime Format
              • Encoding / Decoding
                • From Binary
                • To Binary
                • To Decimal
              • Encryption / Encoding
                • JWT Decode
              • File system permissions
                • Parse Unix file permissions
              • Format conversion
                • CSV to JSON
                • JSON to CSV
              • Hashing
                • Keccak
                • MD2
                • MD4
                • MD5
                • SHA0
                • SHA1
                • SHA2
                • SHA3
                • Shake
                • SM3
              • Networking
                • Defang IP Address
                • Defang URL
                • Extract IP Address
                • Fang IP Address
                • Fang URLs
                • IP to Hexadecimal
                • Parse URI
                • URL Decode
                • URL Encode
              • Other
                • Parse Int
              • String
                • Length
              • Text sample adding
                • Pad Lines
              • Utils
                • Byte to Human Readable
                • Count Occurrences
                • CRC8 Checksum
                • CRC16 Checksum
                • CRC24 Checksum
                • CRC32 Checksum
                • Credit Card Obfuscator
                • Filter
                • Find and Replace
                • Regex
                • Remove Whitespace
                • Reverse String
                • Shuffle
                • Sort
                • Substring
                • Swap Case
                • To Lower Case
                • To Upper Case
          • Flat JSON
          • JSON Transformation
          • JSON Unroll
          • Math Expression
          • Parser
            • PCL (Parser Configuration Language)
        • Utils
          • Unique
      • Data sinks
      • Bulk Changes
      • Publishing & Versioning
      • Test your Pipeline
    • Data sinks
      • Data sink Integrations
        • Amazon S3
        • Amazon SQS
        • Azure Blob Storage
        • Azure Event Hubs
        • Devo
        • Google BigQuery
        • Google Cloud Storage
        • Google Pub/Sub
        • HTTP
        • Jira
        • Mail
        • Null
        • OpenTelemetry
        • PagerDuty
        • Pushover
        • Qradar
        • Relational Databases
        • ServiceNow
        • Slack
        • Splunk HEC
        • Syslog
        • TCP
        • Telegram
        • Twilio
    • Alerts
  • YOUR VAULT
    • Enrichment
    • Data History
    • Actions
  • ADMINISTRATION
    • Tenant Menu
    • Global Settings
      • Your Account
      • Organization Settings
        • Secrets Management
      • Tenant
        • Authentication
        • Users
        • Activity Log
  • MARKETPLACE
    • Onum Marketplace
      • Pulling Pipelines
        • Netskope Events Alert
        • OKTA System Log API
        • Sophos Connector SIEM
Powered by GitBook
On this page
  • Overview
  • Narrow Down Your Data
  • Add Filters
  • Select a Time Range
  • Select Tags
  • Metrics
  • Listeners
  • Data Sink
  • Data Volume
  • Visualize Your Data In/Out
  • Pipelines List
  • Cards View
  • Table view
  • Pipeline detail view
  • Duplicate a Pipeline
  • Create a Pipeline
  • Build your Pipeline

Was this helpful?

Export as PDF
  1. THE WORKSPACE

Pipelines

A Pipeline is Onum's way of streamlining your data

PreviousLabelsNextBuilding a Pipeline

Last updated 3 months ago

Was this helpful?

Overview

Use Pipelines to transform your events and build a data flow linking from and to .

Select the Pipelines tab at the left menu to visualize all your Pipelines in one place. Here's what you will find all the actions you can perform in this area:

  • The graph at the top plots the data volume going through your Pipelines. The purple line graph represents the events in, and the blue one represents the events going out. Use the buttons above the graph to switch between Events/Bytes, and the Frequency slider bar to choose how frequently you want to plot the events/bytes in the chart.

Narrow Down Your Data

There are various ways to narrow down what you see in this view, both the Pipeline list and the informative graphs. To do it, use the options at the top of this view:

Add Filters

Add filters to narrow down the Pipelines you see in the list. Click the + Add filter button and select the required filter type(s). You can filter by:

  • Name: Select a Condition (Contains, Equals, or Matches) and a Value to filter Pipelines by their names.

  • Status: Choose the status(es) you want to filter by: Draft, Running, and/or Stopped. You'll only see Pipelines with the selected status(es).

  • Created by: Filter for the creator of the Pipeline in the window that appears.

  • Updated by: Filter for users to see the Pipeline they last updated.

The filters applied will appear as tags at the top of the view.

Note that you can only add one filter of each type.

Select a Time Range

Select Tags

You can choose to view only those Pipelines that have been assigned the desired tags. You can create these tags in the Pipeline settings or from the cards view. Press the Enter key to confirm the tag, then Save.

To filter by tags, click the + Tags button and select the required tag(s).

Metrics

Below the filters, you will see 3 metrics informing you about various components in your Pipelines.

Listeners

View the events per second (EPS) ingested by all Listeners in your Pipelines for the selected time range, as well as the difference in percentage compared to the previous lapse.

Data Sink

View the events per second (EPS) sent by all Data Sinks in your Pipelines for the selected time range, as well as the difference in percentage compared to the previous.

Data Volume

See the overall data volume processed by all Pipelines for the selected time range, and the difference in percentage with the previous.

Visualize Your Data In/Out

Select between In and Out to see the volume received or sent by your Pipelines for the selected time range. The line graph represents the Events and the bar graph represents Bytes.

Hover over a point on the chart to show a tooltip containing the Events and Bytes in/out for the selected time, as well as a percentage of how much increase/decrease has occurred since the previous lapse of time since the one currently selected.

You can also analyze a different time range directly on the graph. To do it, click a starting date in the map and drag the frame that appears until the required ending date. The time range above will be also updated.

Pipelines List

At the bottom, you have a list of all the Pipelines in your tenant.

Use the Group by drop-down menu at the right area to select a criterion to organize your Pipelines in different groups (Status or None). You can also use the search icon to look for specific Pipelines by name.

Use the bottoms at the left of this area to display the Pipelines as Cards or listed in a Table:

Cards View

In this view, Pipelines are displayed as cards that display useful information. Click a card to open the Pipeline detail view, or double-click it to access it.

This is the information you can check on each card:

  • The percentage at the top left corner indicates the amount of data that goes out of the Pipeline compared to the total incoming events, so you can check how data is optimized at a glance. Hover over it to see the in/out data in bytes and the estimation over the next 24 hours.

  • You can also see the status of the Pipeline (Running, Draft, or Stopped).

  • Next to the status, you can check the Pipeline current version.

  • Click the Add tag button to define tags for the Pipeline. To assign a new tag, simply type the name you wish to assign, make sure to press Enter, and then select the Save button. If the Pipeline has tags defined already, you'll see the number of tags next to the tag icon.

  • Click the ellipses in the right-hand corner of the card to reveal the options to Edit, Copy ID, or Remove it.

Table view

In this view, Pipelines are displayed in a table, where each row represents a Pipeline. Click a row to open the Pipeline detail view, or double-click it to access it.

Click the cog icon at the top left corner to rearrange the column order, hide columns, or pin them. You can click Reset to recover the default configuration.

Pipeline detail view

The details pane is split into three tabs showing the Pipeline at different statuses:

Tab
Description

Running

This is the main tab, where you can see details of the Pipeline versions that are currently running.

Select the drop-down next to the Pipeline version name to see which clusters the Pipeline is currently running in.

Draft

Check the details of the draft versions of your Pipeline.

Stopped

Check the details of the version of your Pipeline that are currently stopped.

Once you have located the Pipeline to work with, click Edit Pipeline to open it.

Duplicate a Pipeline

If you wish to use a Pipeline just like the one you are currently working on, click the ellipses in the Card or Table view and select Duplicate, or from the Configuration pane.

Create a Pipeline

Depending on your permissions, you can create a new Pipeline from this view. There are several ways to create a new Pipeline:




This will open the new Pipeline, ready to be built.

Keep reading to learn how to build a Pipeline from this view.

Build your Pipeline

At the bottom, you will find a list of all the Pipelines in your tenant. You can switch between the Cards view, which shows each Pipeline in a card, and the Table view, which displays Pipelines listed in a table. Learn more about the cards and table views .

If you wish to see data for a specific time period, this is the place to click. Go to to dive into the specifics of how the time range works.

Note that these metrics are affected by the selected.

Click a Pipeline to open its settings in the right-hand pane. Here you can see and edit the Pipeline. Click the ellipses in the top right to Copy ID or Duplicate / Remove it.

From the view

From the page

Give your Pipeline a name and add optional Tags to identify it. You can also assign a in the top-right.

See to learn step by step.

in this article
Selecting a Time Range
time range
Pipeline versions
Pipelines
Home
Version
Building a Pipeline
Actions
Listeners
Data sinks