LogoLogo
WebsiteBlogLogin
  • Onum Docs
  • Use Cases
  • Videos
  • Release Notes
  • Welcome
  • Getting Started
    • About Onum
    • Architecture
    • Deployment
    • Getting Started with Onum
    • Understanding The Essentials
      • Cards and Table Views
      • Data Types
      • Graph Calculations
      • The Time Range Selector
    • Key Terminology
  • THE WORKSPACE
    • Home
    • Listeners
      • Cloud Listeners
      • Listener Integrations
        • Amazon SQS
        • Amazon S3
        • Apache Kafka
        • Azure Event Hubs
        • Cisco NetFlow
        • Google Pub/Sub
        • HTTP
        • HTTP Pull
        • Microsoft 365
        • OpenTelemetry
        • Syslog
        • TCP
      • Labels
    • Pipelines
      • Building a Pipeline
        • AI Assistant
          • AI Pipeline Assistant
          • AI Action Assistant
      • Listeners
      • Actions
        • Advanced
          • Anonymizer
          • Bring Your Own Code
          • Field Generator
          • For Each
          • Google DLP
          • HTTP Request
          • Redis
        • Aggregation
          • Accumulator
          • Group By
        • AI
          • Amazon GenAI
          • BLIP-2
          • Cog
          • Google GenAI
          • Llama
          • Replicate
        • Detection
          • Sigma Rules
        • Enrichment
          • Lookup
        • Filtering
          • Conditional
          • Sampling
        • Formatting
          • Message Builder
        • Transformation
          • Field Transformation
            • Field Transformation Operations
              • Arithmetic / Logic
                • Divide Operation
                • Median
                • Multiply Operation
                • Subtract Operation
                • Sum Operation
              • Code tidy
                • JSON Minify
              • Control characters
                • Escape String
                • Unescape String
              • Conversion
                • Convert Area
                • Convert Data Units
                • Convert Distance
                • Convert Mass
                • Convert Speed
                • List to String
                • String to List
              • Data format
                • From Base
                • From Base64
                • From Hex
                • To Base
                • To Base64
                • To Hex
              • Date / Time
                • From Unix Timestamp
                • To Timestamp
                • To Unix Timestamp
                • Translate Datetime Format
              • Encoding / Decoding
                • From Binary
                • To Binary
                • To Decimal
              • Encryption / Encoding
                • JWT Decode
              • File system permissions
                • Parse Unix file permissions
              • Format conversion
                • CSV to JSON
                • JSON to CSV
              • Hashing
                • Keccak
                • MD2
                • MD4
                • MD5
                • SHA0
                • SHA1
                • SHA2
                • SHA3
                • Shake
                • SM3
              • Networking
                • Defang IP Address
                • Defang URL
                • Extract IP Address
                • Fang IP Address
                • Fang URLs
                • IP to Hexadecimal
                • Parse URI
                • URL Decode
                • URL Encode
              • Other
                • Parse Int
              • String
                • Length
              • Text sample adding
                • Pad Lines
              • Utils
                • Byte to Human Readable
                • Count Occurrences
                • CRC8 Checksum
                • CRC16 Checksum
                • CRC24 Checksum
                • CRC32 Checksum
                • Credit Card Obfuscator
                • Filter
                • Find and Replace
                • Regex
                • Remove Whitespace
                • Reverse String
                • Shuffle
                • Sort
                • Substring
                • Swap Case
                • To Lower Case
                • To Upper Case
          • Flat JSON
          • JSON Transformation
          • JSON Unroll
          • Math Expression
          • Parser
            • PCL (Parser Configuration Language)
        • Utils
          • Unique
      • Data sinks
      • Bulk Changes
      • Publishing & Versioning
      • Test your Pipeline
    • Data sinks
      • Data sink Integrations
        • Amazon S3
        • Amazon SQS
        • Azure Blob Storage
        • Azure Event Hubs
        • Devo
        • Google BigQuery
        • Google Cloud Storage
        • Google Pub/Sub
        • HTTP
        • Jira
        • Mail
        • Null
        • OpenTelemetry
        • PagerDuty
        • Pushover
        • Qradar
        • Relational Databases
        • ServiceNow
        • Slack
        • Splunk HEC
        • Syslog
        • TCP
        • Telegram
        • Twilio
    • Alerts
  • YOUR VAULT
    • Enrichment
    • Data History
    • Actions
  • ADMINISTRATION
    • Tenant Menu
    • Global Settings
      • Your Account
      • Organization Settings
        • Secrets Management
      • Tenant
        • Authentication
        • Users
        • Activity Log
  • MARKETPLACE
    • Onum Marketplace
      • Pulling Pipelines
        • Netskope Events Alert
        • OKTA System Log API
        • Sophos Connector SIEM
Powered by GitBook
On this page
  • Overview
  • 1. General settings
  • 2. The metrics bar
  • 3. Add to the Pipeline
  • 4. Canvas
  • 5. Navigation options
  • Ports

Was this helpful?

Export as PDF
  1. THE WORKSPACE
  2. Pipelines

Building a Pipeline

PreviousPipelinesNextAI Assistant

Last updated 1 month ago

Was this helpful?

Overview

The Pipeline canvas provides infinite possibilities to use your data.


1. General settings

This pane shows the general properties of your Pipeline. Click the ellipses next to its name to Copy ID.

Depending on your permissions, you can view or modify:

  • Name: When you create a pipeline by default, the first recommendation is to change the default name. But you can modify the name at any time clicking on the pencil icon close to the Pipeline's name

  • Tags: Click the tag icon to open the menu.

  • Clusters: Here you can see how many clusters your Pipeline is running in, as well as update them.

  • Versions: View and run multiple versions of the Pipeline.

  • Stop/Start Pipeline: Stop and start the Pipeline in some or all of the clusters it is running in.

  • Publish

When you modify your Pipeline, you will be creating a new version. When your modifications are complete, you can Publish this new version using this button in the top right.


2. The metrics bar

If the Pipeline is running, the Metrics bar provides a visual, graphical overview of the data being processed in your Pipeline.

  • Events In: View the total events in per second for the selected period, compared to the previous range (in %).

  • Bytes In: The total bytes in per second for the selected time range, compared to the previous (in %).

  • Events Out: View the total events out per second for the selected period, compared to the previous range (in %).

  • Bytes Out: The total bytes out per second for the selected time range, compared to the previous (in %).

  • Latency: The time (in nanoseconds) it takes for data to travel from one point to another, compared to the previous (in %).

Set a time range

You can set a time range to view the metrics for a specific period of time. This will be used to calculate the percentages, compared to the previous time of the same period selected.

Hide/Show metrics

Use the Hide metrics/Show metrics button to hide/show the metrics pane.


3. Add to the Pipeline

Simply drag and drop an element from the left-hand side onto the canvas to add it to your Pipeline.


4. Canvas

The canvas is where you will build your Pipeline. Drag and drop an element from the left pane to add it to your Pipeline.

Delete a node

If you have enough permissions to modify this Pipeline, click the node in the canvas and select the Remove icon.


5. Navigation options

Zoom in/out, Center, undo, and redo changes using the buttons on the right.

Use the window in the bottom-right to move around the Canvas.

Connect the separate nodes of the canvas to form a Pipeline from start to finish.

Simply click the port you wish to link from and drag to the port you wish to link to. When you let go, you will see a link form between the two.

To Unlink, click anywhere on the link and select unlink in the menu.


Ports

Notice the ports of each element in the canvas. Ports are used as connectors to other nodes of the Pipeline, linking either incoming or outgoing data.

Listener: As a Listener is used to send information on, there are no in ports, and one out port.

Action: Actions generally have one in port, injecting it with data. When information is output, it will be sent via the default port. If there are problems sending on the data, it will not be lost, bur rather output via the error port.

Datasink: A datasink is the end stop for our data, so there is only one in port that receives your processed data.

Click one to read more about how to configure them:

Go to to learn more.

You can carry out all these actions in if you wish to modify more than one Pipeline at a time.

Go to to learn more about the specifics of how this works.

For Listeners, you can drag the specific down to the required level. Once in the Pipeline, you can see which Listener the label belongs to by hovering over it, or in the Metrics area of the configuration pane.

Click it in the canvas to open its .

Create links between your nodes to create a flow of data between them. Learn more about below.

Managing versions
bulk
Selecting a Time Range
Label
Properties
links

Learn about how to set up and use Listeners

Discover Actions to manage and customize your data

Add the final piece of the puzzle for simpler data

Listeners
Actions
Datasinks