LogoLogo
WebsiteBlogLogin
  • Onum Docs
  • Use Cases
  • Videos
  • Release Notes
  • Welcome
  • Getting Started
    • About Onum
    • Architecture
    • Deployment
    • Getting Started with Onum
    • Understanding The Essentials
      • Cards and Table Views
      • Data Types
      • Graph Calculations
      • The Time Range Selector
    • Key Terminology
  • THE WORKSPACE
    • Home
    • Listeners
      • Cloud Listeners
      • Listener Integrations
        • Amazon SQS
        • Amazon S3
        • Apache Kafka
        • Azure Event Hubs
        • Cisco NetFlow
        • Google Pub/Sub
        • HTTP
        • HTTP Pull
        • Microsoft 365
        • OpenTelemetry
        • Syslog
        • TCP
      • Labels
    • Pipelines
      • Building a Pipeline
        • AI Assistant
          • AI Pipeline Assistant
          • AI Action Assistant
      • Listeners
      • Actions
        • Advanced
          • Anonymizer
          • Bring Your Own Code
          • Field Generator
          • For Each
          • Google DLP
          • HTTP Request
          • Redis
        • Aggregation
          • Accumulator
          • Group By
        • AI
          • Amazon GenAI
          • BLIP-2
          • Cog
          • Google GenAI
          • Llama
          • Replicate
        • Detection
          • Sigma Rules
        • Enrichment
          • Lookup
        • Filtering
          • Conditional
          • Sampling
        • Formatting
          • Message Builder
        • Transformation
          • Field Transformation
            • Field Transformation Operations
              • Arithmetic / Logic
                • Divide Operation
                • Median
                • Multiply Operation
                • Subtract Operation
                • Sum Operation
              • Code tidy
                • JSON Minify
              • Control characters
                • Escape String
                • Unescape String
              • Conversion
                • Convert Area
                • Convert Data Units
                • Convert Distance
                • Convert Mass
                • Convert Speed
                • List to String
                • String to List
              • Data format
                • From Base
                • From Base64
                • From Hex
                • To Base
                • To Base64
                • To Hex
              • Date / Time
                • From Unix Timestamp
                • To Timestamp
                • To Unix Timestamp
                • Translate Datetime Format
              • Encoding / Decoding
                • From Binary
                • To Binary
                • To Decimal
              • Encryption / Encoding
                • JWT Decode
              • File system permissions
                • Parse Unix file permissions
              • Format conversion
                • CSV to JSON
                • JSON to CSV
              • Hashing
                • Keccak
                • MD2
                • MD4
                • MD5
                • SHA0
                • SHA1
                • SHA2
                • SHA3
                • Shake
                • SM3
              • Networking
                • Defang IP Address
                • Defang URL
                • Extract IP Address
                • Fang IP Address
                • Fang URLs
                • IP to Hexadecimal
                • Parse URI
                • URL Decode
                • URL Encode
              • Other
                • Parse Int
              • String
                • Length
              • Text sample adding
                • Pad Lines
              • Utils
                • Byte to Human Readable
                • Count Occurrences
                • CRC8 Checksum
                • CRC16 Checksum
                • CRC24 Checksum
                • CRC32 Checksum
                • Credit Card Obfuscator
                • Filter
                • Find and Replace
                • Regex
                • Remove Whitespace
                • Reverse String
                • Shuffle
                • Sort
                • Substring
                • Swap Case
                • To Lower Case
                • To Upper Case
          • Flat JSON
          • JSON Transformation
          • JSON Unroll
          • Math Expression
          • Parser
            • PCL (Parser Configuration Language)
        • Utils
          • Unique
      • Data sinks
      • Bulk Changes
      • Publishing & Versioning
      • Test your Pipeline
    • Data sinks
      • Data sink Integrations
        • Amazon S3
        • Amazon SQS
        • Azure Blob Storage
        • Azure Event Hubs
        • Devo
        • Google BigQuery
        • Google Cloud Storage
        • Google Pub/Sub
        • HTTP
        • Jira
        • Mail
        • Null
        • OpenTelemetry
        • PagerDuty
        • Pushover
        • Qradar
        • Relational Databases
        • ServiceNow
        • Slack
        • Splunk HEC
        • Syslog
        • TCP
        • Telegram
        • Twilio
    • Alerts
  • YOUR VAULT
    • Enrichment
    • Data History
    • Actions
  • ADMINISTRATION
    • Tenant Menu
    • Global Settings
      • Your Account
      • Organization Settings
        • Secrets Management
      • Tenant
        • Authentication
        • Users
        • Activity Log
  • MARKETPLACE
    • Onum Marketplace
      • Pulling Pipelines
        • Netskope Events Alert
        • OKTA System Log API
        • Sophos Connector SIEM
Powered by GitBook
On this page
  • Test Pipeline
  • Publish
  • Run
  • Running in Clusters
  • Stop
  • Versioning

Was this helpful?

Export as PDF
  1. THE WORKSPACE
  2. Pipelines

Publishing & Versioning

PreviousBulk ChangesNextTest your Pipeline

Last updated 5 months ago

Was this helpful?

Imagine your data stream requires modifications and you do not wish to redo your Pipeline configuration every time. Onum allows you to simultaneously work on multiple versions of the same Pipeline.

When you modify a Pipeline, you will be creating a version history, which appears in the top right.

When you create or modify a Pipeline for the first time, you will create a Draft. This will be where you can build your Pipeline before deciding to publish it. This means you can work on various versions in the background, whilst choosing only one to be published.

Once you are happy with the Draft version of your Pipeline, there are various things you can do.

Test Pipeline

Before moving further, you can test your Pipeline to see how the results will look without publishing it. Click Test Pipeline, enter the maximum number of events to allow and select Debug.

Click through the different tabs to see your data as it moves through each Pipeline element. You can also view the specifics of what enters and exits each port.

Publish

If you have finished your Pipeline and would like to confirm its completion, you can click the Publish button in the top right. Publishing is a way to validate this version.

Now, you can decide whether to run this Pipeline on a Cluster (see the section below) or simply validate this version without pushing it to the cluster.

Run

After clicking Publish, you can decide to run this Pipeline on selected or all clusters.

Select the desired clusters and click Publish.

Choosing to Cancel this action will stop the Pipeline from running on the cluster, but will not undo the Publish action.

Notice the Running In menu that appears. When you open a Pipeline, the default version that opens will be Running.

Running in Clusters

Use the Running in Clusters drop-down menu to view a list of the clusters your Pipeline is currently running in.

Update these settings by selecting the Cluster from the list and clicking Update x Clusters.

Once you have pushed to a cluster, you can no longer retrieve it or unselect it from the list. Instead, you can Stop it.

You can view running and run Pipelines from the configuration pane, as well as select which clusters to do so in.

The cluster selector only appears when you have more than 10 clusters to choose from.

Stop

You can Stop a Pipeline that is running in a Cluster at any time using the Stop Pipeline button that appears on any running Pipeline.

Select which, if not all, clusters to stop running on and confirm Stop Pipeline again.

When stopping a Pipeline, the worry is that your data will be lost. Don't worry! This will simply save your Pipeline in the Stopped versions list.

You can change the description using the pencil icon.

In the Pipeline Settings, you can see a comprehensive list of all of its previous versions and the changes in each.

Versioning

Choosing to Publish a Pipeline will generate a new version, which you can view in the tree, or in the Pipeline preview. Expand one to see its details and click the pencil icon to edit the name.

Use the Running in drop-down to see which clusters it is running in.

Aside from the Pipeline version, each Listener, Action and Datasink has its own version to ensure they are constantly evolving.

You can see the current version of each in the article dedicated to it.

For more information on Clusters, read all about the

For older versions, go to the section.

Onum Architecture.
Changelog