Publishing & Versioning

Pipeline publishing

Imagine your data stream requires modifications and you do not wish to redo your Pipeline configuration every time. Onum allows you to simultaneously work on multiple versions of the same Pipeline.

When you create a Pipeline for the first time and add the very first element to the canvas, you will create a Draft and a version history, which appears at the top right. This will be where you can build your Pipeline before deciding to publish it. This means you can work on various versions in the background, whilst choosing only one to be published.

Once you are happy with the draft version of your Pipeline, there are various things you can do:

Test Pipeline

Before moving further, you can test your Pipeline to see how the results will look without publishing it. Click Test Pipeline, enter the maximum number of events to allow and select Debug.

Click through the different tabs to see your data as it moves through each Pipeline element. You can also view the specifics of what enters and exits each port. Click Stop test once you're done.

Publish Pipeline

If you have finished your Pipeline and would like to confirm its completion, you can click the Publish button in the top right. Publishing is a way to validate this version.

After clicking Publish, you can decide to run this Pipeline on selected or all clusters. Select the desired clusters and click Publish. Notice the Running in menu that appears next to the versions dropdown menu. When you open a Pipeline, the default version that opens will be Running.

Use the Running in dropdown menu to view a list of the clusters your Pipeline is currently running in. You can update these settings by selecting the required cluster(s) from the list and clicking Update.

Once you have pushed to a cluster, you can no longer retrieve it or unselect it from the list. Instead, you can Stop the Pipeline.

Stop Pipeline

You can Stop a Pipeline that is running in a cluster at any time using the Stop pipeline button that appears on any running Pipeline. Select which, if not all, clusters to stop running on and confirm Stop pipeline again.

When stopping a Pipeline, the worry is that your data will be lost. Don't worry! This will simply save your Pipeline in the Stopped versions list.

Pipeline versioning

Choosing to Publish a Pipeline will generate a new version, which you can view in the tree, or in the Pipeline preview. Expand one to see its details and click the pencil icon to edit the name.

Use the Running in dropdown to see which clusters it is running in. Aside from the Pipeline version, each Listener, Action and Data Sink has its own version to ensure they are constantly evolving. You can see the current version of each in the article dedicated to it. For older versions, go to the Changelog section.