Data Reduction & Optimization
Last updated
Was this helpful?
Last updated
Was this helpful?
Data on the same event is often produced by multiple devices, leading to staggering amounts of duplication and data bloat.
You can reduce down data to what you really need and optimize it before sending it on.
Filter your data to reduce what you send, remove incomplete or duplicated data, as well as tranform it into any format to match the requirements to make it more actionable.
You receive data regarding firewall activity across an entire platform when you are only interested in threatening IPs. You can reduce unwanted data and send on only the required information. We will use the Parse to give structure and separate into fields. Use message builder to decide which to keep and send on.
Let's do this together.
Go to the Pipelines tab and select New Pipeline.
Select the pencil icon to rename it Reduction firewall.paloalto.threat and click Enter to confirm.
Now, you'll need to find the Listener firewall.paloalto.threat from the list and drag it onto the middle canvas to add it to your Pipeline. This Listener provides information on all paloalto threat activity at the source.
We will need to parse the data to separate out the fields to easily identify the desired information.
Drag and drop the Parser and Message Builder actions from the Actions pane.
Link the Listener to the Parser by dragging from the out port to the in port of the Parser.
Click the Parser in the canvas and select Configuration.
First we must select the field to parse from the Listener in order to separate more specific data. This is the field containing the raw data.
Now we have decided which field, from where, and how to parse, we need to specify how it is output to the next action. Edit the field name.
Click Save.
Click the Message Builder in the canvas and select Configuration.
This is where we define what the final message will be by selecting which fields to send on.
Compose the message using to join it to a coherent message to generate a CSV.
Click Save.
Select the out port (as opposed to the error out port) of the Parser and link it to the Message builder in the same way.
We have now successfully reduced the data from the listener to a concise message to be sent on to the end destination.
Finally, we must tell Onum where and how to send the data.
Drag the Syslog from the Data sinks tab and link the out output port of the Message Builder to the input port.
Click Publish. Your data has now been optimized.
