LogoLogo
WebsiteBlogLogin
  • Onum Docs
  • Use Cases
  • Videos
  • Release Notes
  • Welcome
  • Getting Started
    • About Onum
    • Architecture
    • Deployment
    • Getting Started with Onum
    • Understanding The Essentials
      • Cards and Table Views
      • Data Types
      • Graph Calculations
      • The Time Range Selector
    • Key Terminology
  • THE WORKSPACE
    • Home
    • Listeners
      • Cloud Listeners
      • Listener Integrations
        • Amazon S3
        • Amazon SQS
        • Apache Kafka
        • Azure Event Hubs
        • Cisco NetFlow
        • Google Cloud Storage
        • Google Pub/Sub
        • HTTP
        • HTTP Pull
          • Netskope integration
          • OKTA integration
          • Sophos integration
          • CrowdStrike integration
          • Cortex integration
        • Microsoft 365
        • OpenTelemetry
        • Syslog
        • TCP
        • Tick
      • Labels
    • Pipelines
      • Building a Pipeline
        • AI Assistant
          • AI Pipeline Assistant
          • AI Action Assistant
      • Listeners
      • Actions
        • Advanced
          • Anonymizer
          • Bring Your Own Code
          • Field Generator
          • For Each
          • Google DLP
          • HTTP Request
          • Redis
        • Aggregation
          • Accumulator
          • Group By
        • AI
          • Amazon GenAI
          • BLIP-2
          • Cog
          • Google GenAI
          • Llama
          • Replicate
        • Detection
          • Sigma Rules
        • Enrichment
          • Lookup
        • Filtering
          • Conditional
          • Sampling
        • Formatting
          • Message Builder
        • Schemas
          • OCSF
        • Transformation
          • Field Transformation
            • Field Transformation Operations
              • Arithmetic / Logic
                • Divide Operation
                • Median
                • Multiply Operation
                • Subtract Operation
                • Sum Operation
              • Code tidy
                • JSON Minify
              • Control characters
                • Escape String
                • Unescape String
              • Conversion
                • Convert Area
                • Convert Data Units
                • Convert Distance
                • Convert Mass
                • Convert Speed
                • List to String
                • String to List
              • Data format
                • From Base
                • From Base64
                • From Hex
                • To Base
                • To Base64
                • To Hex
              • Date / Time
                • From Unix Timestamp
                • To Timestamp
                • To Unix Timestamp
                • Translate Datetime Format
              • Encoding / Decoding
                • From Binary
                • To Binary
                • To Decimal
              • Encryption / Encoding
                • JWT Decode
              • File system permissions
                • Parse Unix file permissions
              • Format conversion
                • CSV to JSON
                • JSON to CSV
                • Protobuf to JSON
              • Hashing
                • Keccak
                • MD2
                • MD4
                • MD5
                • SHA0
                • SHA1
                • SHA2
                • SHA3
                • Shake
                • SM3
              • List manipulation
                • Index list boolean
                • Index list float
                • Index list integer
                • Index list string
                • Index list timestamp
              • Networking
                • Defang IP Address
                • Defang URL
                • Extract IP Address
                • Fang IP Address
                • Fang URLs
                • IP to Hexadecimal
                • Parse URI
                • URL Decode
                • URL Encode
              • Other
                • Parse Int
              • String
                • Length
              • Text sample adding
                • Pad Lines
              • Utils
                • Byte to Human Readable
                • Count Occurrences
                • CRC8 Checksum
                • CRC16 Checksum
                • CRC24 Checksum
                • CRC32 Checksum
                • Credit Card Obfuscator
                • Filter
                • Find and Replace
                • Regex
                • Remove Whitespace
                • Reverse String
                • Shuffle
                • Sort
                • Substring
                • Swap Case
                • To Lower Case
                • To Upper Case
          • Flat JSON
          • JSON Transformation
          • JSON Unroll
          • Math Expression
          • Parser
            • PCL (Parser Configuration Language)
        • Utils
          • Unique
      • Data sinks
      • Bulk Changes
      • Publishing & Versioning
      • Test your Pipeline
    • Data sinks
      • Data sink Integrations
        • Amazon S3
        • Amazon SQS
        • Azure Blob Storage
        • Azure Event Hubs
        • Devo
        • Google BigQuery
        • Google Cloud Storage
        • Google Pub/Sub
        • HTTP
        • Jira
        • Mail
        • Null
        • OpenTelemetry
        • PagerDuty
        • Pushover
        • Qradar
        • Relational Databases
        • ServiceNow
        • Slack
        • Splunk HEC
        • Syslog
        • TCP
        • Telegram
        • Twilio
    • Alerts
  • YOUR VAULT
    • Enrichment
    • Data History
    • Actions
  • ADMINISTRATION
    • Tenant Menu
    • Global Settings
      • Your Account
      • Organization Settings
        • Secrets Management
      • Tenant
        • Authentication
        • Users
        • Activity Log
        • API Keys
  • MARKETPLACE
    • Onum Marketplace
      • Pulling Pipelines
        • CrowdStrike Event Stream Logs - Falcon API
        • Netskope Events Alert
        • OKTA System Log API
        • Sophos Connector SIEM
Powered by GitBook
On this page
  • Description
  • Data types
  • Input data
  • Output data
  • Parameters
  • Example

Was this helpful?

Export as PDF
  1. THE WORKSPACE
  2. Pipelines
  3. Actions
  4. Transformation
  5. Field Transformation
  6. Field Transformation Operations
  7. Utils

Filter

PreviousCredit Card ObfuscatorNextFind and Replace

Last updated 3 months ago

Was this helpful?

Description

This operation allows you to filter lines of text based on specific conditions or patterns. This operation is useful when you want to keep only lines that match a certain pattern or exclude lines that don't meet specific criteria.


Data types

These are the input/output expected data types for this operation:

Input data

- The input strings to be split and filtered.

Output data

- The filtered results after applying the regular expression and delimiter.


Parameters

These are the parameters you need to configure to use this operation (mandatory parameters are marked with a *):

Delimiter*

Choose the delimiter that separates each segment in your input data:

  • Line feed - Select this to have each line of text as a separate value.

  • Space - Your segments are separated by spaces.

  • Comma - Your segments are separated by commas (,)

  • Semi-colon - Your segments are separated by semi-colons (;)

  • Colon - Your segments are separated by colons (:)

  • CRLF - Carriage return line feed (CRLF) is a control character that originally referred to moving the carriage on typewriters back to the starting position. In computing, it’s used in classic Mac OS and some Windows-based systems to mark the end of a line. If your input uses \r as the line-ending character, you can set the delimiter to \r to correctly separate values. For example: 100\r200\r300

Regex*

Define the pattern for filtering using a regular expression, which enables more complex matching conditions.

Invert

Set this parameter to true if you want to reverse the filter logic, so it keeps lines that do not match the specified pattern.


Example

Suppose you want to filter a series of strings to extract only segments that start with error:

  1. In the Operation field, choose Filter.

  2. Set Delimiter to Line feed

  3. Set Regex to ^Error

  4. Set Invert to false.

  5. Give your Output field a name and click Save. The strings in your input field will be filtered with the specified conditions.

For example, this text:

Error: File not found
Warning: Low memory
Info: Operation completed
Error: Disk full

will be filtered as:

Error: File not found
Error: Disk full

You can try out operations with specific values using the Input field above the operation. You can enter the value in the example above and check the result in the Output field.

In your Pipeline, open the required configuration and select the input Field.

Action