LogoLogo
WebsiteBlogLogin
  • Onum Docs
  • Use Cases
  • Videos
  • Release Notes
  • Welcome
  • Getting Started
    • About Onum
    • Architecture
    • Deployment
    • Getting Started with Onum
    • Understanding The Essentials
      • Cards and Table Views
      • Data Types
      • Graph Calculations
      • The Time Range Selector
    • Key Terminology
  • THE WORKSPACE
    • Home
    • Listeners
      • Cloud Listeners
      • Listener Integrations
        • Amazon SQS
        • Amazon S3
        • Apache Kafka
        • Azure Event Hubs
        • Cisco NetFlow
        • Google Pub/Sub
        • HTTP
        • HTTP Pull
        • Microsoft 365
        • OpenTelemetry
        • Syslog
        • TCP
      • Labels
    • Pipelines
      • Building a Pipeline
        • AI Assistant
          • AI Pipeline Assistant
          • AI Action Assistant
      • Listeners
      • Actions
        • Advanced
          • Anonymizer
          • Bring Your Own Code
          • Field Generator
          • For Each
          • Google DLP
          • HTTP Request
          • Redis
        • Aggregation
          • Accumulator
          • Group By
        • AI
          • Amazon GenAI
          • BLIP-2
          • Cog
          • Google GenAI
          • Llama
          • Replicate
        • Detection
          • Sigma Rules
        • Enrichment
          • Lookup
        • Filtering
          • Conditional
          • Sampling
        • Formatting
          • Message Builder
        • Transformation
          • Field Transformation
            • Field Transformation Operations
              • Arithmetic / Logic
                • Divide Operation
                • Median
                • Multiply Operation
                • Subtract Operation
                • Sum Operation
              • Code tidy
                • JSON Minify
              • Control characters
                • Escape String
                • Unescape String
              • Conversion
                • Convert Area
                • Convert Data Units
                • Convert Distance
                • Convert Mass
                • Convert Speed
                • List to String
                • String to List
              • Data format
                • From Base
                • From Base64
                • From Hex
                • To Base
                • To Base64
                • To Hex
              • Date / Time
                • From Unix Timestamp
                • To Timestamp
                • To Unix Timestamp
                • Translate Datetime Format
              • Encoding / Decoding
                • From Binary
                • To Binary
                • To Decimal
              • Encryption / Encoding
                • JWT Decode
              • File system permissions
                • Parse Unix file permissions
              • Format conversion
                • CSV to JSON
                • JSON to CSV
              • Hashing
                • Keccak
                • MD2
                • MD4
                • MD5
                • SHA0
                • SHA1
                • SHA2
                • SHA3
                • Shake
                • SM3
              • Networking
                • Defang IP Address
                • Defang URL
                • Extract IP Address
                • Fang IP Address
                • Fang URLs
                • IP to Hexadecimal
                • Parse URI
                • URL Decode
                • URL Encode
              • Other
                • Parse Int
              • String
                • Length
              • Text sample adding
                • Pad Lines
              • Utils
                • Byte to Human Readable
                • Count Occurrences
                • CRC8 Checksum
                • CRC16 Checksum
                • CRC24 Checksum
                • CRC32 Checksum
                • Credit Card Obfuscator
                • Filter
                • Find and Replace
                • Regex
                • Remove Whitespace
                • Reverse String
                • Shuffle
                • Sort
                • Substring
                • Swap Case
                • To Lower Case
                • To Upper Case
          • Flat JSON
          • JSON Transformation
          • JSON Unroll
          • Math Expression
          • Parser
            • PCL (Parser Configuration Language)
        • Utils
          • Unique
      • Data sinks
      • Bulk Changes
      • Publishing & Versioning
      • Test your Pipeline
    • Data sinks
      • Data sink Integrations
        • Amazon S3
        • Amazon SQS
        • Azure Blob Storage
        • Azure Event Hubs
        • Devo
        • Google BigQuery
        • Google Cloud Storage
        • Google Pub/Sub
        • HTTP
        • Jira
        • Mail
        • Null
        • OpenTelemetry
        • PagerDuty
        • Pushover
        • Qradar
        • Relational Databases
        • ServiceNow
        • Slack
        • Splunk HEC
        • Syslog
        • TCP
        • Telegram
        • Twilio
    • Alerts
  • YOUR VAULT
    • Enrichment
    • Data History
    • Actions
  • ADMINISTRATION
    • Tenant Menu
    • Global Settings
      • Your Account
      • Organization Settings
        • Secrets Management
      • Tenant
        • Authentication
        • Users
        • Activity Log
  • MARKETPLACE
    • Onum Marketplace
      • Pulling Pipelines
        • Netskope Events Alert
        • OKTA System Log API
        • Sophos Connector SIEM
Powered by GitBook
On this page
  • Overview
  • Minimum requirements
  • Configuration
  • Metadata
  • Configuration

Was this helpful?

Export as PDF
  1. THE WORKSPACE
  2. Listeners
  3. Listener Integrations

Amazon S3

Most recent version: v1.0.0

PreviousAmazon SQSNextApache Kafka

Last updated 25 days ago

Was this helpful?

See the changelog of this Listener type .

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Minimum requirements

Before configuring and starting to send data with the Amazon S3 Listener, you need to take into consideration the following requirements:

  • Your Amazon user needs at least permission to use the GetObject operation (S3) and the ReceiveMessage and DeleteMessageBatch operations (SQS Bucket) to make this Listener work.

  • You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new files are added. Learn how to do it below:

Configure your Amazon S3 bucket to send notifications to an Amazon SQS when new files are added

1. Create an Amazon SQS Queue

  1. Sign in to the AWS Management Console and open the Amazon SQS console.

  2. Choose Create Queue and configure the queue settings as needed.

  3. After creating the queue, note its Amazon Resource Name (ARN), which follows this format: arn:aws:sqs:<region>:<account-id>:<queue-name>.

2. Modify the SQS Queue Policy to Allow S3 to Send Messages

  1. In the Amazon SQS console, select your queue.

  2. Navigate to the Access Policy tab and choose Edit.

  3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details:

  {
    "Version": "2012-10-17",
    "Id": "S3ToSQSPolicy",
    "Statement": [
      {
        "Sid": "AllowS3Bucket",
        "Effect": "Allow",
        "Principal": {
          "Service": "s3.amazonaws.com"
        },
        "Action": "SQS:SendMessage",
        "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:s3:::<bucket-name>"
          },
          "StringEquals": {
            "aws:SourceAccount": "<account-id>"
          }
        }
      }
    ]
  }

Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue.

3. Configure S3 Event Notifications

  1. Open the Amazon S3 console and select the bucket you want to configure.

  2. Go to the Properties tab and find the "Event notifications" section.

  3. Click on Create event notification.

  4. Provide a descriptive name for the event notification.

  5. In the Event types section, select All object create events or specify particular events that should trigger notifications.

  6. In the Destination section, choose SQS Queue and select the queue you configured earlier.

  7. Save the configuration.

4. Test the Configuration

  1. Upload a new file to your S3 bucket.

  2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly.

Additional Considerations

  • Cross-Region Configurations: Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets.

  • Permissions: Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions.

  • Object Key Name Filtering: If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded.

Configuration

Now you need to specify how and where to collect the data, and how to establish a connection with AWS S3.

Metadata

Enter the basic information for the new Listener.

Parameter
Description

Name*

Enter a name for the new Listener.

Description

Optionally, enter a description for the Listener.

Tags

Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

Now, add the configuration to establish the connection.

Objects

Parameter
Description

Compression*

Select the compression method used in the ingested S3 files. This accepts the standard compression codecs (gzip, zlib, bzip2), none for no compression, and auto to autodetect the compression type from the file extension.

Format*

Select the format of the ingested S3 files. This currently accepts JSON array (a big JSON array containing a JSON object for each event), JSON lines (a JSON object representing an event on each line), and auto to autodetect the compression type from the file extension (.json or .jsonl, respectively).

Bucket

Parameter
Description

Region*

Choose the region the bucket is found in, also found in your Buckets area, next to the name.

Name

Access key ID*

The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

  1. In the left panel, click on Users.

  2. Select your IAM user.

  3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

Secret access key*

Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.

Bucket Advanced

Proceed with caution when modifying these advanced options. Default values should be enough in most cases.

Parameter
Description

Service endpoint

Optionally, Amazon S3 provides different types of service endpoints based on the region and access type.

  1. Select your bucket.

  2. Go to the Properties tab.

  3. Under Bucket ARN & URL, find the S3 endpoint URL.

Amazon Service Endpoint will usually be chosen automatically, so you should not normally have to fill this up. However, in case you need to override the default access point, you can do it here.

Queue

Parameter
Description

Region

Choose the region your queue is created in from the drop-down provided.

URL*

The URL of your existing Amazon SQS queue to send the data to.

  1. Go to the AWS Management Console.

  2. In the Search Bar, type SQS and click on Simple Queue Service (SQS).

  3. Click on Queues in the left panel.

  4. Locate your queue from the list and click it.

  5. The Queue URL will be displayed in the table under URL.

This is the correct URL format: https://sqs.region.localhost/awsaccountnumber/storedinenvvar

Access key ID

The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

  1. In the left panel, click on Users.

  2. Select your IAM user.

  3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

Note that this can be the same as in the bucket, in which case you don't need to repeat it here, or it can be different, depending on how you have configured your bucket & queue

Secret access key

This can be the same as for the bucket, in which case you don't need to repeat it here, or it can be different, depending on how you have configured your bucket & queue.

Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one. Note that this can be the same as in the bucket, in which case you don't need to repeat it here, or it can be different, depending on how you have configured your bucket & queue.

Event name

When you configure your bucket to send notifications to your SQS queue, you choose a name for those notification events. You can provide that name here to check the notifications to match that name when they are read by the Listener, or leave this empty to avoid such checks.

Queue Advanced

Proceed with caution when modifying these advanced options. Default values should be enough in most cases.

Parameter
Description

Service endpoint

If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default.

Maximum number of messages*

Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.

Visibility timeout*

Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is 30s, and the maximum value is 12h. The default value is 1h.

Wait time*

When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is 5s, and the maximum and default value is 20s.

General Advanced

Proceed with caution when modifying these advanced options. Default values should be enough in most cases.

Parameter
Description

Event batch size*

Enter a limit for the number of events allowed through per batch. The minimum value is 1, and the maximum and default value is 1000000.

Minimum retry time*

Set the minimum amount of time to wait before retrying. The default and minimum value is 1s, and the maximum value is 10m.

Maximum retry time*

Set the maximum amount of time to wait before retrying. The default value is 5m, and the maximum value is 10m. The minimum value is the one set in the parameter above.

Onum supports integration with . Select Amazon S3 from the list of Listener types and click Configuration to start.

The your data is stored in. This is the bucket name found in your Buckets area. You can fill this if you want to check that notifications come from that bucket, or leave it empty to avoid such checks.

Select the access key ID from your or click New secret to generate a new one.

Select the secret access key from your or click New secret to generate a new one.

Select the access key ID from your or click New secret to generate a new one.

Select the secret access key from your or click New secret to generate a new one.

Click Create labels to move on to the next step and define the required if needed.

Amazon S3
Labels
AWS bucket
Secrets
Secrets
Secrets
Secrets
here