Collect data from Amazon S3

Most recent version: v1.2.0

See the changelog of the Amazon S3 Listener here.

The Amazon S3 Listener is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Amazon Simple Storage Service is a fully managed object storage service. Users typically use it to store big files at a reasonable cost for long periods of time. In particular, it's commonly used as a data lake storage layer, storing files containing user events with some format/encoding/compression.

Amazon S3 also supports sending notifications to an SQS queue when new files are added to some bucket. You can see a sample notification here.

By leveraging all the above, our S3 Listener is able to react to new files being added to the bucket, get the files, and ingest their events into Onum. All that is needed is an existing SQS queue, an existing S3 bucket, and having the bucket correctly configured to send notifications to the queue.

Prerequisites

Before configuring and starting to send data with the Amazon S3 Listener, you need to take into consideration the following requirements:

• Your Amazon user needs at least permission to use the GetObject operation (S3) and the ReceiveMessage and DeleteMessageBatch operations (SQS Bucket) to make this Listener work.

• Cross-Region Configurations: Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets.

• Permissions: Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions.

• Object Key Name Filtering: If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded.

Amazon S3 Setup

You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new files are added.

1

Create an Amazon SQS Queue

• Sign in to the AWS Management Console and open the Amazon SQS console.

• Choose Create Queue and configure the queue settings as needed.

• After creating the queue, note its Amazon Resource Name (ARN), which follows this format: arn:aws:sqs:<region>:<account-id>:<queue-name>.

2

Modify the SQS Queue Policy to Allow S3 to Send Messages

1. In the Amazon SQS console, select your queue.

2. Navigate to the Access Policy tab and choose Edit.

3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details:

  {
    "Version": "2012-10-17",
    "Id": "S3ToSQSPolicy",
    "Statement": [
      {
        "Sid": "AllowS3Bucket",
        "Effect": "Allow",
        "Principal": {
          "Service": "s3.amazonaws.com"
        },
        "Action": "SQS:SendMessage",
        "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:s3:::<bucket-name>"
          },
          "StringEquals": {
            "aws:SourceAccount": "<account-id>"
          }
        }
      }
    ]
  }

Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue.

3

Configure S3 Event Notifications

1. Open the Amazon S3 console and select the bucket you want to configure.

2. Go to the Properties tab and find the "Event notifications" section.

3. Click on Create event notification.

4. Provide a descriptive name for the event notification.

5. In the Event types section, select All object create events or specify particular events that should trigger notifications.

6. In the Destination section, choose SQS Queue and select the queue you configured earlier.

7. Save the configuration.

4

Test the Configuration

1. Upload a new file to your S3 bucket.

2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Amazon S3 Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

In the Objects section, enter the required Compression method used in the ingested S3 files and Format of the ingested S3 files.

• Compression - This accepts the standard compression codecs (gzip, zlib, bzip2), none for no compression, and auto to autodetect the compression type from the file extension.

• Format - This currently accepts JSON array (a big JSON array containing a JSON object for each event), JSON lines (a JSON object representing an event on each line), CSV, and auto to autodetect the format from the file extension (.json or .jsonl, respectively).

  • If you select CSV, more options appear:

CSV Options

• Header Row - select true to include a header for your CSV rows.

• Delimiter - decide between comma, semicolon and tab.

• Text Encoding - refers to the scheme (or key) that maps the characters used to store and read the data in the CSV file: UTF-8, UTF-16, UTF-16 Little Endian, UTF-16 Big Endian, ISO 8859-1 (Latin-1), Windows-1252.

• Output Format - CSV or JSON.

  • JSON Output (outputFormat: "json"):

    • Converts CSV records to structured JSON objects

    • Field names are derived from header row (if present) or auto-generated (field_0, field_1, etc.)

    • Provides structured data for easier processing in pipelines

  • CSV Output (outputFormat: "csv"):

    • Preserves original CSV formatting

    • Each CSV record becomes a separate event containing the raw CSV line

    • Useful when you want to maintain the original CSV structure

• Trim Leading Space - select true to remove any whitespace characters that appear immediately before the first non-whitespace character in a cell.

• Lazy Quotes - select true to allow double quotes to appear in fields without strictly following the formal CSV escaping rules.

• Fields per Second - this number determines the speed and efficiency of S3 as it reads and interprets the data within the CSV file.

• Comment Character - use the hash symbol (#) to designate lines of text that should be ignored during the data parsing process.

5

Define the Bucket to listen from.

• Region* - Find this in your Buckets area, next to the name.

• Name - The AWS bucket your data is stored in. This is the bucket name found in your Buckets area. You can fill this if you want to check that notifications come from that bucket, or leave it empty to avoid such checks.

• Authentication Type*- Choose manual to enter your access key ID and secret access key manually in the parameters below, or auto to authenticate automatically. The default value is manual.

• Access key ID*- Select the access key ID from your Secrets or click New secret to generate a new one.

  The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

  1. In the left panel, click on Users.

  2. Select your IAM user.

  3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

• Secret access key*- Select the secret access key from your Secrets or click New secret to generate a new one. Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.

6

Proceed with caution when modifying the Bucket advanced options. Default values should be enough in most cases.

Optionally, Amazon S3 provides different types of service endpoints based on the region and access type.

1. Select your bucket.

2. Go to the Properties tab.

3. Under Bucket ARN & URL, find the S3 endpoint URL.

Amazon Service Endpoint will usually be chosen automatically, so you should not normally have to fill this up. However, in case you need to override the default access point, you can do it here.

7

In the Queue section, choose the region your queue is created in from the dropdown provided.

8

Then, enter the URL of your existing Amazon SQS queue to send the data to.

1. Go to the AWS Management Console.

2. In the Search Bar, type SQS and click on Simple Queue Service (SQS).

3. Click on Queues in the left panel.

4. Locate your queue from the list and click it.

5. The Queue URL will be displayed in the table under URL.

This is the correct URL format: https://sqs.region.localhost/awsaccountnumber/storedinenvvar

9

Choose your Authentication Type*

Choose manual to enter your access key ID and secret access key manually in the parameters below, or auto to authenticate automatically.

10

If you have configured your bucket and queue to require different Access Key IDs and Secret Access Keys, enter them here. If these are the same as your bucket, you don't need to repeat them here.

11

Proceed with caution when modifying the Queue advanced options. Default values should be enough in most cases.

• Service endpoint - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default.

• Maximum number of messages* - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.

• Visibility timeout* - Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is 30s, and the maximum value is 12h. The default value is 1h.

• Wait time*- When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is 5s, and the maximum and default value is 20s.

12

Proceed with caution when modifying the General advanced options. Default values should be enough in most cases.

• Event batch size*- Enter a limit for the number of events allowed through per batch. The minimum value is 1, and the maximum and default value is 1000000.

• Minimum retry time* - Set the minimum amount of time to wait before retrying. The default and minimum value is 1s, and the maximum value is 10m.

• Maximum retry time* - Set the maximum amount of time to wait before retrying. The default value is 5m, and the maximum value is 10m. The minimum value is the one set in the parameter above.

13

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

Click Create listener when you're done.