Collect S3 Application Logs

You can use the Amazon S3 Listener to collect AWS application logs, such as

• AWS CloudTrail

• AWS CloudWatch

• AWS WAF

Prerequisites

Before configuring and starting to send data with the Amazon S3 Listener, you need to take into consideration the following requirements:

• Your Amazon user needs at least permission to use the GetObject operation (S3) and the ReceiveMessage and DeleteMessageBatch operations (SQS Bucket) to make this Listener work.

• Cross-Region Configurations: Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets.

• Permissions: Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions.

• Object Key Name Filtering: If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded.

Amazon S3 Setup

You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new logs are added.

1

Create an Amazon SQS Queue

• Sign in to the AWS Management Console and open the Amazon SQS console.

• Choose Create Queue and configure the queue settings as needed.

• After creating the queue, note its Amazon Resource Name (ARN), which follows this format: arn:aws:sqs:<region>:<account-id>:<queue-name>.

2

Modify the SQS Queue Policy to Allow S3 to Send Messages

1. In the Amazon SQS console, select your queue.

2. Navigate to the Access Policy tab and choose Edit.

3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details:

  {
    "Version": "2012-10-17",
    "Id": "S3ToSQSPolicy",
    "Statement": [
      {
        "Sid": "AllowS3Bucket",
        "Effect": "Allow",
        "Principal": {
          "Service": "s3.amazonaws.com"
        },
        "Action": "SQS:SendMessage",
        "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:s3:::<bucket-name>"
          },
          "StringEquals": {
            "aws:SourceAccount": "<account-id>"
          }
        }
      }
    ]
  }

Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue.

3

S3 Event Notification Rules for Logs

Get notifications for Cloudwatch, Cloudtrail or WAF logs.

• Open the Amazon S3 console and select the bucket you want to configure.

• Go to the Properties tab and find the "Event notifications" section and configure according to type of log below.

Cloudwatch

{
  "QueueConfigurations": [
    {
      "Id": "CloudWatchLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "cloudwatch-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".log"
            }
          ]
        }
      }
    }
  ]
}

Cloudtrail

{
  "QueueConfigurations": [
    {
      "Id": "CloudTrailLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "AWSLogs/"
            },
            {
              "Name": "suffix",
              "Value": ".json.gz"
            }
          ]
        }
      }
    }
  ]
}

WAF

{
  "QueueConfigurations": [
    {
      "Id": "WAFLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "aws-waf-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".gz"
            }
          ]
        }
      }
    }
  ]
}

4

Test the Configuration

1. Upload a new file to your S3 bucket.

2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Amazon S3 Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

In the Objects section, enter the required Compression method used in the ingested S3 files and Format of the ingested S3 files.

• Compression - This accepts the standard compression codecs (gzip, zlib, bzip2), none for no compression, and auto to autodetect the compression type from the file extension.

• Format - This currently accepts JSON, JSON lines (a JSON object representing an event on each line), and CSV.

If you select JSON or CSV, more options appear:

JSON Options

• Path - Enter the path of the JSON element you want to retrieve. The path are keys separated by dots. For example one.two.three will select the element in {"one":{"two":{"three":[1,2,3]}}. To select the root, you can leave the path empty or enter a single dot . (default option).

For Cloudwatch and Cloudtrail logs, you must enter the .Records path.

• Array Unroll - Activate this toggle if you want to generate one event for each element in the array. The element that the path points to must be a JSON array.

CSV Options

• Header Row - select true to include a header for your CSV rows.

• Delimiter - decide between comma, semicolon and tab.

• Text Encoding - refers to the scheme (or key) that maps the characters used to store and read the data in the CSV file: UTF-8, UTF-16, UTF-16 Little Endian, UTF-16 Big Endian, ISO 8859-1 (Latin-1), Windows-1252.

• Output Format - CSV or JSON.

  • JSON Output (outputFormat: "json"):

    • Converts CSV records to structured JSON objects

    • Field names are derived from header row (if present) or auto-generated (field_0, field_1, etc.)

    • Provides structured data for easier processing in pipelines

  • CSV Output (outputFormat: "csv"):

    • Preserves original CSV formatting

    • Each CSV record becomes a separate event containing the raw CSV line

    • Useful when you want to maintain the original CSV structure

• Trim Leading Space - select true to remove any whitespace characters that appear immediately before the first non-whitespace character in a cell.

• Lazy Quotes - select true to allow double quotes to appear in fields without strictly following the formal CSV escaping rules.

• Fields per Second - this number determines the speed and efficiency of S3 as it reads and interprets the data within the CSV file.

• Comment Character - use the hash symbol (#) to designate lines of text that should be ignored during the data parsing process.

5

Define the Bucket to listen from.

• Region* - Find this in your Buckets area, next to the name.

• Name - The AWS bucket your data is stored in. This is the bucket name found in your Buckets area. You can fill this if you want to check that notifications come from that bucket, or leave it empty to avoid such checks.

• Authentication Type*- Choose manual to enter your access key ID and secret access key manually in the parameters below, or auto to authenticate automatically. The default value is manual.

Note for Cloud Deployments: If you select the auto authentication and have a Cloud Deployment, please contact Support to request the ARN information for your Onum-managed instance.

• Access key ID*- Select the access key ID from your Secrets or click New secret to generate a new one.

  The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

  1. In the left panel, click on Users.

  2. Select your IAM user.

  3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

• Secret access key*- Select the secret access key from your Secrets or click New secret to generate a new one. Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.

6

Access external S3 and SQS resources using AssumeRole

This role should have the following permissions to access both the S3 bucket and the configured SQS notification queue:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetBucketNotification"
      ],
      "Resource": [
        "<S3 bucket ARN>"
      ]
    },
    {
      "Sid": "SQSAccess",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:DeleteMessageBatch",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl"
      ],
      "Resource": [
        "<SQS queue ARN>"
      ]
    }
  ]
}

AWS credentials and AssumeRole configuration associated can be configured individually for S3 and SQS but if SQS credentials are not configured it will use the ones for S3, AssumeRole configuration included.

The configuration options are as follows:

• Role ARN* - Amazon Resource Name used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use.

• External ID* - shared secret used to authenticate the usage of this role.

• Role Session - name of the session, used to audit usage of this role ( s3-listener by default)

• STS Region - if not set it will use the bucket or the queue region.

• STS Session Duration - how much the AssumeRole session will last before reauthentication. Uses Golang duration strings, like 1s, 1m, 1h. If not Set, it uses the maximum session duration configured for that role. The minimum duration is 15m and the maximum is configured in the role, no longer than 12h.

7

Proceed with caution when modifying the Bucket advanced options. Default values should be enough in most cases.

Optionally, Amazon S3 provides different types of service endpoints based on the region and access type.

1. Select your bucket.

2. Go to the Properties tab.

3. Under Bucket ARN & URL, find the S3 endpoint URL.

Amazon Service Endpoint will usually be chosen automatically, so you should not normally have to fill this up. However, in case you need to override the default access point, you can do it here.

8

In the Queue section, choose the region your queue is created in from the dropdown provided.

9

Then, enter the URL of your existing Amazon SQS queue to send the data to.

1. Go to the AWS Management Console.

2. In the Search Bar, type SQS and click on Simple Queue Service (SQS).

3. Click on Queues in the left panel.

4. Locate your queue from the list and click it.

5. The Queue URL will be displayed in the table under URL.

This is the correct URL format: https://sqs.region.localhost/awsaccountnumber/storedinenvvar

10

Choose your Authentication Type*

Choose manual to enter your access key ID and secret access key manually in the parameters below, or auto to authenticate automatically.

11

If you have configured your bucket and queue to require different Access Key IDs and Secret Access Keys, enter them here. If these are the same as your bucket, you don't need to repeat them here.

12

Proceed with caution when modifying the Queue advanced options. Default values should be enough in most cases.

• Service endpoint - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default.

• Maximum number of messages* - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.

• Visibility timeout* - Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is 30s, and the maximum value is 12h. The default value is 1h.

• Wait time*- When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is 5s, and the maximum and default value is 20s.

13

Proceed with caution when modifying the General advanced options. Default values should be enough in most cases.

• Event batch size*- Enter a limit for the number of events allowed through per batch. The minimum value is 1, and the maximum and default value is 1000000.

• Minimum retry time* - Set the minimum amount of time to wait before retrying. The default and minimum value is 1s, and the maximum value is 10m.

• Maximum retry time* - Set the maximum amount of time to wait before retrying. The default value is 5m, and the maximum value is 10m. The minimum value is the one set in the parameter above.

14

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

15

Click Create listener when you're done.