Collect Amazon VPC Flow Logs

See the changelog of the Amazon VPC Flow Logs Listener here.

The Amazon VPC Flow Logs Listener is only available in certain Tenants. Get in touch with us if you don't see it and want to access it.

Overview

You can use the Amazon VPC Flow Logs Listener to receive and parse AWS VPC Flow Logs stored in S3, delivered via SQS notifications.

Prerequisites

Before configuring and starting to send data to the Amazon VPC Flow Logs Listener, you need to take into consideration the following requirements:

• Your Amazon user needs at least permission to use the GetObject operation (S3) and the ReceiveMessage and DeleteMessageBatch operations (SQS Bucket) to make this Listener work.

• Cross-Region Configurations - Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets.

• Permissions - Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions.

• Object Key Name Filtering - If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded.

When S3 events flow through SNS before reaching SQS, enabling Raw Message Delivery on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery here.

Amazon S3 Setup

You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new logs are added.

1

Create an Amazon SQS Queue

• Sign in to the AWS Management Console and open the Amazon SQS console.

• Choose Create Queue and configure the queue settings as needed.

• After creating the queue, note its Amazon Resource Name (ARN), which follows this format: arn:aws:sqs:<region>:<account-id>:<queue-name>.

2

Modify the SQS Queue Policy to Allow S3 to Send Messages

1. In the Amazon SQS console, select your queue.

2. Navigate to the Access Policy tab and choose Edit.

3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details:

  {
    "Version": "2012-10-17",
    "Id": "S3ToSQSPolicy",
    "Statement": [
      {
        "Sid": "AllowS3Bucket",
        "Effect": "Allow",
        "Principal": {
          "Service": "s3.amazonaws.com"
        },
        "Action": "SQS:SendMessage",
        "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:s3:::<bucket-name>"
          },
          "StringEquals": {
            "aws:SourceAccount": "<account-id>"
          }
        }
      }
    ]
  }

Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue.

3

S3 Event Notification Rules for Logs

Get notifications for CloudWatch, CloudTrail or WAF logs.

• Open the Amazon S3 console and select the bucket you want to configure.

• Go to the Properties tab and find the Event notifications section and configure according to type of log below.

Cloudwatch

{
  "QueueConfigurations": [
    {
      "Id": "CloudWatchLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "cloudwatch-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".log"
            }
          ]
        }
      }
    }
  ]
}

Cloudtrail

{
  "QueueConfigurations": [
    {
      "Id": "CloudTrailLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "AWSLogs/"
            },
            {
              "Name": "suffix",
              "Value": ".json.gz"
            }
          ]
        }
      }
    }
  ]
}

WAF

{
  "QueueConfigurations": [
    {
      "Id": "WAFLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "aws-waf-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".gz"
            }
          ]
        }
      }
    }
  ]
}

4

Test the Configuration

1. Upload a new file to your S3 bucket.

2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly.

Onum setup

1

In Onum, go to the Listeners area and click New listener. Select the Amazon VPC Flow Logs Listener from the list.

2

Enter a Name for the Listener. Optionally, add a Description and some Tags to identify the Listener.

3

In the Objects section, enter the required Path pattern.

This allows you to filter which S3 objects are processed based on their key (path). It accepts Go regular expressions (RE2 syntax), which are matched against the full object key after URL-decoding. Objects whose key does not match the pattern are silently skipped and their SQS notification is still acknowledged, preventing poison-pill scenarios where non-processable files (e.g. CloudTrail digest files) block the queue.

This field is required and the default value is AWSLogs/./vpcflowlogs/.* Find some examples below:

RegEx	Description
.*	Matches all keys (default, backward-compatible)
^data/.*.json(.gz)?$	Only JSON (or gzipped JSON) files under data/
^(?!.Digest).$	Excludes CloudTrail digest files
CloudTrail/[^/]+/.*.json.gz$	Only CloudTrail log files (not digests)
^logs/	Only objects with the logs/ prefix
.(csv|parquet)$	Only CSV or Parquet files regardless of path
^[^/]+/[^/]+/year=2024/	Only objects partitioned under year=2024

4

In the Bucket section, enter the following:

• Region - Choose your S3 region.

• Name - You can enter the bucket name you want to get events from. If you left it empty, the system will automatically extract it from your SQS notifications. In this case, you can use more than one bucket, as long as credentials are valid for all them and they all notify to the queue indicated below.

• Authentication type - Choose between Manual or Auto:

  • Manual - Choose this option if you want to enter your Access Key ID and Secret Key manually from your secret list. To define a new secret, click the Access key ID or Secret access key field and select New secret. In the window that appears, give your secret a Name and turn off the Expiration date toggle if not needed. Then, click Add new value and paste the token you generated. Click Save when you're done and select your secret from the list.

  • Auto - Choose this option to authenticate automatically.

5

You can enable the Assume Role option, which provides a more secure and scalable alternative to traditional credential-based authentication.

Instead of using static credentials, this method leverages an Amazon Resource Name (ARN) to define and assume an IAM role with the necessary permissions to access your SQS queue. By introducing the ARN into the Listener configuration, Falcon Onum can securely assume the designated role and interact with your AWS resources in a controlled and auditable way. To enable access, we need cross-account access from our AWS infrastructure to yours. This is a one-time setup.

First, you must create an IAM role in your AWS account (or use an existing one) with the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::721751517261:role/<CustomersRole>"
      },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}

This trust policy allows our AWS account to assume the role. You can attach whatever permissions you want to the role — the trust policy only controls who can assume it, not what it can do.

Once the IAM role is ready, please contact us to share your full role ARN (e.g. arn:aws:iam::012345678901:role/YourRoleName) so we can complete the configuration on our end. Then, configure the Assume Role options as follows:

• Role ARN - Amazon Resource Name used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use.

• External ID - Shared secret used to authenticate the usage of this role.

• Role Session - Name of the session, used to audit usage of this role ( s3-listener by default)

• STS Region - If not set, it will use the bucket or the queue region.

• STS Session Duration - How much the Assume Role session will last before reauthentication. Uses Golang duration strings, like 1s, 1m, 1h. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h.

6

In the Bucket advanced section, you may enter a specific Service endpoint if you don't want to use the default regional S3 service endpoint.

7

In the Queue section, enter the following:

• Region - Choose the region your queue is created in from the dropdown provided.

• URL - Enter the URL of your existing Amazon SQS queue to send the data to. To find it:

  • Go to the AWS Management Console.

  • In the Search Bar, type SQS and click on Simple Queue Service (SQS).

  • Click on Queues in the left panel.

  • Locate your queue from the list and click it.

  • The Queue URL will be displayed in the table under URL. This is the correct URL format: https://sqs.region.localhost/awsaccountnumber/storedinenvvar

• Authentication type - Choose between Manual or Auto:

  • Manual - Choose this option if you want to enter your Access Key ID and Secret Key manually from your secret list. To define a new secret, click the Access key ID or Secret access key field and select New secret. In the window that appears, give your secret a Name and turn off the Expiration date toggle if not needed. Then, click Add new value and paste the token you generated. Click Save when you're done and select your secret from the list.

  • Auto - Choose this option to authenticate automatically.

• Event name - The event name specified when configuring bucket notifications. If not empty, received notifications will be filtered out if they have a different event name.

8

Enable the Assume Role option if needed. Read more about this above.

9

In the Queue advanced section, enter the following:

Proceed with caution when modifying the Queue advanced options. Default values should be enough in most cases.

• Service endpoint - Enter a specific SQS endpoint if you don't want to use the default regional service endpoint.

• Maximum number of messages - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.

• Visibility timeout - Set how many time to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is 30s, and the maximum value is 12h. The default value is 30m.

• Wait time - When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is 5s, and the maximum and default value is 20s.

10

In the General advanced section, configure the following options:

Proceed with caution when modifying the General advanced options. Default values should be enough in most cases.

• Event batch size - Enter a limit for the number of events allowed through per batch. The minimum value is 1, and the maximum and default value is 1000000.

• Minimum retry time - Set the minimum amount of time to wait before retrying. The minimum and default value is 1s, and the maximum value is 10m.

• Maximum retry time - Set the maximum amount of time to wait before retrying. The default value is 5m, and the maximum value is 10m. The minimum value is the one set in the parameter above.

11

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

12

Click Create listener when you're done.

Output Ports

The Amazon VPC Flow Logs Listener has a single output port:

• Default port - Events are sent through this port if no error occurs while processing them.