Collect data from CrowdStrike Falcon Data Replicator

See the changelog of the CrowdStrike Falcon Data Replicator Listener here.

Overview

You can use the CrowdStrike Falcon Data Replicator Listener to receive and process CrowdStrike Falcon Data Replicator (FDR) events stored in S3, delivered via custom SQS notifications.

Prerequisites

Before configuring and starting to send data with the Amazon S3 Listener, you need to take into consideration the following requirements:

• Your Amazon user needs at least permission to use the GetObject operation (S3) and the ReceiveMessage and DeleteMessageBatch operations (SQS Bucket) to make this Listener work.

• Cross-Region Configurations - Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets.

• Permissions - Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions.

• Object Key Name Filtering - If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded.

When S3 events flow through SNS before reaching SQS, enabling Raw Message Delivery on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery here.

Amazon S3 Setup

You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new logs are added.

1

Create an Amazon SQS Queue

• Sign in to the AWS Management Console and open the Amazon SQS console.

• Choose Create Queue and configure the queue settings as needed.

• After creating the queue, note its Amazon Resource Name (ARN), which follows this format: arn:aws:sqs:<region>:<account-id>:<queue-name>.

2

Modify the SQS Queue Policy to Allow S3 to Send Messages

1. In the Amazon SQS console, select your queue.

2. Navigate to the Access Policy tab and choose Edit.

3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details:

  {
    "Version": "2012-10-17",
    "Id": "S3ToSQSPolicy",
    "Statement": [
      {
        "Sid": "AllowS3Bucket",
        "Effect": "Allow",
        "Principal": {
          "Service": "s3.amazonaws.com"
        },
        "Action": "SQS:SendMessage",
        "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:s3:::<bucket-name>"
          },
          "StringEquals": {
            "aws:SourceAccount": "<account-id>"
          }
        }
      }
    ]
  }

Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue.

3

S3 Event Notification Rules for Logs

Get notifications for CloudWatch, CloudTrail or WAF logs.

• Open the Amazon S3 console and select the bucket you want to configure.

• Go to the Properties tab and find the Event notifications section and configure according to type of log below.

Cloudwatch

{
  "QueueConfigurations": [
    {
      "Id": "CloudWatchLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "cloudwatch-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".log"
            }
          ]
        }
      }
    }
  ]
}

Cloudtrail

{
  "QueueConfigurations": [
    {
      "Id": "CloudTrailLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "AWSLogs/"
            },
            {
              "Name": "suffix",
              "Value": ".json.gz"
            }
          ]
        }
      }
    }
  ]
}

WAF

{
  "QueueConfigurations": [
    {
      "Id": "WAFLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "aws-waf-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".gz"
            }
          ]
        }
      }
    }
  ]
}

4

Test the Configuration

1. Upload a new file to your S3 bucket.

2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly.

Onum setup

First, you must configure a new CrowdStrike Falcon Data Replicator Listener in Onum:

1

In Onum, go to the Listeners area and click New listener. Select the CrowdStrike Falcon Data Replicator Listener from the list.

2

Enter a Name for the Listener. Optionally, add a Description and some Tags to identify the Listener.

3

In the Connection settings section, you can enter the Bucket name you want to get events from. If you left it empty, the system will automatically extract it from your SQS notifications. In this case, you can use more than one bucket, as long as credentials are valid for all them and they all notify to the queue indicated below.

In case you copy the whole FDR feed URL, make sure you only enter the bucket name in this field, removing the s3:// part and the path. Otherwise, you will get an error.

4

Choose your S3 region in the Select region field.

5

In the Authentication type section, choose between Manual or Auto:

• Manual - Choose this option if you want to enter your Access Key ID and Secret Key manually from your secret list. To define a new secret, click the Select access key ID or Select secret key field and select New secret. In the window that appears, give your secret a Name and turn off the Expiration date toggle if not needed. Then, click Add new value and paste the token you generated. Click Save when you're done and select your secret from the list.

• Auto - Choose this option to authenticate automatically.

6

You can enable the Assume Role option, which provides a more secure and scalable alternative to traditional credential-based authentication.

Instead of using static credentials, this method leverages an Amazon Resource Name (ARN) to define and assume an IAM role with the necessary permissions to access your SQS queue. By introducing the ARN into the Listener configuration, Falcon Onum can securely assume the designated role and interact with your AWS resources in a controlled and auditable way. To enable access, we need cross-account access from our AWS infrastructure to yours. This is a one-time setup.

First, you must create an IAM role in your AWS account (or use an existing one) with the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::721751517261:role/<CustomersRole>"
      },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}

This trust policy allows our AWS account to assume the role. You can attach whatever permissions you want to the role — the trust policy only controls who can assume it, not what it can do.

Once the IAM role is ready, please contact us to share your full ARN (e.g. arn:aws:iam::012345678901:role/YourRoleName) so we can complete the configuration on our end. Then, configure the Assume Role options as follows:

• Role ARN - Amazon Resource Name used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use.

• External ID - Shared secret used to authenticate the usage of this role.

• Role Session - Name of the session, used to audit usage of this role ( s3-listener by default)

• STS Region - If not set, it will use the bucket or the queue region.

• STS Session Duration - How much the Assume Role session will last before reauthentication. Uses Golang duration strings, like 1s, 1m, 1h. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h.

7

In the SQS queue connection section, enter the following:

• Queue URL - Enter the URL of your existing Amazon SQS queue to send the data to. To find it:

  • Go to the AWS Management Console.

  • In the Search Bar, type SQS and click on Simple Queue Service (SQS).

  • Click on Queues in the left panel.

  • Locate your queue from the list and click it.

  • The Queue URL will be displayed in the table under URL. This is the correct URL format: https://sqs.region.localhost/awsaccountnumber/storedinenvvar

• Select region - Choose the region your queue is created in from the dropdown provided.

8

Activate the Use authentication toggle if you want to specify a set of credentials different than your S3 authentication settings. In this case, choose between Manual or Auto:

• Manual - Choose this option if you want to enter your Access Key ID and Secret Key manually from your secret list. To define a new secret, click the Select access key ID or Select secret key field and select New secret. In the window that appears, give your secret a Name and turn off the Expiration date toggle if not needed. Then, click Add new value and paste the token you generated. Click Save when you're done and select your secret from the list.

• Auto - Choose this option to authenticate automatically.

9

Enable the Assume Role option if needed. Read more about this above.

10

In the SQS consumer settings section, enter the following:

• Wait time - When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is 5s, and the maximum and default value is 20s.

• Maximum number of messages - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.

• Visibility timeout - Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is 30, and the maximum value is 43200. The default value is 1800.

11

In the Processing settings section, configure the following options:

• Batch configuration - Enter a limit for the number of events allowed through per batch in the Event batch size field. The minimum value is 1, and the maximum and default value is 1000000.

• Retry configuration

  • Minimum retry time - Set the minimum amount of seconds to wait before retrying. The minimum value is 1, and the maximum value is 600. The default value is 300.

  • Maximum retry time - Set the maximum amount of seconds to wait before retrying. The default value is 1, and the maximum value is 600. The default value is 300.

  • Max consecutive ignorable errors - Maximum number of consecutive ignorable errors (e.g. malformed records, connection resets) allowed per S3 object before the object is skipped and the SQS message is acknowledged. Must be at least 1.

12

In the Content identification (objects) section, enter the required Path pattern.

This allows you to filter which S3 objects are processed based on their key (path). It accepts Go regular expressions (RE2 syntax), which are matched against the full object key after URL-decoding. Objects whose key does not match the pattern are silently skipped and their SQS notification is still acknowledged, preventing poison-pill scenarios where non-processable files (e.g. CloudTrail digest files) block the queue.

This field is required and the default value is .* (matches everything, backward-compatible with previous versions that did not have this filter). Find some examples below:

RegEx	Description
.*	Matches all keys (default, backward-compatible)
^data/.*.json(.gz)?$	Only JSON (or gzipped JSON) files under data/
^(?!.Digest).$	Excludes CloudTrail digest files
CloudTrail/[^/]+/.*.json.gz$	Only CloudTrail log files (not digests)
^logs/	Only objects with the logs/ prefix
.(csv|parquet)$	Only CSV or Parquet files regardless of path
^[^/]+/[^/]+/year=2024/	Only objects partitioned under year=2024

13

Optionally, open the Advanced configuration section and configure the following settings:

• S3 custom endpoint - Enter a specific S3 endpoint if you don't want to use the default regional service endpoint.

• SQS custom endpoint - Enter a specific SQS endpoint if you don't want to use the default regional service endpoint.

14

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

15

Click Create listener when you're done.

Output Ports

The Crowdstrike Falcon Data Replicator Listener has a single output port:

• Default port - Events are sent through this port if no error occurs while processing them.