> For the complete documentation index, see [llms.txt](https://docs.onum.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.onum.com/listeners/listener-integrations/collect-data-from-crowdstrike-falcon-data-replicator.md).

# Collect data from CrowdStrike Falcon Data Replicator

{% hint style="info" %}
See the changelog of the **CrowdStrike Falcon Data Replicator** Listener [here](/listeners-changelog/crowdstrike-falcon-data-replicator-listener.md).
{% endhint %}

## Overview <a href="#data-sink-configuration" id="data-sink-configuration"></a>

You can use the **CrowdStrike Falcon Data Replicator** Listener to receive and process CrowdStrike Falcon Data Replicator (FDR) events stored in S3, delivered via custom SQS notifications.

## Prerequisites

Before configuring and starting to send data with the **Amazon S3** Listener, you need to take into consideration the following requirements:

* Your Amazon user needs at least permission to use the `GetObject` operation (S3) and the  `ReceiveMessage` and `DeleteMessageBatch` operations (SQS Bucket) to make this Listener work.
* **Cross-Region Configurations** - Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets.
* **Permissions** - Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions.
* **Object Key Name Filtering** - If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded.

{% hint style="warning" %}
When S3 events flow through SNS before reaching SQS, enabling [**Raw Message Delivery**](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery [here](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html).
{% endhint %}

## Amazon S3 Setup

You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new logs are added.

{% stepper %}
{% step %}

#### **Create an Amazon SQS Queue**

* Sign in to the AWS Management Console and open the Amazon SQS console.
* Choose **Create Queue** and configure the queue settings as needed.
* After creating the queue, note its Amazon Resource Name (ARN), which follows this format: `arn:aws:sqs:<region>:<account-id>:<queue-name>`.
  {% endstep %}

{% step %}

#### Modify the SQS Queue Policy to Allow S3 to Send Messages

1. In the Amazon SQS console, select your queue.
2. Navigate to the **Access Policy** tab and choose **Edit**.
3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details:

```json
  {
    "Version": "2012-10-17",
    "Id": "S3ToSQSPolicy",
    "Statement": [
      {
        "Sid": "AllowS3Bucket",
        "Effect": "Allow",
        "Principal": {
          "Service": "s3.amazonaws.com"
        },
        "Action": "SQS:SendMessage",
        "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:s3:::<bucket-name>"
          },
          "StringEquals": {
            "aws:SourceAccount": "<account-id>"
          }
        }
      }
    ]
  }
```

Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue.
{% endstep %}

{% step %}

#### S3 Event Notification Rules for Logs&#x20;

Get notifications for CloudWatch, CloudTrail or WAF logs.

* Open the Amazon S3 console and select the bucket you want to configure.
* Go to the **Properties** tab and find the **Event notifications** section and configure according to type of log below.

{% code title="Cloudwatch" %}

```json
{
  "QueueConfigurations": [
    {
      "Id": "CloudWatchLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "cloudwatch-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".log"
            }
          ]
        }
      }
    }
  ]
}
```

{% endcode %}

{% code title="Cloudtrail" %}

```json
{
  "QueueConfigurations": [
    {
      "Id": "CloudTrailLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "AWSLogs/"
            },
            {
              "Name": "suffix",
              "Value": ".json.gz"
            }
          ]
        }
      }
    }
  ]
}
```

{% endcode %}

{% code title="WAF" %}

```json
{
  "QueueConfigurations": [
    {
      "Id": "WAFLogNotification",
      "QueueArn": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "aws-waf-logs/"
            },
            {
              "Name": "suffix",
              "Value": ".gz"
            }
          ]
        }
      }
    }
  ]
}
```

{% endcode %}
{% endstep %}

{% step %}

#### Test the Configuration

1. Upload a new file to your S3 bucket.
2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly.
   {% endstep %}
   {% endstepper %}

## Onum setup <a href="#data-sink-configuration" id="data-sink-configuration"></a>

First, you must configure a new **CrowdStrike Falcon Data Replicator** Listener in Onum:

{% stepper %}
{% step %}
In Onum, go to the **Listeners** area and click **New listener**. Select the **CrowdStrike Falcon Data Replicator** Listener from the list.
{% endstep %}

{% step %}
Enter a **Name** for the Listener. Optionally, add a **Description** and some **Tags** to identify the Listener.
{% endstep %}

{% step %}
In the **Connection settings** section, you can enter the **Bucket name** you want to get events from. If you left it empty, the system will automatically extract it from your SQS notifications. In this case, you can use more than one bucket, as long as credentials are valid for all them and they all notify to the queue indicated below.

{% hint style="warning" %}
In case you copy the whole FDR feed URL, make sure you only enter the bucket name in this field, removing the `s3://` part and the path. Otherwise, you will get an error.
{% endhint %}
{% endstep %}

{% step %}
Choose your S3 region in the **Select region** field.
{% endstep %}

{% step %}
In the **Authentication type** section, choose between **Manual** or **Auto**:

* **Manual** - Choose this option if you want to enter your Access Key ID and Secret Key manually from your secret list. To define a new secret, click the **Select access key ID** or **Select secret key** field and select **New secret**. In the window that appears, give your secret a **Name** and turn off the **Expiration date** toggle if not needed. Then, click **Add new value** and paste the token you generated. Click **Save** when you're done and select your secret from the list.
* **Auto** - Choose this option to authenticate automatically.
  {% endstep %}

{% step %}
You can enable the **Assume Role** option, which provides a more secure and scalable alternative to traditional credential-based authentication.

Instead of using static credentials, this method leverages an Amazon Resource Name (ARN) to define and assume an IAM role with the necessary permissions to access your SQS queue. By introducing the ARN into the Listener configuration, Falcon Onum can securely assume the designated role and interact with your AWS resources in a controlled and auditable way. To enable access, we need cross-account access from our AWS infrastructure to yours. This is a one-time setup.&#x20;

First, you must create an IAM role in your AWS account (or use an existing one) with the following trust policy:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::721751517261:role/<CustomersRole>"
      },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}
```

{% hint style="warning" %}
This trust policy allows our AWS account to assume the role. You can attach whatever permissions you want to the role — the trust policy only controls who can assume it, not what it can do.
{% endhint %}

Once the IAM role is ready, please [contact us](broken://spaces/cSjT21I4EUhzghjc1rER/pages/nW8oZycpfVthtI5KYdQs) to share your full ARN (e.g. `arn:aws:iam::012345678901:role/YourRoleName`) so we can complete the configuration on our end. Then, configure the **Assume Role** options as follows:

* **Role ARN** -  **Amazon Resource Name** used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use.
* **External ID** - Shared secret used to authenticate the usage of this role.
* **Role Session** - Name of the session, used to audit usage of this role ( `s3-listener` by default)
* **STS Region** - If not set, it will use the bucket or the queue region.
* **STS Session Duration** - How much the Assume Role session will last before reauthentication. Uses Golang duration strings, like `1s`, `1m`, `1h`. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h.
  {% endstep %}

{% step %}
In the **SQS queue connection** section, enter the following:

* **Queue URL** - Enter the URL of your existing Amazon SQS queue to send the data to. To find it:
  * Go to the **AWS Management Console**.
  * In the **Search Bar**, type **SQS** and click on **Simple Queue Service (SQS)**.
  * Click on **Queues** in the left panel.
  * Locate your queue from the list and click it.
  * The **Queue URL** will be displayed in the table under **URL**. This is the correct URL format: `https://sqs.region.localhost/awsaccountnumber/storedinenvvar`<br>
* **Select region** - Choose the region your queue is created in from the dropdown provided.
  {% endstep %}

{% step %}
Activate the **Use authentication** toggle if you want to specify a set of credentials different than your S3 authentication settings. In this case, choose between **Manual** or **Auto**:

* **Manual** - Choose this option if you want to enter your Access Key ID and Secret Key manually from your secret list. To define a new secret, click the **Select access key ID** or **Select secret key** field and select **New secret**. In the window that appears, give your secret a **Name** and turn off the **Expiration date** toggle if not needed. Then, click **Add new value** and paste the token you generated. Click **Save** when you're done and select your secret from the list.
* **Auto** - Choose this option to authenticate automatically.
  {% endstep %}

{% step %}
Enable the **Assume Role** option if needed. Read more about this above.
{% endstep %}

{% step %}
In the **SQS consumer settings** section, enter the following:

* **Wait time** - When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is `5s`, and the maximum and default value is `20s`.
* **Maximum number of messages** - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `1`, and the maximum and default value is `10`.
* **Visibility timeout** - Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is `30`, and the maximum value is `43200`. The default value is `1800`.
  {% endstep %}

{% step %}
In the **Processing settings** section, configure the following options:

* **Batch configuration** - Enter a limit for the number of events allowed through per batch in the **Event batch size** field. The minimum value is `1`, and the maximum and default value is `1000000`.
* **Retry configuration**
  * **Minimum retry time** - Set the minimum amount of seconds to wait before retrying. The minimum value is `1`, and the maximum value is `600`. The default value is `300`.
  * **Maximum retry time** - Set the maximum amount of seconds to wait before retrying. The default value is `1`, and the maximum value is `600`. The default value is `300`.
  * **Max consecutive ignorable errors** - Maximum number of consecutive ignorable errors (e.g. malformed records, connection resets) allowed per S3 object before the object is skipped and the SQS message is acknowledged. Must be at least `1`.
    {% endstep %}

{% step %}
In the **Content identification (objects)** section, enter the required **Path pattern**.

This allows you to filter which S3 objects are processed based on their key (path). It accepts Go regular expressions (RE2 syntax), which are matched against the full object key after URL-decoding. Objects whose key does not match the pattern are silently skipped and their SQS notification is still acknowledged, preventing poison-pill scenarios where non-processable files (e.g. CloudTrail digest files) block the queue.&#x20;

This field is required and the default value is `.*` (matches everything, backward-compatible with previous versions that did not have this filter). Find some examples below:

<table data-search="false"><thead><tr><th width="163.25">RegEx</th><th>Description</th></tr></thead><tbody><tr><td><code>.*</code></td><td>Matches all keys (default, backward-compatible)</td></tr><tr><td><code>^data/.*.json(.gz)?$</code></td><td>Only JSON (or gzipped JSON) files under <code>data/</code></td></tr><tr><td><code>^(?!.Digest).$</code></td><td>Excludes CloudTrail digest files</td></tr><tr><td><code>CloudTrail/[^/]+/.*.json.gz$</code></td><td>Only CloudTrail log files (not digests)</td></tr><tr><td><code>^logs/</code></td><td>Only objects with the <code>logs/</code> prefix</td></tr><tr><td><code>.(csv|parquet)$</code></td><td>Only CSV or Parquet files regardless of path</td></tr><tr><td><code>^[^/]+/[^/]+/year=2024/</code></td><td>Only objects partitioned under <code>year=2024</code></td></tr></tbody></table>
{% endstep %}

{% step %}
Optionally, open the **Advanced configuration** section and configure the following settings:

* **S3 custom endpoint** - Enter a specific S3 endpoint if you don't want to use the default regional service endpoint.
* **SQS custom endpoint** - Enter a specific SQS endpoint if you don't want to use the default regional service endpoint.
  {% endstep %}

{% step %}
Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabelled**.

{% hint style="info" %}
Learn more about labels in [this article](https://docs.onum.com/the-workspace/listeners/labels).
{% endhint %}
{% endstep %}

{% step %}
Click **Create listener** when you're done.
{% endstep %}
{% endstepper %}

## Output Ports <a href="#ports" id="ports"></a>

The **Crowdstrike Falcon Data Replicator** Listener has a single output port:

* **Default port** - Events are sent through this port if no error occurs while processing them.