> For the complete documentation index, see [llms.txt](https://docs.onum.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.onum.com/listeners/listener-integrations/collect-data-from-amazon-s3/collect-s3-bucket-file-content.md). # Collect S3 Bucket File Content You can send the files in your Amazon S3 buckets to Onum using the **Amazon S3** Listener. ## Prerequisites Before configuring and starting to send data with the **Amazon S3** Listener, you need to take into consideration the following requirements: * Your Amazon user needs at least permission to use the `GetObject` operation (S3) and the `ReceiveMessage` and `DeleteMessageBatch` operations (SQS Bucket) to make this Listener work. * **Cross-Region Configurations** - Ensure that your S3 bucket and SQS queue are in the same AWS Region, as S3 event notifications do not support cross-region targets. * **Permissions** - Confirm that the AWS Identity and Access Management (IAM) roles associated with your S3 bucket and SQS queue have the necessary permissions. * **Object Key Name Filtering** - If you use special characters in your prefix or suffix filters for event notifications, ensure they are URL-encoded. {% hint style="warning" %} When S3 events flow through SNS before reaching SQS, enabling [**Raw Message Delivery**](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery [here](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html). {% endhint %} ## Amazon S3 Setup You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new files are added. {% stepper %} {% step %} #### **Create an Amazon SQS Queue** * Sign in to the AWS Management Console and open the Amazon SQS console. * Choose **Create Queue** and configure the queue settings as needed. * After creating the queue, note its Amazon Resource Name (ARN), which follows this format: `arn:aws:sqs:::`. {% endstep %} {% step %} #### Modify the SQS Queue Policy to Allow S3 to Send Messages 1. In the Amazon SQS console, select your queue. 2. Navigate to the **Access Policy** tab and choose **Edit**. 3. Replace the existing policy with the following, ensuring you update the placeholders with your specific details: ```json { "Version": "2012-10-17", "Id": "S3ToSQSPolicy", "Statement": [ { "Sid": "AllowS3Bucket", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "arn:aws:sqs:::", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:::" }, "StringEquals": { "aws:SourceAccount": "" } } } ] } ``` Save the changes. This policy grants your S3 bucket permission to send messages to your SQS queue. {% endstep %} {% step %} #### S3 Event Notification Rules for Files Get notified when files are added or modified using the following JSON: ```json { "QueueConfigurations": [ { "Id": "FileUploadNotification", "QueueArn": "arn:aws:sqs:::", "Events": ["s3:ObjectCreated:*"], "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "files/" } ] } } } ] } ``` 1. Open the Amazon S3 console and select the bucket you want to configure. 2. Go to the **Properties** tab and find the "Event notifications" section. 3. Click on **Create event notification**. 4. Provide a descriptive name for the event notification. 5. In the **Event types** section, enter the JSON. 6. In the **Destination** section, choose **SQS Queue** and select the queue you configured earlier. 7. Save the configuration. {% endstep %} {% step %} #### Test the Configuration 1. Upload a new file to your S3 bucket. 2. Check your SQS queue to verify that a message has been received, indicating that the notification setup is functioning correctly. {% endstep %} {% endstepper %} ## Onum Setup {% stepper %} {% step %} Log in to your Onum tenant and click **Listeners > New listener**. {% endstep %} {% step %} Double-click the **Amazon S3** Listener. {% endstep %} {% step %} Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener. {% endstep %} {% step %} In the **Objects** section, enter the required **Compression** method used in the ingested S3 files and **Format** of the ingested S3 files. * **Compression** - This accepts the standard compression codecs (**gzip**, **zlib**, **bzip2**), **none** for no compression, and **auto** to autodetect the compression type from the file extension. * **Path pattern** - This allows you to filter which S3 objects are processed based on their key (path). It accepts Go regular expressions (RE2 syntax), which are matched against the full object key after URL-decoding. Objects whose key does not match the pattern are silently skipped and their SQS notification is still acknowledged, preventing poison-pill scenarios where non-processable files (e.g. CloudTrail digest files) block the queue. This field is required and the default value is `.*` (matches everything, backward-compatible with previous versions that did not have this filter). Find some examples below:
RegExDescription
.*Matches all keys (default, backward-compatible)
^data/.*\.json(\.gz)?$Only JSON (or gzipped JSON) files under data/
^(?!.*Digest).*$Excludes CloudTrail digest files
CloudTrail/[^/]+/.*\.json\.gz$Only CloudTrail log files (not digests)
^logs/Only objects with the logs/ prefix
\.(csv|parquet)$Only CSV or Parquet files regardless of path
^[^/]+/[^/]+/year=2024/Only objects partitioned under year=2024
* **Format** - This currently accepts **JSON**, **JSON lines** (a JSON object representing an event on each line), **Text** and **CSV**. If you select **JSON, Text** or **CSV**, more options appear:
JSON Options * **Path** - Enter the path of the JSON element you want to retrieve. The path are keys separated by dots. For example `one.two.three` will select the element in `{"one":{"two":{"three":[1,2,3]}}`. To select the root, you can leave the path empty or enter a single dot `.` (default option). {% hint style="warning" %} For CloudWatch and CloudTrail logs, you must enter the `.Records` path. {% endhint %} * **Array Unroll** - Activate this toggle if you want to generate one event for each element in the array. The element that the path points to must be a JSON array.
CSV Options * **Header Row** - Select **true** to include a header for your CSV rows. * **Delimiter** - Decide between **comma**, **semicolon** and **tab**. * **Text Encoding** - Choose the scheme (or key) that maps the characters used to store and read the data in the CSV file. * **Output Format** - Choose either **CSV** or **JSON**. * **JSON Output:** * Converts CSV records to structured JSON objects * Field names are derived from header row (if present) or auto-generated (`field_0`, `field_1`, etc.) * Provides structured data for easier processing in Pipelines. * **CSV Output:** * Preserves original CSV formatting. * Each CSV record becomes a separate event containing the raw CSV line. * Useful when you want to maintain the original CSV structure. * **Trim Leading Space** - Select **true** to remove any whitespace characters that appear immediately before the first non-whitespace character in a cell. * **Lazy Quotes** - Select **true** to allow double quotes to appear in fields without strictly following the formal CSV escaping rules. * **Fields per Record** - The number of fields expected in each row (record) of your CSV file. * **Comment Character** - Use the hash symbol (`#`) to designate lines of text that should be ignored during the data parsing process.
Text Options * **Framing Method** - Select between **Non Transparent Framing (newline, zero, or none)** and **Octet counting**
{% endstep %} {% step %} Define the **Bucket** to listen from. * **Region** - Find this in your **Buckets** area, next to the name. * **Name** - The [AWS bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#:~:text=A%20bucket%20is%20a%20container,the%20object%20within%20the%20bucket.) your data is stored in. This is the bucket name found in your **Buckets** area. You can fill this if you want to check that notifications come from that bucket, or leave it empty to avoid such checks. * **Authentication Type** - Choose **manual** to enter your access key ID and secret access key manually in the parameters below, or **auto** to authenticate automatically. The default value is **manual**. {% hint style="warning" %} If you select the **auto** authentication and you're working on an Onum cloud deployment, please [contact Support](https://docs.onum.com/support) to request the ARN information for your Onum-managed instance. {% endhint %} * **Access key ID** - Select the access key ID from your [Secrets](broken://pages/SNC8zGVDtIAIFa5E19n5) or click **New secret** to generate a new one. The **Access Key ID** is found in the **IAM Dashboard** of the **AWS Management Console**. 1. In the left panel, click on **Users**. 2. Select your **IAM user**. 3. Under the **Security Credentials** tab, scroll to **Access Keys**, and you will find existing **Access Key IDs** (but not the secret access key). * **Secret access key** - Select the secret access key from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or click **New secret** to generate a new one. Under **Access keys**, you can see your **Access Key IDs**, but AWS **will not show the Secret Access Key**. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one. {% endstep %} {% step %} You can enable the **Assume Role** option, which provides a more secure and scalable alternative to traditional credential-based authentication. Instead of using static credentials, this method leverages an Amazon Resource Name (ARN) to define and assume an IAM role with the necessary permissions to access your SQS queue. By introducing the ARN into the Listener configuration, Falcon Onum can securely assume the designated role and interact with your AWS resources in a controlled and auditable way. To enable access, we need cross-account access from our AWS infrastructure to yours. This is a one-time setup. First, you must create an IAM role in your AWS account (or use an existing one) with the following trust policy: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::721751517261:role/" }, "Action": ["sts:AssumeRole", "sts:TagSession"] } ] } ``` {% hint style="warning" %} This trust policy allows our AWS account to assume the role. You can attach whatever permissions you want to the role — the trust policy only controls who can assume it, not what it can do. {% endhint %} Once the IAM role is ready, please [contact us](broken://spaces/cSjT21I4EUhzghjc1rER/pages/nW8oZycpfVthtI5KYdQs) to share your full role ARN (e.g. `arn:aws:iam::012345678901:role/YourRoleName`) so we can complete the configuration on our end. Then, configure the **Assume Role** options as follows: * **Role ARN** - **Amazon Resource Name** used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use. * **External ID** - Shared secret used to authenticate the usage of this role. * **Role Session** - Name of the session, used to audit usage of this role ( `s3-listener` by default) * **STS Region** - If not set, it will use the bucket or the queue region. * **STS Session Duration** - How much the Assume Role session will last before reauthentication. Uses Golang duration strings, like `1s`, `1m`, `1h`. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h. {% endstep %} {% step %} Optionally, you can enter a **Service endpoint** in the **Bucket Advanced** section. Amazon S3 provides different types of **service endpoints** based on the **region and access type**. 1. Select your **bucket**. 2. Go to the **Properties** tab. 3. Under **Bucket ARN & URL**, find the S3 **endpoint URL**. {% hint style="warning" %} Proceed with caution when modifying the **Bucket Advanced** options. Default values should be enough in most cases. Amazon Service Endpoint will usually be chosen automatically, so you should not normally have to fill this up. However, in case you need to override the default access point, you can do it here. {% endhint %} {% endstep %} {% step %} In the **Queue** section, enter the following information: * **Region** - Choose the region your queue is created in from the dropdown provided. * **URL** - Enter the URL of your existing Amazon SQS queue to send the data to. To find it: * Go to the **AWS Management Console**. * In the **Search Bar**, type **SQS** and click on **Simple Queue Service (SQS)**. * Click on **Queues** in the left panel. * Locate your queue from the list and click it. * The **Queue URL** will be displayed in the table under **URL**. This is the correct URL format: `https://sqs.region.localhost/awsaccountnumber/storedinenvvar` * **Authentication Type** - Choose **manual** to enter your access key ID and secret access key manually in the parameters below, or **auto** to authenticate automatically. * **Event name** - Name of the S3 event that triggers the notification. If not specified, S3 will capture all events. {% endstep %} {% step %} Optionally, configure the following **Queue Advanced** options: {% hint style="warning" %} Proceed with caution when modifying the **Queue Advanced** options. Default values should be enough in most cases. {% endhint %} * **Service endpoint** - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default. * **Maximum number of messages** - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `1`, and the maximum and default value is `10`. * **Visibility timeout** - Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is `30s`, and the maximum value is `12h`. The default value is `1h`. * **Wait time** - When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is `5s`, and the maximum and default value is `20s`. {% endstep %} {% step %} Optionally, configure the following **General Advanced** options: {% hint style="warning" %} Proceed with caution when modifying the **General Advanced** options. Default values should be enough in most cases. {% endhint %} * **Event batch size** - Enter a limit for the number of events allowed through per batch. The minimum value is `1`, and the maximum and default value is `1000000`. * **Minimum retry time** - Set the minimum amount of time to wait before retrying. The default and minimum value is `1s`, and the maximum value is `10m`. * **Maximum retry time** - Set the maximum amount of time to wait before retrying. The default value is `5m`, and the maximum value is `10m`. The minimum value is the one set in the parameter above. {% endstep %} {% step %} Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabelled**. {% hint style="info" %} Learn more about labels in [this article](/listeners/labels.md). {% endhint %} {% endstep %} {% step %} Click **Create listener** when you're done. {% endstep %} {% endstepper %} ## Output Ports The **Amazon S3** Listener has only a single output port: * **Default port** - Events are sent through this port if no error occurs while processing them.