Collect data from Amazon S3

Most recent version: v2.2.0

See the changelog of the Amazon S3 Listener here.

The Amazon S3 Listener is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Amazon Simple Storage Service (S3) is a fully managed object storage service. Users typically use it to store big files at a reasonable cost for long periods of time. In particular, it's commonly used as a data lake storage layer, storing files containing user events with some format/encoding/compression.

Amazon S3 also supports sending notifications to an SQS queue when new files are added to some bucket. You can see a sample notification here.

By leveraging all the above, our S3 Listener is able to react to new files being added to the bucket, get the files, and ingest their events into Onum. All that is needed is an existing SQS queue, an existing S3 bucket, and having the bucket correctly configured to send notifications to the queue.

Using the Amazon S3 Listener, you can read the following AWS content:

• Collect AWS Application Logs

• Collect Bucket Content