Amazon S3
Most recent version: v0.0.1
Last updated
Was this helpful?
Most recent version: v0.0.1
Last updated
Was this helpful?
This is a Pull listener, therefore should not be used in environments with more than one cluster.
Onum supports integration with AWS S3. Select Amazon S3 from the list of Listener types and click Configuration to start.
Before configuring and starting to send data with the Amazon S3 Listener, you need to take into consideration the following requirements:
Your Amazon user needs at least permission to use the GetObject operation (S3) and the ReceiveMessage and DeleteMessageBatch operations (SQS Bucket) to make this Listener work.
You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new files are added. Learn how to do it below:
Now you need to specify how and where to collect the data, and how to establish a connection with AWS S3.
Enter the basic information for the new Listener.
Name*
Enter a name for the new Listener.
Description
Optionally, enter a description for the Listener.
Tags
Add tags to easily identify your Listener. Hit the Enter key after you define each tag.
Compression*
Select the compression method used in the ingested S3 files. This accepts the standard compression codecs (gzip, zlib, bzip2), none for no compression, and auto to autodetect the compression type from the file extension.
Format*
Select the format of the ingested S3 files. This currently accepts json array (a big json array containing a json object for each event), json lines (a json object representing an event on each line), and auto to autodetect the compression type from the file extension (.json or .jsonl, respectively).
Region*
Choose the region the bucket is found in, found also in your Buckets area next to the name.
Name
Access key ID*
In the left panel, click on Users.
Select your IAM user.
Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).
Secret access key*
Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one
Proceed with caution when modifying these advanced options. Default values should be enough in most cases.
Service endpoint
Optionally, Amazon S3 provides different types of service endpoints based on the region and access type.
Select your bucket.
Go to the Properties tab.
Under Bucket ARN & URL, find the S3 endpoint URL.
Amazon Service Endpoint will usually be chosen automatically, so you should not normally have to fill this up. However, in case you need to override the default access point, you can do it here.
Region
Choose the region your queue is created in from the drop-down provided.
URL*
The URL of your existing Amazon SQS queue to send the data to.
Go to the AWS Management Console.
In the Search Bar, type SQS and click on Simple Queue Service (SQS).
Click on Queues in the left panel.
Locate your queue from the list and click it.
The Queue URL will be displayed in the table under URL.
The correct format: https://sqs.region.localhost/awsaccountnumber/storedinenvvar
Access key ID
In the left panel, click on Users.
Select your IAM user.
Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).
Note that this can be the same as in the bucket, in which case you don't need to repeat it here, or it can be different, depending on how you have configured your bucket & queue
Secret access key
This can be the same as for the bucket, in which case you don't need to repeat it here, or it can be different, depending on how you have configured your bucket & queue.
Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one. Note that this can be the same as in the bucket, in which case you don't need to repeat it here, or it can be different, depending on how you have configured your bucket & queue.
Event name
When you configure your bucket to send notifications to your SQS queue, you choose a name for those notification events. You can provide that name here to check the notifications to match that name when they are read by the Listener, or leave this empty to avoid such checks.
Proceed with caution when modifying these advanced options. Default values should be enough in most cases.
Service endpoint
If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default.
Maximum number of messages*
Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.
Visibility timeout*
Set how many seconds to leave a message as hidden in the queue after being delivered, before redelivering it to another consumer if not acknowledged. The minimum value is 30s, and the maximum value is 12h. The default value is 1h.
Wait time*
When the queue is empty, set how long to wait for messages before deeming the request as timed out. The minimum value is 5s, and the maximum and default value is 20s.
Proceed with caution when modifying these advanced options. Default values should be enough in most cases.
Event batch size*
Enter a limit for the number of events allowed through per batch. The minimum value is 1, and the maximum and default value is 1000000.
Minimum retry time*
Set the minimum amount of time to wait before retrying. The default and minimum value is 1s, and the maximum value is 10m.
Maximum retry time*
Set the maximum amount of time to wait before retrying. The default value is 5m, and the maximum value is 10m. The minimum value is the one set in the parameter above.
Click Create labels to move on to the next step and define the required Labels.
The your data is stored in. This is the bucket name found in your Buckets area. You can fill this if you want to check that notifications come from that bucket, or leave it empty to avoid such checks.
Add the access key from your or create one. The Access Key ID is found in the IAM Dashboard of the AWS Management Console.
Add the secret access key from your or create one.
Add the access key from your or create one. The Access Key ID is found in the IAM Dashboard of the AWS Management Console.
Add the secret access key from your or create one.
