S3

Current version v0.1.0

See the changelog of this Sink here.

Overview

Onum supports integration with AWS S3. Select S3 from the list of sink types and click Configuration to start.

Configuration

Now you need to specify how and where to send the data, and how to establish a connection with S3.

Metadata

Enter the basic information for the new Data Sink.

Parameters	Description
Name*	Enter a name for the new Data Sink.
Description	Optionally, enter a description for the Data Sink.
Tags	Add tags to easily identify your Sink. Hit the Enter key after you define each tag.

---

Metrics

Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.

Configuration

Now add the configuration to establish the connection.

Parameter	Description
Event field*	This is the name of the input event field.
Framing method*	This parameter defines how events are separated within an S3 object (further defined in the S3 object section of the Sink). Choose between the various options. Newline: uses a newline character ('\n') to separate individual records in the output. Length: the S3 framing method length is 10 bytes. No Framing: all events are contained in one line, leading to a long line until the maximum size is reached, with only one region.
Compress data	Choose between true/false to enable/disable compression.

AWS

Parameter	Description
Bucket*	The AWS bucket your data is stored in. This is the bucket name found in your Buckets area.
Region*	Choose the region the cloud server is found in, found also in your Buckets area next to the name.

S3 object

S3 objects are files or data sets that are stored in a bucket. Each object is identified by a key that uses prefixes to simulate a folder structure. Click the bucket name to view its objects and properties. Click an object to open it and see the following parameters.

Parameter	Description
Key format	Prefix: the prefix used to organize your S3 data. Partition by: this indicates the frequency with which to generate a new S3 object e.g. every year, month, day hour, minute. If left blank, the value used will be the max object size entered below to determine when to create a new object.
Storage class	The desired S3 storage class. See this in the main objects table, or by clicking the object and going to Storage Class.
Canned ACL	Choose the S3 Access Control List.

Authorization

Only if your Bucket requires authorization.

Parameter	Description
Access key ID*	Add the access key from your Secrets or create one. The Access Key ID is found in the IAM Dashboard of the AWS Management Console. In the left panel, click on Users. Select your IAM user. Under the Security Credentials tab, scroll to Access Keys and you will find existing Access Key IDs (but not the secret access key).
Secret access key*	Add the secret access key from your Secrets or create one. Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one

Advanced options

Parameter	Description
Max object size	Instead of partitioning by time, you can partition by size of the message. Assign the object's maximum size (in MB). If you do not select a partition by value, a new object is created upon reaching this limit.
Custom Endpoint	If you have one, enter your custom endpoint.

---

Click Finish when complete.