Amazon Kinesis Data Stream
Most recent version: v0.0.1
This is a Pull Listener and therefore should not be used in environments with more than one cluster.
Overview
Onum supports integration with Amazon Kinesis Data Stream.
Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that allows you to ingest, store, and process real-time data streams. It's designed for high-throughput, low-latency data ingestion from various sources, enabling real-time analytics and applications.
Select Amazon Kinesis Data Stream from the list of Data sink types and click Configuration to start.
Data sink configuration
Now you need to specify how and where to send the data, and how to establish a connection with Amazon Kinesis.
Metadata
Enter the basic information for the new Sink.
Name*
Enter a name for the new Data sink.
Description
Optionally, enter a description for the Data sink.
Tags
Add tags to easily identify your Data sink. Hit the Enter key after you define each tag.
Metrics display
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
Configuration
Now, add the configuration to establish the connection.
AWS Authentication
Region*
Choose the region of your AWS data center. Your region is displayed in the top right-hand corner of your AWS console.
Access Key ID*
Select the access key ID from your Secrets or click New secret to generate a new one.
The Access Key ID is found in the IAM Dashboard of the AWS Management Console.
In the left panel, click on Users.
Select your IAM user.
Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).
Secret Access Key*
Select the secret access key from your Secrets or click New secret to generate a new one.
Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.
Data Stream configuration
An Amazon Kinesis Data Stream is a real-time data streaming service provided by AWS that allows you to collect, process, and analyze large volumes of streaming data in real time.
Stream name*
Name that identifies your stream. To get it:
Select Data Streams under Amazon Kinesis in the sidebar.
The Stream Name will be in the first column, e.g.
my-kinesis-stream-prod
Partition key*
A string that you assign to each record you put into a stream. It plays a critical role in determining which shard the data record is routed to within the stream e.g. user_456 means all login events for this user will be sent to the same shard, preserving the order of events for that user.
Advanced configuration
Toggle on the Bulk configuration switch to enable bulk configuration. Configure the following settings:
Event time limit*
Time in milliseconds to wait before sending the batch.
Number of events*
Maximum number of events to send in a single batch. The minimum value is 1, and the maximum value is 500.
Batch size*
Maximum number of events to send in a single batch. The minimum value is 1024, and the maximum value is 5242880.
Other configuration
Custom endpoint
If you have non-default URL that directs API requests to a specific Kinesis service endpoint, enter it here.
Click Finish when complete. Your new Data sink will appear in the Data sinks area list.
Pipeline configuration
When it comes to using this Data sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.
Output configuration
Event Field*
Select the event field that contains the output message. The data type must be string.
Click Save to save your configuration.
Last updated
Was this helpful?