WebsiteBlogLogin

Amazon Kinesis Data Stream

See the changelog of this Listener type here.

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Amazon Kinesis Data Stream.

Select Amazon Kinesis Data Stream from the list of Listener types and click Configuration to start.

Configuration

Now you need to specify how and where to collect the data, and how to establish a connection with Amazon Kinesis.

Metadata

Enter the basic information for the new Listener.

Parameter
Description

Name*

Enter a name for the new Listener.

Description

Optionally, enter a description for the Listener.

Tags

Add tags to easily identify your Listener. Hit the Enter key after you define each tag.

Configuration

Now, add the configuration to establish the connection.

AWS authentication

Parameter
Description

Region

The region of your AWS data center. Your region is displayed in the top right-hand corner of your AWS console.

Session Token*

Add the Secret if you have previously added this credential, or create it here.

The Session Token is found in the IAM Dashboard of the AWS Management Console.

A typical format is something like FwoGZXIvYXdzEN7//////////wEaDDlEXAMPLESESSIONTOKENvN/7k3GVfJQEXAMPLETOKENkE=

Access Key ID*

Select the access key ID from your Secrets or click New secret to generate a new one.

The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

  1. In the left panel, click on Users.

  2. Select your IAM user.

  3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

Secret Access Key*

Select the secret access key from your Secrets or click New secret to generate a new one.

Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.

Data Stream configuration

An Amazon Kinesis Data Stream is a real-time data streaming service provided by AWS that allows you to collect, process, and analyze large volumes of streaming data in real time.

Parameter
Description

Stream Name*

  1. Go to: https://console.aws.amazon.com/kinesis

  2. Select Data Streams under Amazon Kinesis in the sidebar.

  1. The Stream Name will be in the first column e.g. my-kinesis-stream-prod

Shard ID

The Shard is the basic unit of capacity in a Kinesis Data Stream, acting like a partition for your data stream and determining how your data is ingested, stored, and consumed.

Click your data stream name to find your Shard ID in the Shards tab e.g. shardId-000000000000 shardId-000000000001

Starting position*

This tells Kinesis where in the shard to begin reading data.

  • TRIM_HORIZON Starts reading from the oldest available record in the shard (i.e., from the very beginning of the stream’s retained data).

  • LATEST Starts reading only new records that arrive after the shard iterator is created. Good for real-time streaming.

  • AT_TIMESTAMP Starts reading from records at or after a specific timestamp. You must also specify a Timestamp value.

  • AT_SEQUENCE_NUMBER Starts reading exactly at the record with the given sequence number. Used for precise record targeting.

  • AFTER_SEQUENCE_NUMBER Starts reading just after the given sequence number, skipping that specific record. Useful for checkpointing.

Advanced Configuration

Parameter
Description

Custom Endpoint

If you have non-default URL that directs API requests to a specific Kinesis service endpoint, enter it here.

Click Create labels to move on to the next step and define the required Labels if needed.

Last updated

Was this helpful?