WebsiteBlogLogin

Collect data from Amazon Kinesis Data Stream

Most recent version: v0.0.1

Note that this Listener is only available in certain Tenants. Get in touch with us if you don't see it and want to access it.

See the changelog of this Listener type here.

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Amazon Kinesis Data Stream.

Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that allows you to ingest, store, and process real-time data streams. It's designed for high-throughput, low-latency data ingestion from various sources, enabling real-time analytics and applications.

Prerequisites

You will need an IAM User, role or group with the correct permissions to access and manage Kinesis.

Amazon Kinesis Data Stream Setup

You need to configure your Amazon S3 bucket to send notifications to an Amazon Simple Queue Service (SQS) queue when new files are added.

1

Go to IAM (Identity and Access Management) to manage users, groups, roles and permissions.

Under Permissions Policies, make sure you have assigned the policy AmazonKinesisFullAccess to give full access to Kinesis resources. Alternatively, if you have custom permissions, go to Policies - Create Policy and in the JSON tab, paste your custom JSON e.g.

  {
    "Version": "2012-10-17",
      "Statement": [
       {
         "Effect": "Allow"
         "Action": [
         "kinesis:CreateStream",
         "kinesis:DescribeStream",
         "kinesis:PutRecord"
         ],
         "Resource": "*"
 		}        
	]
  }
2

Test the Configuration

Run aws kinesis list-streams

If you IAM permission are correct, you'll see a list of streams.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Amazon Kinesis Data Stream Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

In the AWS authentication section, enter the region of your AWS data center. Your region is displayed in the top right-hand corner of your AWS console.

5

Select the Access Key ID from your Secrets or click New secret to generate a new one.

The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

  1. In the left panel, click on Users.

  2. Select your IAM user.

  3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

6

Select the Secret Access Key from your Secrets or click New secret to generate a new one.

Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.

Learn more about secrets in Onum in this article.

7

Configure your Data Stream.

  • Stream Name*

    1. Go to: https://console.aws.amazon.com/kinesis

    2. Select Data Streams under Amazon Kinesis in the sidebar.

    3. The Stream Name will be in the first column e.g. my-kinesis-stream-prod

  • Shard ID

The Shard is the basic unit of capacity in a Kinesis Data Stream, acting like a partition for your data stream and determining how your data is ingested, stored, and consumed.

Click your data stream name to find your Shard ID in the Shards tab e.g. shardId-000000000000 shardId-000000000001

8

In the Advanced Configuration section, enter the Custom endpoint if you have a non-default URL that directs API requests to a specific Kinesis service endpoint.

9

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabeled. Click Create listener when you're done.

Learn more about labels in this article.

PreviousCollect data from Amazon S3NextAmazon SQS

Last updated

Was this helpful?