Collect data from Amazon Kinesis

See the changelog of the Amazon Kinesis Listener here.

• The Amazon Kinesis Listener is only available in certain Tenants. Get in touch with us if you don't see it and want to access it.

• The Amazon Kinesis Listener is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Amazon Kinesis Data Stream.

Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that allows you to ingest, store, and process real-time data streams. It's designed for high-throughput, low-latency data ingestion from various sources, enabling real-time analytics and applications.

Select Amazon Kinesis from the list of Listener types and click Configuration to start.

Prerequisites

In order to use the Amazon Kinesis Listener, you must activate the following environment variable in your distributor using docker compose: SINGLETON_LISTENER_EXECUTOR=true

Amazon Kinesis Data Stream Setup

1

Go to IAM (Identity and Access Management) to manage users, groups, roles and permissions.

Under Permissions Policies, make sure you have assigned the policy AmazonKinesisFullAccess to give full access to Kinesis resources. Alternatively, if you have custom permissions, go to Policies - Create Policy and in the JSON tab, paste your custom JSON e.g.

{
  "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow"
        "Action": [
        "kinesis:CreateStream",
        "kinesis:DescribeStream",
        "kinesis:PutRecord"
        ],
        "Resource": "*"
 		  }        
	 ]
}

2

Run the following command to test the configuration:

aws kinesis list-streams

If your IAM permission are correct, you'll see a list of streams.

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the Amazon Kinesis Data Stream Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

In the Connection settings section, click Select region* and choose the region of your AWS data center. Your region is displayed in the top right-hand corner of your AWS console.

5

Click Select access key ID* and choose it from your Secrets, or click New secret to generate a new one.

The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

1. In the left panel, click on Users.

2. Select your IAM user.

3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

6

Click Select secret key* and choose it from your Secrets, or click New secret to generate a new one.

Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.

Learn more about secrets in Onum in this article.

7

You can enable a temporary external access enabling the Assume Role option. AssumeRole is an AWS STS (Security Token Service) action that allows an entity (user, service, or application) to temporarily assume an IAM role and obtain short-lived credentials.

This role should have the following permissions to access Kinesis streams:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "KinesisReadAccess",
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:ListShards",
        "kinesis:SubscribeToShard",
        "kinesis:DescribeStreamSummary",
        "kinesis:RegisterStreamConsumer"
      ],
      "Resource": [
        "<Kinesis stream ARN>"
      ]
    },
    {
      "Sid": "KinesisListAccess",
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams",
        "kinesis:ListStreamConsumers"
      ],
      "Resource": "*"
    }
  ]
}

AWS credentials and AssumeRole configuration can be configured specifically for Kinesis access. If credentials are not explicitly configured for Kinesis, the system will use the default AWS credentials.

The configuration options are as follows:

• Role ARN* - Amazon Resource Name used to access Kinesis resources. This is the unique identifier for the specific IAM Role that you want to assume and use (format: arn:aws:iam::123456789012:role/KinesisReadRole).

• External ID* - Shared secret used to authenticate the usage of this role.

• Role session - Name of the session, used to audit usage of this role (kinesis-listener by default)

• STS region - If not set, it will use the Kinesis stream region. This specifies which region's STS endpoint to use when assuming the role.

• STS session duration - How much the AssumeRole session will last before re-authentication. Uses Golang duration strings, like 1s, 1m, 1h. If not set, it uses the maximum session duration configured for that role. The minimum duration is 15m and the maximum is configured in the role, no longer than 12h.

8

Enter the following information in the Data Stream configuration section:

• Stream name* - Enter the unique identifier of your Kinesis Data Stream. To get it:

  1. Go to: https://console.aws.amazon.com/kinesis

  2. Select Data Streams under Amazon Kinesis in the sidebar.

  3. The Stream name will be in the first column e.g. my-kinesis-stream-prod

• Shard ID - The shard is the basic unit of capacity in a Kinesis Data Stream, acting like a partition for your data stream and determining how your data is ingested, stored, and consumed. Click your Data Stream name to find your Shard ID in the Shards tab, e.g.: shardId-000000000001

9

Activate the Enable KMS Decryption option if you want to automatically decrypt KMS-encrypted messages. If you activate it, you'll need to configure the following settings:

• Select KMS Key ID* - Choose your KMS Key ID from your Secrets or click New secret to define a new one. To find your KMS Key ID, go to the KMS console and select Customer managed keys. You will find it in the Key ID column.

• Set decryption timeout - Activate this option if you need to set a timeout for the decryption. Indicate it in the Seconds* field.

• Enable KMS encryption context - You can set an optional set of non-secret key–value pairs that can contain additional contextual information about the data. Use the Field* and Value* settings and click Add field to add the required pairs.

10

In the Advanced Configuration section, activate the Use compression option if you need to compress your data. Choose the required type (Gzip, Bzip2 or Zlib).

11

Optionally, enter the Custom endpoint if you have a non-default URL that directs API requests to a specific Kinesis service endpoint.

12

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabeled.

Learn more about labels in this article.

13

Click Create listener when you're done.

Output Ports

The Amazon Kinesis Listener has two output ports:

• Default port - Events are sent through this port if no error occurs while processing them.

• Error port - Events are sent through this port if an error occurs while processing them.