Send data to Amazon Kinesis

See the changelog of the Amazon Kinesis Data Stream Data Sink here.

Overview

Onum supports integration with Amazon Kinesis Data Stream.

Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that allows you to ingest, store, and process real-time data streams. It's designed for high-throughput, low-latency data ingestion from various sources, enabling real-time analytics and applications.

Prerequisites

You will need an IAM User, role or group with the correct permissions to access and manage Kinesis.

Amazon Kinesis Data Stream Setup

1

Go to IAM (Identity and Access Management) to manage users, groups, roles and permissions.

Under Permissions Policies, make sure you have assigned the policy AmazonKinesisFullAccess to give full access to Kinesis resources. Alternatively, if you have custom permissions, go to Policies - Create Policy and in the JSON tab, paste your custom JSON e.g.

  {
    "Version": "2012-10-17",
      "Statement": [
       {
         "Effect": "Allow"
         "Action": [
         "kinesis:CreateStream",
         "kinesis:DescribeStream",
         "kinesis:PutRecord"
         ],
         "Resource": "*"
 		}        
	]
  }      

2

Test the Configuration

Run aws kinesis list-streams

If you IAM permission are correct, you'll see a list of streams.

Onum Setup

1

Log in to your Onum tenant and click Data Sinks > New Data sink.

2

Double-click the Amazon Kinesis Data Stream Data Sink.

3

Enter a Name for the new Data Sink. Optionally, add a Description and some Tags to identify the Data Sink.

4

Decide whether or not to include this Data Sink info in the metrics and graphs of the Home area.

5

In the Connection settings section, click Select region and choose the region of your AWS data center. Your region is displayed in the top right-hand corner of your AWS console.

6

In the Authentication section, click Select access key ID and choose it from your Secrets, or click New secret to generate a new one.

The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

1. In the left panel, click on Users.

2. Select your IAM user.

3. Under the Security Credentials tab, scroll to Access Keys, and you will find existing Access Key IDs (but not the secret access key).

7

Click Select secret key and choose it from your Secrets, or click New secret to generate a new one.

Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.

Learn more about secrets in Onum in this article.

8

You can enable the Assume Role option, which provides a more secure and scalable alternative to traditional credential-based authentication.

Instead of using static credentials, this method leverages an Amazon Resource Name (ARN) to define and assume an IAM role with the necessary permissions to access your SQS queue. By introducing the ARN into the Listener configuration, Falcon Onum can securely assume the designated role and interact with your AWS resources in a controlled and auditable way. To enable access, we need cross-account access from our AWS infrastructure to yours. This is a one-time setup.

First, you must create an IAM role in your AWS account (or use an existing one) with the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::721751517261:role/<CustomersRole>"
      },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}

This trust policy allows our AWS account to assume the role. You can attach whatever permissions you want to the role — the trust policy only controls who can assume it, not what it can do.

Once the IAM role is ready, please contact us to share your full role ARN (e.g. arn:aws:iam::012345678901:role/YourRoleName) so we can complete the configuration on our end. Then, configure the Assume Role options as follows:

• Role ARN - Amazon Resource Name used to access S3 and SQS resources. This is the unique identifier for the specific IAM Role that you want to assume and use.

• External ID - Shared secret used to authenticate the usage of this role.

• Role Session - Name of the session, used to audit usage of this role ( s3-listener by default)

• STS Region - If not set, it will use the bucket or the queue region.

• STS Session Duration - How much the Assume Role session will last before reauthentication. Uses Golang duration strings, like 1s, 1m, 1h. If not set, it uses the maximum session duration configured for that role. The maximum and default duration is 1h.

9

Now enter your Data Stream configuration:

• Stream Name - Enter your AWS Kinesis stream name. To get it:

  1. Go to: https://console.aws.amazon.com/kinesis

  2. Select Data Streams under Amazon Kinesis in the sidebar.

  3. The Stream Name will be in the first column e.g. my-kinesis-stream-prod

• Partition key - This is a string that you assign to each record you put into a stream. It plays a critical role in determining which shard the data record is routed to within the stream e.g. user_456 means all login events for this user will be sent to the same shard, preserving the order of events for that user.

10

In the Advanced Configuration section, toggle on the Bulk configuration switch to enable bulk configuration. Configure the following settings:

• Event time limit - Time in milliseconds to wait before sending the batch.

Click Select condition and choose one of the following:

• Number of events - Maximum number of events to send in a single batch. The minimum value is 1, and the maximum value is 500.

• Batch size - Maximum number of events to send in a single batch. The minimum value is 1024, and the maximum value is 5242880.

Enter the required setting in the Value field.

11

If you have non-default URL that directs API requests to a specific Kinesis service endpoint, enter it here in the Custom endpoint.

12

Click Finish when complete. Your new Data Sink will appear in the Data sinks area list.

Pipeline configuration

When it comes to using this Data Sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.

Output configuration

Parameter	Description
Event Field	Select the event field that contains the output message. The data type must be string.

Click Save to save your configuration.