# Send data to Amazon Kinesis
{% hint style="info" %}
See the changelog of this Data Sink type [here](/listeners/amazon-kinesis-data-stream-listener.md).
{% endhint %}
## Overview
Onum supports integration with [Amazon Kinesis Data Stream](https://aws.amazon.com//kinesis/data-streams/).
Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that allows you to ingest, store, and process real-time data streams. It's designed for high-throughput, low-latency data ingestion from various sources, enabling real-time analytics and applications.
## Prerequisites
You will need an IAM User, role or group with the correct permissions to access and manage Kinesis.
## Amazon Kinesis Data Stream Setup
{% stepper %}
{% step %}
#### **Go to IAM (Identity and Access Management)** to manage users, groups, roles and permissions.
Under **Permissions Policies**, make sure you have assigned the policy `AmazonKinesisFullAccess` to give full access to Kinesis resources. Alternatively, if you have custom permissions, go to **Policies - Create Policy** and in the **JSON** tab, paste your custom JSON e.g.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow"
"Action": [
"kinesis:CreateStream",
"kinesis:DescribeStream",
"kinesis:PutRecord"
],
"Resource": "*"
}
]
}
```
{% endstep %}
{% step %}
#### Test the Configuration
`Run aws kinesis list-streams`
If you IAM permission are correct, you'll see a list of streams.
{% endstep %}
{% endstepper %}
## Onum Setup
{% stepper %}
{% step %}
Log in to your Onum tenant and click **Data Sinks> New Data sink**.
{% endstep %}
{% step %}
Double-click the **Amazon Kinesis Data Stream** Sink.
{% endstep %}
{% step %}
Enter a **Name** for the new Data Sink. Optionally, add a **Description** and some **Tags** to identify the Sink.
{% endstep %}
{% step %}
Decide whether or not to include this Data sink info in the metrics and graphs of the [**Home**](/the-workspace/home.md) area.
{% endstep %}
{% step %}
In the **AWS authentication** section, enter the region of your AWS data center. Your region is displayed in the top right-hand corner of your AWS console.
{% endstep %}
{% step %}
Select the **Access Key ID** from your [Secrets](/administration/tenant-menu.md) or click **New secret** to generate a new one.
The **Access Key ID** is found in the **IAM Dashboard** of the **AWS Management Console**.
1. In the left panel, click on **Users**.
2. Select your **IAM user**.
3. Under the **Security Credentials** tab, scroll to **Access Keys**, and you will find existing **Access Key IDs** (but not the secret access key).
{% endstep %}
{% step %}
Select the **Secret Access Key** from your [Secrets](/administration/tenant-menu.md) or click **New secret** to generate a new one.
Under **Access keys**, you can see your **Access Key IDs**, but AWS **will not show the Secret Access Key**. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.
{% hint style="info" %}
Learn more about secrets in Onum in [this article](/administration/global-settings/organization-settings/secrets-management.md).
{% endhint %}
{% endstep %}
{% step %}
**Access external Kinesis resources using** [`AssumeRole`](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_assumerole.html)
This role should have the following permissions to access Kinesis streams:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "KinesisReadAccess",
"Effect": "Allow",
"Action": [
"kinesis:DescribeStream",
"kinesis:GetRecords",
"kinesis:GetShardIterator",
"kinesis:ListShards",
"kinesis:SubscribeToShard",
"kinesis:DescribeStreamSummary",
"kinesis:RegisterStreamConsumer"
],
"Resource": [
""
]
},
{
"Sid": "KinesisListAccess",
"Effect": "Allow",
"Action": [
"kinesis:ListStreams",
"kinesis:ListStreamConsumers"
],
"Resource": "*"
}
]
}
```
AWS credentials and AssumeRole configuration can be configured specifically for Kinesis access. If credentials are not explicitly configured for Kinesis, the system will use the default AWS credentials.
The configuration options are as follows:
* **Role ARN****\*** - **Amazon Resource Name** used to access Kinesis resources. This is the unique identifier for the specific IAM Role that you want to assume and use (format: `arn:aws:iam::123456789012:role/KinesisReadRole`).
* **External ID****\*** - shared secret used to authenticate the usage of this role.
* **Role Session** - name of the session, used to audit usage of this role (`kinesis-listener` by default)
* **STS Region** - if not set, it will use the Kinesis stream region. This specifies which region's STS endpoint to use when assuming the role.
* **STS Session Duration** - how much the `AssumeRole` session will last before reauthentication. Uses Golang duration strings, like `1s`, `1m`, `1h`. If not Set, it uses the maximum session duration configured for that role. The minimum duration is 15m and the maximum is configured in the role, no longer than 12h.
{% endstep %}
{% step %}
Configure your **Data Stream**.
* **Stream Name****\***
1. Go to:
2. Select **Data Streams** under **Amazon Kinesis** in the sidebar.
3. The **Stream Name** will be in the first column e.g. `my-kinesis-stream-prod`
* A **Partition key****\*** is a string that you assign to each record you put into a stream. It plays a critical role in determining which shard the data record is routed to within the stream e.g. `user_456` means all login events for this user will be sent to the same shard, preserving the order of events for that user.
{% endstep %}
{% step %}
In the **Advanced Configuration** section, toggle on the **Bulk configuration** switch to enable bulk configuration. Configure the following settings:
* **Event time limit****\*** - Time in milliseconds to wait before sending the batch.
* **Number of events****\*** - Maximum number of events to send in a single batch. The minimum value is `1`, and the maximum value is `500`.
* **Batch size****\*** - Maximum number of events to send in a single batch. The minimum value is `1024`, and the maximum value is `5242880`.
{% endstep %}
{% step %}
If you have non-default URL that directs API requests to a specific Kinesis service endpoint, enter it here in the **Custom endpoint**.
{% endstep %}
{% endstepper %}
Click **Create data sink** when complete.
Your new Data sink will appear in the **Data sinks** area list.
## Pipeline configuration
When it comes to using this Data sink in a [Pipeline](/the-workspace/pipelines.md), you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select **Configuration**.
### Output configuration
Parameter
Description
Event Field*
Select the event field that contains the output message. The data type must be string.
Click **Save** to save your configuration.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.onum.com/the-workspace/data-sinks/data-sink-integrations/send-data-to-aws-products/send-data-to-amazon-kinesis.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.