# Collect data from Amazon SQS

{% hint style="info" %}
See the changelog of the **Amazon SQS** Listener [here](https://app.gitbook.com/s/1OZWDcmMPrhfCtF1rMJP/amazon-sqs-listener).
{% endhint %}

{% hint style="warning" %}
The **Amazon SQS** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster.
{% endhint %}

## Overview

Onum supports integration with [Amazon SQS](https://aws.amazon.com/sqs/?nc1=h_ls).

Amazon Simple Queue Service (AWS SQS) is a fully managed message queuing service. Among its many features, the following ones are of special interest to our use case:

* It supports both standard queues (with at-least-once, occasionally unordered delivery semantics) and FIFO queues (exactly-once and fully ordered delivery semantics).
* It supports scaling through the concept of visibility timeout (a period after a consumer reads one message during which this becomes invisible to other consumers). That allows a consumer group to read from the same queue and distribute messages without duplication.

So, what we have is a Listener that we can configure to **read from an existing SQS queue and inject queue messages as events into our platform**. Please note that because of the nature of the API offered to access SQS messages (HTTP-based, max 10 messages each time), this is not a high-throughput Listener.

Select **Amazon SQS** from the list of Listener types and click **Configuration** to start.

## Prerequisites

You will need an **IAM** User, role or group with the correct permissions to access and manage SQS. 

{% hint style="warning" %}
When S3 events flow through SNS before reaching SQS, enabling [**Raw Message Delivery**](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html) on the SNS subscription is essential. Without this setting, S3 notifications become wrapped in an SNS JSON envelope, creating nested JSON that's difficult to parse. See more about Raw Message Delivery [here](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html).
{% endhint %}

## Amazon SQS Setup

**Go to IAM (Identity and Access Management)** to manage users, groups, roles and permissions. 

Under **Permissions Policies**, make sure you have assigned the policy `AmazonSQSFullAccess` to give full access to SQS resources. Alternatively, if you have custom permissions, go to **Policies > Create Policy** and in the **JSON** tab, paste your custom JSON. For example:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:CreateQueue",
        "sqs:GetQueueAttributes",
        "sqs:SendMessage"
      ],
      "Resource": "*"
    }
  ]
}
```

## Onum Setup

{% stepper %}
{% step %}
Log in to your Onum tenant and click **Listeners > New listener**.
{% endstep %}

{% step %}
Double-click the **Amazon SQS** Listener.
{% endstep %}

{% step %}
Enter a **Name**<mark style="color:$primary;">**\***</mark> for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener.
{% endstep %}

{% step %}
Enter the **Region** displayed in the top right-hand corner of your AWS console.
{% endstep %}

{% step %}
Enter the **Queue URL**<mark style="color:red;">**\***</mark> of your existing Amazon SQS queue, acting as the endpoint to interact with the desired queue. Use the `GetQueueUrl` command or:

1. **Go to the** AWS Management Console.
2. In the **Search Bar**, type `SQS` and click on **Simple Queue Service (SQS)**.
3. Click on **Queues** in the left panel.
4. Locate your queue from the list and click it.
5. The **Queue URL** will be displayed in the table under **URL**.

This is the correct URL format: `sqs.region.localhost/awsaccountnumber/storedinenvv`
{% endstep %}

{% step %}
Now, choose your authentication type. Authentication is not specific to SQS but rather AWS IAM (Identity and Access Management). If you are connecting from an IAM console, enter the authentication credentials here.

* **Access key ID**<mark style="color:red;">**\***</mark> - Add the access key from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or create one. The **Access Key ID** is found in the **IAM Dashboard** of the **AWS Management Console**.
  1. In the left panel, click on **Users**.
  2. Select your **IAM user**.
  3. Under the **Security Credentials** tab, scroll to **Access Keys** and you will find existing **Access Key IDs** (but not the secret access key).
* **Secret access key**<mark style="color:red;">**\***</mark> - Add the secret access key from your [Secrets](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management) or create one.&#x20;

  Under **Access keys**, you can see your **Access Key IDs**, but AWS **will not show the Secret Access Key**. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one.
  {% endstep %}

{% step %}
Optionally, specify which **Message system attributes** are wanted in the response. The set of system attributes chosen by the user correspond to attributes inlined in the message/event.

1. In the **Queues** area, click on **More** or scroll down and go to the **Monitoring** tab.
2. You will see some system attributes (like deduplication and group ID). However, detailed system attributes are typically accessed via the CLI or SDKs.
   {% endstep %}

{% step %}
Optionally, configure the settings in the **Advanced** section:

{% hint style="warning" %}
Proceed with caution when modifying the **Advanced** options. Default values should be enough in most cases.
{% endhint %}

* **Service endpoint** - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default.
* **Maximum number of messages**<mark style="color:red;">**\***</mark> - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `1`, and the maximum and default value is `10`.
* **Visibility timeout**<mark style="color:red;">**\***</mark>- Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `1`, and the maximum and default value is `10`.
* **Wait time**<mark style="color:red;">**\***</mark> - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is `5`, and the maximum and default value is `10`.
* **Minimum retry time**<mark style="color:red;">**\***</mark> - Set the minimum amount of time to wait before retrying. The default and minimum value is `1s`, and the maximum value is `10m`.
* **Maximum retry time**<mark style="color:red;">**\***</mark> - Set the minimum amount of time to wait before retrying. The default and minimum value is `1s`, and the maximum value is `10m`.
  {% endstep %}

{% step %}
Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabelled**.  &#x20;

{% hint style="info" %}
Learn more about labels in [this article](https://docs.onum.com/the-workspace/listeners/labels).
{% endhint %}
{% endstep %}

{% step %}
Click **Create listener** when you're done.
{% endstep %}
{% endstepper %}