Collect data from Amazon SQS

Most recent version: v0.0.1

See the changelog of this Listener type here.

This is a Pull Listener and therefore should not be used in environments with more than one cluster.

Overview

Onum supports integration with Amazon SQS.

Amazon Simple Queue Service (AWS SQS) is a fully managed message queuing service. Among its many features, the following ones are of special interest to our use case:

• It supports both standard queues (with at-least-once, occasionally unordered delivery semantics) and FIFO queues (exactly-once and fully ordered delivery semantics).

• It supports scaling through the concept of visibility timeout (a period after a consumer reads one message during which this becomes invisible to other consumers). That allows a consumer group to read from the same queue and distribute messages without duplication.

So, what we want is a Listener that we can configure to read from an existing SQS queue and inject queue messages as events into our platform. Please note that because of the nature of the API offered to access SQS messages (HTTP-based, max 10 messages each time), this is not a high-throughput Listener.

Prerequisites

You will need an IAM User, role or group with the correct permissions to access and manage SQS.

Amazon SQS Setup

Go to IAM (Identity and Access Management) to manage users, groups, roles and permissions.

Under Permissions Policies, make sure you have assigned the policy AmazonSQSFullAccess to give full access to SQS resources. Alternatively, if you have custom permissions, go to Policies - Create Policy and in the JSON tab, paste your custom JSON e.g.

  {
    "Version": "2012-10-17",
      "Statement": [
       {
         "Effect": "Allow"
         "Action": [
         "kinesis:CreateStream",
         "kinesis:DescribeStream",
         "kinesis:PutRecord"
         ],
         "Resource": "*"
 		}        
	]
  }      

Onum Setup

1

Log in to your Onum tenant and click Listeners > New listener.

2

Double-click the AWS SQS Listener.

3

Enter a Name for the new Listener. Optionally, add a Description and some Tags to identify the Listener.

4

Enter the Region displayed in the top right-hand corner of your AWS console.

5

Enter the Queue URL* of your existing Amazon SQS queue, acting as the endpoint to interact with the desired queue. Use the GetQueueUrl command or:

1. Go to the AWS Management Console.

2. In the Search Bar, type "SQS" and click on Simple Queue Service (SQS).

3. Click on Queues in the left panel.

4. Locate your queue from the list and click it.

5. The Queue URL will be displayed in the table under URL.

This is the correct URL format: sqs.region.localhost/awsaccountnumber/storedinenvv

6

Choose your Authentication Type*

Authentication is not specific to SQS but rather AWS IAM (Identity and Access Management). If you are connecting from an IAM console, enter the authentication credentials here.

• Access key ID*

  Add the access key from your Secrets or create one. The Access Key ID is found in the IAM Dashboard of the AWS Management Console.

  1. In the left panel, click on Users.

  2. Select your IAM user.

  3. Under the Security Credentials tab, scroll to Access Keys and you will find existing Access Key IDs (but not the secret access key).

• Secret access key* Add the secret access key from your Secrets or create one.

  Under Access keys, you can see your Access Key IDs, but AWS will not show the Secret Access Key. You must have it saved somewhere. If you don't have the secret key saved, you need to create a new one

7

Optionally, specify which Message system attributes are wanted in the response. The set of system attributes chosen by the user correspond to attributes inlined in the message/event.

1. In the Queues area, click on More or scroll down and go to the Monitoring tab.

2. You will see some system attributes (like deduplication and group ID). However, detailed system attributes are typically accessed via the CLI or SDKs.

8

Proceed with caution when modifying the Advanced options. Default values should be enough in most cases.

9

Proceed with caution when modifying the General advanced options. Default values should be enough in most cases.

• Service endpoint - If you have a custom endpoint, enter it here. The default SQS regional service endpoint will be used by default.

• Maximum number of messages* - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.

• Visibility timeout*- Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 1, and the maximum and default value is 10.

• Wait time* - Set a limit for the maximum number of messages to receive in the notifications queue for each request. The minimum value is 5, and the maximum and default value is 10.

• Minimum retry time* - Set the minimum amount of time to wait before retrying. The default and minimum value is 1s, and the maximum value is 10m.

• Maximum retry time* - Set the minimum amount of time to wait before retrying. The default and minimum value is 1s, and the maximum value is 10m.

10

Finally, click Create labels. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as Unlabelled.

Learn more about labels in this article.

Click Create listener when you're done.