Send data to Google Pub/Sub
Most recent version: v1.0.0
Overview
Onum supports integration with Google Pub/Sub.
Google Pub/Sub is an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages. Pub/Sub allows services to communicate asynchronously.
Prerequisites
You will need:
an existing project in the Google Cloud Console
Google Cloud Storage Setup
Once you have your project to connect to, make sure your user or service account has the proper roles e.g.
roles/storage.adminGo to IAM & Admin > IAM > Add Principal and choose a role.Enable the Cloud Storage API in your project.
Onum setup
Log in to your Onum tenant and click Data Sinks> New Data sink.
Double-click the Google Pub/Sub Sink.
Enter a Name for the new Data Sink. Optionally, add a Description and some Tags to identify the Sink.
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
Enter the Project ID* unique string with the following format my-project-123456.
To get it:
Go to the Google Cloud Console.
In the top left corner, click on the project drop-down next to the Google Cloud logo (where your current project name is shown).
Each project will have a Project Name and a Project ID.
You can also find it in the Settings tab on the left-hand side.
Publish a message to a specific topic using the Topic ID* found in the Pub/Sub Topics area.
Enter either a global or regional endpoint to connect to, if you need to override the default regional endpoint. Regional endpoints are in the format:
https://<REGION_NAME>-pubsub.googleapis.com
Note that only gRPC is supported. HTTP is not supported. For the list of endpoints, check this article.
The Google Cloud Pub/Sub connector uses OAuth 2.0 credentials for authentication and authorization.
Enter the Credentials File* by creating a secret containing these credentials or select one already created. To find the credentials file:
Go to Google Cloud Settings > Interoperability.
Scroll down to the Service Account area.
You need to generate and download a service account key from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
To see existing Service Accounts, go to the menu in the top left and select APIs & Services > Credentials.
Click New secret to create a new one:
Give the secret a Name.
Turn off the Expiration date option.
Click Add new value and paste the secret corresponding to the JWT token you generated before. Remember that the token will be added in the Zscaler configuration.
Click Save.
Learn more about secrets in Onum in this article.
If you want to use bulk sending, toggle on the Bulk configuration button and configure the following settings:
Event amount
Enter the maximum number of events to accumulate into a batch. The default value is 10.
Event time limit
The number of seconds to wait before considering the bulk as full and sending it on. The default value is 10.
Batch publish timeout
The number of milliseconds to wait before considering a request as timed out. The default value is 100.
Event max size
The maximum bytes to allow through a batch. The default value is 1024.
Max queue size
The maximum number of messages that can be pending acknowledgment. The default value is 100.
Max queue size bytes
Maximum number of bytes that can be pending acknowledgment. The default value is 1024.
Max workers
Number of concurrent workers used to send messages in parallel. The default value is 10.
Click Create data sink when complete.
Your new Data sink will appear in the Data sinks area list.
Pipeline configuration
When it comes to using this Data sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.
Output configuration
Message*
Select the incoming field containing the message to send on.
Click Save to complete the process.
Last updated
Was this helpful?