# Send data to Google Pub/Sub

{% hint style="info" %}
See the changelog of the **Google Pub/Sub** Data sink type [here](https://app.gitbook.com/s/zgCDPkYt2378cQelEdK8/google-pub-sub-data-sink).
{% endhint %}

## Overview

Onum supports integration with [Google Pub/Sub](https://cloud.google.com/pubsub/docs/overview).

Google Pub/Sub is an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages. Pub/Sub allows services to communicate asynchronously.

## Prerequisites

You will need:

* a [Google Cloud account](https://app.gitbook.com/u/WYsFmD7ND4SwgucUwxcsTxMTNhg2)
* an existing project in the [Google Cloud Console](https://console.cloud.google.com/projectselector2/home/dashboard)

## Google Cloud Storage Setup 

* Once you have your project to connect to, make sure your user or service account has the proper roles e.g. `roles/storage.admin`  Go to **IAM & Admin > IAM > Add Principal** and choose a role.
* Enable the **Cloud Storage API** in your project.

## Onum setup

{% stepper %}
{% step %}
Log in to your Onum tenant and click **Data Sinks> New Data sink**.
{% endstep %}

{% step %}
Double-click the **Google Pub/Sub** Sink.
{% endstep %}

{% step %}
Enter a **Name** for the new Data Sink. Optionally, add a **Description** and some **Tags** to identify the Sink.
{% endstep %}

{% step %}
Decide whether or not to include this Data sink info in the metrics and graphs of the [**Home**](https://docs.onum.com/the-workspace/home) area.
{% endstep %}

{% step %}
Enter the **Project ID**<mark style="color:red;">**\***</mark> unique string with the following format `my-project-123456`.&#x20;

To get it:

1. Go to the Google Cloud Console.
2. In the top left corner, click on the **project drop-down** next to the Google Cloud logo (where your current project name is shown).
3. Each project will have a **Project Name** and a **Project ID**.
4. You can also find it in the **Settings** tab on the left-hand side.

<figure><img src="https://965373739-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkxZeV4nlXcIAjMGZxzLI%2Fuploads%2FOMpEN1NqtexX9BZvBaqv%2Fimage.png?alt=media&#x26;token=33b0a2c9-7bb2-4bbb-837e-013245441009" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Publish a message to a specific topic using the **Topic ID**<mark style="color:red;">**\***</mark> found in the Pub/Sub **Topics** area.

<figure><img src="https://965373739-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkxZeV4nlXcIAjMGZxzLI%2Fuploads%2F4CKw8AdMzEUklKwc1FBk%2Flight-large%20(18).png?alt=media&#x26;token=d56af2e4-5b7d-4633-99cb-95ce8b95cea5" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Enter either a global or regional endpoint to connect to, if you need to override the default regional endpoint. Regional endpoints are in the format:

\
`https://<REGION_NAME>-pubsub.googleapis.com`

\
Note that only gRPC is supported. HTTP is not supported. For the list of endpoints, check [this article](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_of_locational_endpoints).
{% endstep %}

{% step %}
The Google Cloud Pub/Sub connector uses OAuth 2.0 credentials for authentication and authorization.&#x20;

Enter the **Credentials File**<mark style="color:red;">**\***</mark> by creating a [secret](https://docs.onum.com/administration) containing these credentials or select one already created. To find the credentials file:

1. Go to Google Cloud **Settings > Interoperability**.
2. Scroll down to the **Service Account** area.
3. You need to generate and download a **service account key** from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
4. To see existing Service Accounts, go to the menu in the top left and select **APIs & Services > Credentials**.

<figure><img src="https://965373739-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkxZeV4nlXcIAjMGZxzLI%2Fuploads%2FWJXJq6rsEwZwP5CSwkXy%2Flight-large%20(17).png?alt=media&#x26;token=6bb0380b-a469-4ccc-8569-03db8172156a" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **New secret** to create a new one:

* Give the secret a **Name**.
* Turn off the **Expiration date** option.
* Click **Add new value** and paste the secret corresponding to the JWT token you generated before. Remember that the token will be added in the Zscaler configuration.
* Click **Save**.

<figure><picture><source srcset="https://965373739-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkxZeV4nlXcIAjMGZxzLI%2Fuploads%2FlUo7CuVpPgIVm5VNjLw6%2Fnenenew.png?alt=media&#x26;token=eb7a7231-0ac2-4099-93f9-18f9ead5add1" media="(prefers-color-scheme: dark)"><img src="https://965373739-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkxZeV4nlXcIAjMGZxzLI%2Fuploads%2FTSD53FxGQOjijA3W3DhE%2Fimage.png?alt=media&#x26;token=9941a3c0-100a-4759-b603-30079fbc90de" alt=""></picture><figcaption></figcaption></figure>

{% hint style="info" %}
Learn more about secrets in Onum in [this article](https://docs.onum.com/administration/global-settings/organization-settings/secrets-management).
{% endhint %}
{% endstep %}
{% endstepper %}

Click **Create data sink** when complete.&#x20;

Your new Data sink will appear in the **Data sinks** area list.

## Pipeline configuration

When it comes to using this Data sink in a [Pipeline](https://docs.onum.com/the-workspace/pipelines), you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select **Configuration**.

### Output configuration

**Message**<mark style="color:red;">\*</mark> - Select the incoming field containing the message to send on.

**Add Attributes** (if required) to insert specific pipeline attributes into the outgoing message.

* **Attribute name**<mark style="color:red;">\*</mark> - Write a name to recognise the value by.
* **Value**<mark style="color:red;">\*</mark> **-** Select the value from the drop-down of Pipeline fields or enter it manually.

**Advanced Configuration**

Toggle ON **Bulk Configuration** and enter values, or simply leave them as the default values.

<table><thead><tr><th width="169.70703125">Parameter</th><th>Description</th></tr></thead><tbody><tr><td><strong>Event amount</strong> </td><td>Enter the maximum number of events to accumulate into a batch. The default value is <code>10</code>.</td></tr><tr><td><strong>Event time limit</strong> </td><td>The number of seconds to wait before considering the bulk as full and sending it on. The default value is <code>10</code>.</td></tr><tr><td><strong>Batch publish timeout</strong></td><td>The number of milliseconds to wait before considering a request as timed out. The default value is <code>100</code>.</td></tr><tr><td><strong>Event max size</strong> </td><td>The maximum bytes to allow through a batch. The default value is <code>1024</code>.</td></tr><tr><td><strong>Max queue size</strong></td><td>The maximum number of messages that can be pending acknowledgment. The default value is <code>100</code>.</td></tr><tr><td><strong>Max queue size bytes</strong></td><td>Maximum number of bytes that can be pending acknowledgment. The default value is <code>1024</code>.</td></tr><tr><td><strong>Max workers</strong></td><td>Number of concurrent workers used to send messages in parallel. The default value is <code>10</code>.</td></tr></tbody></table>

Click **Save** to complete the process.