> For the complete documentation index, see [llms.txt](https://docs.onum.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.onum.com/listeners/listener-integrations/collect-data-from-google-cloud-storage.md).

# Collect data from Google Cloud Storage

{% hint style="info" %}
See the changelog of the **Google Cloud Storage** Listener [here](/listeners-changelog/google-cloud-storage-listener.md).
{% endhint %}

{% hint style="warning" %}

* The **Google Cloud Storage** Listener is only available in certain Tenants. [Get in touch with us](broken://spaces/cSjT21I4EUhzghjc1rER/pages/nW8oZycpfVthtI5KYdQs) if you don't see it and want to access it.
* The **Google Cloud Storage** Listener is a **Pull** Listener and therefore should not be used in environments with more than one cluster. 
  {% endhint %}

## Overview

Onum supports integration with [Google Cloud Storage](https://cloud.google.com/storage?hl=en).

Google Cloud Storage is an online object storage service that allows users to store and retrieve data. It is a managed service, meaning Google handles the underlying infrastructure, making it scalable and reliable. GCS is designed for a variety of use cases, including storing data for web applications, big data analytics, and backups.

## Prerequisites

{% hint style="warning" %}
In order to use this Listener, you must activate the environment variable in your distributor using docker compose (`GOOGLE_CLOUD_STORAGE_LISTENER_EXECUTION_ENABLED`)
{% endhint %}

## Google Cloud Storage Setup

To source data from Google Cloud Storage you need to have a GCS bucket with data, appropriate permissions (like `Storage Admin`) to access the bucket and its objects, and the correct resource path (e.g., `gs://bucket-name/object-name`).

See [the Google Cloud Storage manual](https://cloud.google.com/appengine/docs/legacy/standard/python/googlecloudstorageclient/setting-up-cloud-storage) for help.

## Onum Setup

{% stepper %}
{% step %}
Log in to your Onum tenant and click **Listeners > New listener**.
{% endstep %}

{% step %}
Double-click the **Google Cloud Storage** Listener.
{% endstep %}

{% step %}
Enter a **Name** for the new Listener. Optionally, add a **Description** and some **Tags** to identify the Listener.
{% endstep %}

{% step %}
The Google Cloud connector uses OAuth 2.0 credentials for authentication and authorization. In the **Credentials file** field, create a new [Secret](/settings/organization-settings/secrets-management.md) containing these credentials or select one already created. To get it:

1. To find the **Google Cloud credentials file**, go to **Settings > Interoperability**.
2. Scroll down to the **Service Account** area.
3. You need to generate and download a **service account key** from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
4. To see existing Service Accounts, go to the menu in the top left and select **APIs & Services > Credentials**.

{% hint style="info" %}
Learn more about secrets in Onum in [this article](/settings/organization-settings/secrets-management.md).
{% endhint %}
{% endstep %}

{% step %}
Assign an optional **Event delimiter** to split file content into different events using a delimiter (Examples: `-`, `\n`, `\r\n`, `0x0A`...).
{% endstep %}

{% step %}
Choose the **Compression type** for your files (**None**, **Gzip**, **Bzip2** or **Auto**). 
{% endstep %}

{% step %}
If you set the **Read Bucket Once** parameter to **true**, the Listener will read the entire bucket once and stop the execution. You'll be prompted to enter the following:

* **Prefix** - The optional string that acts like a folder path or directory **structure** when organizing objects within a bucket.
* **Bucket** - Enter the GCP bucket name.
* **Start at** - This will block the Listener from starting until this timestamp. The required date format is `DD/MM/YYYY HH:mm`. The specified time must be in the future and conform to the timezone where the operation is being executed.
  {% endstep %}

{% step %}
The **Project ID** is a unique string with the following format: `my-project-123456`. To get it:   

1. Go to the Google Cloud Console.
2. In the top left corner, click on the **project drop-down** next to the Google Cloud logo (where your current project name is shown).
3. Each project will have a **Project Name** and a **Project ID**.
4. You can also find it in the **Settings** tab on the left-hand side.
   {% endstep %}

{% step %}
Enter your **Subscription** (called **Subscription ID** in the Cloud Console). Follow these steps to get it:

1. Go to **Pub/Sub** in the **Google Cloud Console**.
2. In the top left corner, click on the **menu** and select **View all Products**.
3. Then go to **Analytics** and find **Pub/Sub**. Click it to go to Pub/Sub (you can also use the search bar and type `Pub/Sub`).
4. In the **Pub/Sub** dashboard, select the **Subscriptions** tab on the left.
5. The **Subscription ID** will be displayed in this list.
   {% endstep %}

{% step %}
In case of a failure to connect, enter the following parameters:

* **Number of retries** - Enter the maximum number of retries to perform in case of a failure. The minimum value is `1`, and the maximum value is `5`. The default value is `3`.
* **Retry delay** - Enter the number of milliseconds to wait between retries. The minimum and default value is `100`, and the maximum value is `1000`.
  {% endstep %}

{% step %}
Finally, click **Create labels**. Optionally, you can set labels to be used for internal Onum routing of data. By default, data will be set as **Unlabeled**.

{% hint style="info" %}
Learn more about labels in [this article](/listeners/labels.md).
{% endhint %}
{% endstep %}

{% step %}
Click **Create listener** when you're done.
{% endstep %}
{% endstepper %}

## Output Ports <a href="#ports" id="ports"></a>

The **Google Cloud Storage** Listener has two output ports:

* **Default port** - Events are sent through this port if no error occurs while processing them.
* **Error port** - Events are sent through this port if an error occurs while processing them.