Google Cloud Storage
Most recent version: v1.0.1
This is a Pull Listener and therefore should not be used in environments with more than one cluster.
Note that this Listener is only available in certain Tenants. Get in touch with us if you don't see it and want to access it.
Overview
Onum supports integration with Google Cloud Storage.
Google Cloud Storage is an online object storage service that allows users to store and retrieve data. It is a managed service, meaning Google handles the underlying infrastructure, making it scalable and reliable. GCS is designed for a variety of use cases, including storing data for web applications, big data analytics, and backups.
Select Google Cloud Storage from the list of Listener types and click Configuration to start.
Listener configuration
Now you need to specify how and where to collect the data, and establish a connection with your Google account.
Metadata
Enter the basic information for the new Listener.
Name*
Enter a name for the new Listener.
Description
Optionally, enter a description for the Listener.
Tags
Add tags to easily identify your Listener. Hit the Enter key after you define each tag.
Configuration
Now add the configuration to establish the connection.
Credentials File*
The Google Cloud connector uses OAuth 2.0 credentials for authentication and authorization. Create a new Secret containing these credentials or select one already created. To get it:
To find the Google Cloud credentials file, go to Settings > Interoperability.
Scroll down to the Service Account area.
You need to generate and download a service account key from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
To see existing Service Accounts, go to the menu in the top left and select APIs & Services > Credentials.
Event delimiter
Assign an optional delimiter to simulate a hierarchical directory structure within a flat namespace.
Compression type*
Choose the compression type for your files (None, Gzip, Bzip2 or Auto).
Read Bucket Once
If you set this parameter to true, the Listener will read the entire bucket once and stop the execution. You'll be prompted to enter the following:
Prefix - The optional string that acts like a folder path or directory structure when organizing objects within a bucket.
Bucket* - Enter the GCP bucket name.
Start at* - This will block the Listener from starting until this timestamp. The required date format is
DD/MM/YYYY HH:mm.
If you select false, you must enter the Project ID and Subscription parameters below.
Project ID*
This is a unique string with the following format: my-project-123456. To get it:
Go to the Google Cloud Console.
In the top left corner, click on the project drop-down next to the Google Cloud logo (where your current project name is shown).
Each project will have a Project Name and a Project ID.
You can also find it in the Settings tab on the left-hand side.
Subscription*
Enter your subscription name. Follow these steps to get it:
Go to Pub/Sub in the Google Cloud Console.
In the top left corner, click on the menu and select View all Products.
Then go to Analytics and find Pub/Sub. Click it to go to Pub/Sub (you can also use the search bar and type "Pub/Sub").
In the Pub/Sub dashboard, select the Subscriptions tab on the left.
The Subscription Name will be displayed in this list.
Number of retries*
Enter the maximum number of retries to perform in case of a failure. The minimum value is 1, and the maximum value is 5. The default value is 3.
Retry delay*
Enter the number of milliseconds to wait between retries. The minimum and default value is 100, and the maximum value is 1000.
Click Create labels to move on to the next step and define the required Labels if needed.
Last updated
Was this helpful?