Google Cloud Storage
Most recent version: v0.0.3
Overview
Onum supports integration with Google Cloud Storage.
Google Cloud Storage is an online object storage service that allows users to store and retrieve data. It is a managed service, meaning Google handles the underlying infrastructure, making it scalable and reliable. GCS is designed for a variety of use cases, including storing data for web applications, big data analytics, and backups.
Select Google Cloud Storage from the list of Data sink types and click Configuration to start.
Data sink configuration
Now you need to specify how and where to send the data, and how to establish a connection with Google Cloud.
Metadata
Enter the basic information for the new Data sink.
Name*
Enter a name for the new Data sink.
Description
Optionally, enter a description for the Data sink.
Tags
Add tags to easily identify your Data sink. Hit the Enter key after you define each tag.
Metrics display
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
Configuration
Now, add the configuration to establish the connection.
Google Cloud Storage Configuration
Bucket name*
The bucket that will contain your data. Access Buckets and enter the required bucket name here.
Batch Configuration
Batch type*
Choose the required batch configuration type:
Send after (X minutes) - Choose this option if you want to send data after a specific time. Enter the required time in the Period parameter below.
Number of events - Choose this option if you want to send data after a specific number of events is reached. Enter the required event amount in the Number of events parameter below.
Period
If you select Send after (X minutes) as your Batch type, enter the number of minutes to wait before considering the batch completed. The minimum value is 1, and the maximum value is 10.
Number of events
If you select Number of events as your Batch type, enter how many to collect until considered full. The minimum value is 1.
Max number of unique buffers
Enter the maximum amount of buffers. The maximum value is 50.
Compression
Choose Bzip2 or Gzip if you wish to compress your events, or None if not. If you are sending events in bzip, you´ll need to unzip them upon download. If you're sending gzip-compressed data, the file unzips automatically when you download it.
Authentication
The Google Cloud Storage connector uses OAuth 2.0 credentials for authentication and authorization.
Credentials file*
The Google Cloud connector uses OAuth 2.0 credentials for authentication and authorization. Create a secret containing these credentials or select one already created.
To find the Google Cloud credentials file, go to Settings > Interoperability.
Scroll down to the Service Account area.
You need to generate and download a service account key from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
To see existing Service Accounts, go to the menu in the top left and select APIs & Services > Credentials.
Click Finish when complete. Your new Data sink will appear in the Data sinks area list.
Pipeline configuration
When you use this sink in a Pipeline, you can configure the output parameters. This is where you give the message the required format to be processed in Google Cloud Storage.
Output configuration
Event field*
Choose the incoming field containing the event to be sent on.
Path format
Prefix
Optionally, enter a static prefix for the file path.
Partition by*
Choose the frequency with which to generate a new Google Cloud Storage object.
File format
Prefix
Enter a static prefix for the outgoing file.
Suffix
Enter a static suffix for the outgoing file.
Click Save to complete the process.
Last updated
Was this helpful?