Send data to Google Cloud Storage
Most recent version: v0.0.3
Overview
Onum supports integration with Google Cloud Storage.
Google Cloud Storage is an online object storage service that allows users to store and retrieve data. It is a managed service, meaning Google handles the underlying infrastructure, making it scalable and reliable. GCS is designed for a variety of use cases, including storing data for web applications, big data analytics, and backups.
Prerequisites
You will need:
an existing project in the Google Cloud Console
Google Cloud Storage Setup
Once you have your project to connect to, make sure your user or service account has the proper roles e.g.
roles/storage.adminGo to IAM & Admin > IAM > Add Principal and choose a role.Enable the Cloud Storage API in your project.
Onum setup
Log in to your Onum tenant and click Data Sinks> New Data sink.
Double-click the Google BigQuery Sink.
Enter a Name for the new Data Sink. Optionally, add a Description and some Tags to identify the Sink.
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
The Bucket name* field is the bucket that will contain your data. Access Buckets and enter the required bucket name here.
Choose the required Batch type*:
Send after (X minutes) - Choose this option if you want to send data after a specific time. Enter the required time in the Period parameter below.
Number of events - Choose this option if you want to send data after a specific number of events is reached. Enter the required event amount in the Number of events parameter below.
After choosing the type, configure the specifics:
Period If you select Send after (X minutes) as your Batch type, enter the number of minutes to wait before considering the batch completed. The minimum value is
1, and the maximum value is10.Number of events If you select Number of events as your Batch type, enter how many to collect until considered full. The minimum value is
1.Max number of unique buffers Enter the maximum amount of buffers. The maximum value is
50.Compression Choose Bzip2 or Gzip if you wish to compress your events, or None if not. If you are sending events in bzip, you´ll need to unzip them upon download. If you're sending gzip-compressed data, the file unzips automatically when you download it.
The Google Cloud Storage connector uses OAuth 2.0 credentials for authentication and authorization.
Enter the Credentials File* by creating a secret containing these credentials or select one already created. To find the credentials file:
Go to Google Cloud Settings > Interoperability.
Scroll down to the Service Account area.
You need to generate and download a service account key from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
To see existing Service Accounts, go to the menu in the top left and select APIs & Services > Credentials.
Click New secret to create a new one:
Give the secret a Name.
Turn off the Expiration date option.
Click Add new value and paste the secret corresponding to the JWT token you generated before. Remember that the token will be added in the Zscaler configuration.
Click Save.
Learn more about secrets in Onum in this article.
Click Create data sink when complete.
Your new Data sink will appear in the Data sinks area list.
Pipeline configuration
When it comes to using this Data sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.
Output configuration
Choose the incoming Event field* containing the event to be sent on.
Path format
Optionally, enter a static prefix for the file path.
Choose the frequency with which to generate a new Google Cloud Storage object using Partition by*
File format
Enter a static prefix and suffix for the outgoing file.
Click Save to complete the process.
Last updated
Was this helpful?