Google Cloud Storage
Most recent version: v0.0.2
Last updated
Was this helpful?
Most recent version: v0.0.2
Last updated
Was this helpful?
The Google Cloud Storage Sink sends events to Google Cloud Storage using the HTTP protocol.
Select Google Cloud Storage from the list of Data sink types and click Configuration to start.
Now you need to specify how and where to send the data, and how to establish a connection with Google Cloud.
Enter the basic information for the new Data Sink.
Name*
Enter a name for the new Data Sink.
Description
Optionally, enter a description for the Data Sink.
Tags
Add tags to easily identify your Sink. Hit the Enter key after you define each tag.
Now add the configuration to establish the connection.
Bucket name*
The bucket that will contain your data. Find Google Cloud Storage in the left side menu of the Google Cloud Platform Console, under Storage.
Batch type*
Period to collect every X minutes or number of events to accumulate events.
Period
If you selected period, enter the number of seconds to wait before considering the batch completed.
Number of events
If you selected number of events, enter how many to collect until considered full.
Maximum unique buffers
Enter the maximum amount of buffers.
Compression
Choose Bzip2 or Gzip if you wish to compress your events, none if not. bzip2 compresses single files without archiving them.
The Google Cloud Storage connector uses OAuth 2.0 credentials for authentication and authorization.
Credentials file*
The Google Cloud connector uses OAuth 2.0 credentials for authentication and authorization. Create a secret containing these credentials or select one already created.
To find the Google Cloud credentials file, go to Settings>Interoperability.
Scroll down to the Service Account area.
You need to generate and download a service account key from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
To see existing Service Accounts, go to the menu in the top left and select APIs & Services>Credentials.
Click Finish when complete.
When you use this sink in a Pipeline, you can configure the output parameters. This is where you give the message the required format to be processed in Google Cloud Storage.
Event field*
The incoming field containing the event to be sent on.
Path format
Prefix
The path prefix.
Partition by*
Partition by: this indicates the frequency with which to generate a new Google Cloud Storage object e.g. every year, month, day hour, minute.
File format
Prefix
The prefix of the outgoing file.
Suffix
The suffix of the outgoing file.
Click Save to complete the process.
For information on where to find these values, see .
Decide whether or not to include this Data sink info in the metrics and graphs of the area.
