Google BigQuery
Most recent version: v0.0.1
Last updated
Was this helpful?
Most recent version: v0.0.1
Last updated
Was this helpful?
See the changelog of this Data sink type .
The Google BigQuery sink lets you send data to your Google BigQuery storage unit.
Select Google BigQuery from the list of Data sink types and click Configuration to start.
Now you need to specify how and where to send the data, and how to establish a connection with Google BigQuery.
Enter the basic information for the new Data sink.
Name*
Enter a name for the new Data sink.
Description
Optionally, enter a description for the Data sink.
Tags
Add tags to easily identify your Data sink. Hit the Enter key after you define each tag.
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
Now add the configuration to establish the connection.
Project ID*
This is a unique string with the following format my-project-123456
Go to the Google Cloud Console.
In the top left corner, click on the project drop-down next to the Google Cloud logo (where your current project name is shown).
Each project will have a Project Name and a Project ID.
You can also find it in the Settings tab on the left-hand side.
Credentials File*
The Google BigQuery connector uses OAuth 2.0 credentials for authentication and authorization. Create a secret containing these credentials or select one already created.
To find the credentials file, go to Google Cloud Settings>Interoperability.
Scroll down to the Service Account area.
You need to generate and download a service account key from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
To see existing Service Accounts, go to the menu in the top left and select APIs & Services>Credentials.
Click Finish when complete.
When it comes to using this Data sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.
Dataset*
Give a name to the dataset that will appear in your BigQuery storage.
Table*
Enter a name for the table to insert the values into.
Click Add Element to add as many pairs as needed.
Column
Enter a name for the column that will appear in Big Query.
Value
Select the incoming event that contains the value to send.
Set Bulk allow to true to set your bulk settings, or false to ignore.
Event max amount*
Enter the maximum amount of events to accumulate into a bulk. The minimum value is 0 and the maximum value is 5000.
Event max time in seconds*
The number of seconds to wait before considering the bulk as full and sending it on. The minimum value is 0.
Click Save to complete the process.
