Google BigQuery
Most recent version: v0.0.1
Last updated
Was this helpful?
Most recent version: v0.0.1
Last updated
Was this helpful?
Onum supports integration with Google BigQuery.
Google BigQuery is an autonomous data-to-AI platform, automating the entire data life cycle, from ingestion to AI-driven insights, so you can go from data to AI to action faster.
Select Google BigQuery from the list of Data sink types and click Configuration to start.
Now you need to specify how and where to send the data, and how to establish a connection with Google BigQuery.
Enter the basic information for the new Data sink.
Name*
Enter a name for the new Data sink.
Description
Optionally, enter a description for the Data sink.
Tags
Add tags to easily identify your Data sink. Hit the Enter key after you define each tag.
Decide whether or not to include this Data sink info in the metrics and graphs of the Home area.
Now, add the configuration to establish the connection.
Project ID*
This is a unique string with the following format my-project-123456. To get it:
Go to the Google Cloud Console.
In the top left corner, click on the project drop-down next to the Google Cloud logo (where your current project name is shown).
Each project will have a Project Name and a Project ID.
You can also find it in the Settings tab on the left-hand side.
Credentials File*
The Google BigQuery connector uses OAuth 2.0 credentials for authentication and authorization.
Go to Google Cloud Settings > Interoperability.
Scroll down to the Service Account area.
You need to generate and download a service account key from the Google Cloud Console. You will not be able to view this key, so you must have it copied somewhere already. Otherwise, create one here and save it to paste here.
To see existing Service Accounts, go to the menu in the top left and select APIs & Services > Credentials.
Click Finish when complete. Your new Data sink will appear in the Data sinks area list.
When it comes to using this Data sink in a Pipeline, you must configure the following output parameters. To do it, simply click the Data sink on the canvas and select Configuration.
Dataset*
Give a name to the dataset that will appear in your BigQuery storage.
Table*
Enter a name for the table to insert the values into.
Click Add Element to add as many pairs as needed.
Column*
Enter a name for the column that will appear in Big Query.
Value*
Select the incoming event that contains the value to send.
Bulk allow*
Set this to true and configure the options below if you want to set bulk sending. Otherwise, set it to false.
Event max amount*
Enter the maximum number of events to accumulate into a bulk. The minimum value is 0 and the maximum value is 5000.
Event max time in seconds*
The number of seconds to wait before considering the bulk as full and sending it on. The minimum value is 0.
Click Save to complete the process.
Create a containing these credentials or select one already created. To find the credentials file:
