Let us start with the definition first :
BigQuery is an application that enables fully managed data warehouse that operates without any serverless architecture.In fact,the BigQuery enables analysis of petabytes of data at once and makes the aspect of data analysis easier. BigQuery has features, which helps in maintaining impeccable stance on the analysis of data with the help of geospacial analysis, business intelligence and machine learning features.
If anyone wants to know about it in details, here is the link
But during my time in Rackspace, where I encountered some of the use cases in my project by which maybe I have forgotten the official “definition”, but I will remember the actual use of this.
Let’s see what are options we can explore in terms of querying the data outside the GCP environment.
There are three methods to do that.
“Intelligence is the ability to adapt to change.” – Stephen Hawking
This is a very popular method and the main advantage of this method is, you don’t need to give admin/owner level access.
Let’s see it’s implementation and testing. We need only postman application for the testing.
Create OAuth Credentials
We will use Google uses OAuth 2.0 protocol for authentication and authorization. You will need a valid or approved key to access to the system and send requests to the BigQuery streaming endpoints. We need to avail the OAuth authorization framework 2.0 and retrieve the credentials of the client from the Google Cloud API.
Login to GCP console and after that navigate to APIs & Services. Then enable the BigQuery API.
Then go to “Credential” option and Create an OAuth 2.0 Client IDs
Please see the below configuration.
This will create OAuth ID. Save these credentials (JSON) in your machine. We will use these credentials in Postman to generate a token for every request to the BigQuery streaming API.
Now our task is done in GCP Console. Let’s open the Postman application
Generating Token in Postman
In the authorization section, click on “type” and select OAuth 2.0
Now Configure the following parameters:
Then click the “Get New Access Token”
This will take you to Google cloud console page for login and authentication. Enter your credentials. An access token will be generated and you will be redirected to the Postman. Click on “use token”
Navigate to “Headers” section and enter “Content-Type” in Key and “application/json” in Value.
Navigate to “Body” section and enter the JSON request body. You will construct the body with the data to be sent to BigQuery.
Result in BigQuery
Now Let’s talk about method 2:
Prerequisite – Before setting the environment you need to install the following package in the python environment.
pip install --upgrade google-cloud-bigquery
First step is set up authentication to run client library.One way to do that is to create a service account and set an environment variable.
Create a service account key:
You can store this key in GCS Bucket or Passwordsafe Project. It will be needed for authentication.
For Example :
I created one table in BigQuery for testing
Now please find the below python code for reference.
REST API –
Python Client –
Those methods have their pros and cons, and it is perfectly synchronized with their use cases.
This discussion out of this scope, as this method mainly using to connect any application like Tableau,QlikSense etc to BigQuery.
For more information on writing and citing sources, explore the following resources:
Use the Feedback tab to make any comments or ask questions. You can also start a conversation with us.