Orchestrate BigQuery Workloads with Dataform
Dataform is a service for data analysts to develop, test, version control, and schedule complex SQL workflows for data transformation in BigQuery. In this course you will explore the components of Dataform core, learn how to define tables and dependencies in SQLX, document BigQuery tables and views, understand BigQuery security settings and how to manage these with Dataform, write assertions, execute SQL workflows, and explore additional advanced use cases.
- Duration: 1 day
- Languages: English, German
- Level: Intermediate
- Group size: Up to 16
What you'll learn
- Understand the components of Dataform core.
- Create tables and views in BigQuery using Dataform.
- Document BigQuery tables and views.
- Understand BigQuery security settings using Dataform.
- Use assertions to validate data in Dataform workflows.
- Execute Dataform SQL workflows in an automated fashion.
Prerequisites
- Knowledge of SQL data analysis and BigQuery as discussed in BigQuery for Data Analysis.
Course outline
Dataform Core Components
- SQL workflow
- Repositories and workspaces
- Default files and folders
- Compiled graphs
Table Definitions and Dependencies
- Declare a data source.
- Create a table.
- Create an incremental table.
- Set partitioning and clustering options.
- Create an empty table.
- Create an external BigLake table.
- Create views and materialized views.
- Define dependencies.
Document BigQuery Tables and Views
- Use column descriptions.
- Use globally defined JavaScript constants.
- Add labels.
BigQuery Security Settings
- IAM dataset and table/view access
- Column-level security
- Row-level security
Assertions
- Use built-in assertions.
- Create manual assertions.
- Use assertions to validate data in Dataform workflows.
SQL Workflow Executions
- Dataform code lifecycle.
- What happens during compilation.
- Customize and schedule compilation results.
- Execute workflows (UI, Cloud Scheduler, Cloud Composer).
- Logging and monitoring.
Advanced Use Cases
- Create a BigLake table after file upload using Cloud Run functions.
- Build a Machine Learning pipeline with BigQuery ML.
- Work with Slowly Changing Dimensions Type 2.
Related courses
AlloyDB Essentials
This course explores the benefits of AlloyDB, especially compared to PostgreSQL on Cloud SQL. It will walk you through AlloyDB’s unique architecture, and explain how to configure deployments on Google Cloud.The course is divided into two parts: AlloyDB Administration Essentials (architecture and configuration) and AlloyDB Optimization Essentials (performance tuning).
BigQuery for Data Analysts
This course is designed for data analysts who want to learn about using BigQuery for their data analysis needs. Through a combination of videos, labs, and demos, we cover various topics that discuss how to ingest, transform, and query your data in BigQuery to derive insights that can help in business decision-making.
Data Engineering on Google Cloud
Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hands-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data.
Not sure which course fits?
Book a 30-minute call. We'll look at where your team is, what they need to be able to do, and put together either a specific course date or a tailored learning plan.