Data Engineering on Google Cloud
Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hands-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data.
- Duration: 4 days
- Languages: English, German
- Level: Intermediate
- Group size: Up to 16
What you'll learn
- Design scalable data processing systems in Google Cloud.
- Differentiate data architectures and implement data lakehouse and pipeline concepts.
- Build and manage robust streaming and batch data pipelines.
- Utilize AI/ML tools to optimize performance and gain process and data insights.
Prerequisites
- Understanding of data engineering principles, including ETL/ELT processes, data modeling, and common data formats (Avro, Parquet, JSON).
- Familiarity with data architecture concepts, specifically Data Warehouses and Data Lakes.
- Proficiency in SQL for data querying.
- Proficiency in a common programming language (Python recommended).
- Familiarity with using Command Line Interfaces (CLI).
- Familiarity with core Google Cloud concepts and services (Compute, Storage, and Identity management).
Course outline
Data Engineering Tasks and Components
- The role of a data engineer
- Data sources versus data sinks
- Data formats
- Storage solution options on Google Cloud
- Metadata management options on Google Cloud
- Sharing datasets using Analytics Hub
Data Replication and Migration
- Replication and migration architecture
- The gcloud command-line tool
- Moving datasets
- Datastream
The Extract and Load Data Pipeline Pattern
- Extract and load architecture
- The bq command-line tool
- BigQuery Data Transfer Service
- BigLake
The Extract, Load, and Transform Data Pipeline Pattern
- Extract, load, and transform (ELT) architecture
- SQL scripting and scheduling with BigQuery
- Dataform
The Extract, Transform, and Load Data Pipeline Pattern
- Extract, transform, and load (ETL) architecture
- Google Cloud GUI tools for ETL data pipelines
- Batch data processing using Dataproc
- Streaming data processing options
- Bigtable and data pipelines
Automation Techniques
- Automation patterns and options for pipelines
- Cloud Scheduler and Workflows
- Cloud Composer
- Cloud Run Functions
- Eventarc
Introduction to Modern Data Engineering on Google Cloud
- The classics: Data lakes and data warehouses
- The modern approach: Data lakehouse
- Choosing the right architecture
Building a data lakehouse with Cloud Storage, open formats, and BigQuery
- Building a data lake foundation
- Introduction to Apache Iceberg open table format
- BigQuery as the central processing engine
- Combining operational data in AlloyDB
- Combining operational and analytical data with federated queries
- Real world use case
Modernizing Data Warehouses with BigQuery and BigLake
- BigQuery fundamentals
- Partitioning and clustering in BigQuery
- Introducing BigLake and external tables
Advanced lakehouse patterns and data governance
- Data governance and security in a unified platform
- Demo: Data Loss Prevention
- Analytics and machine learning on the lakehouse
- Real-world lakehouse architectures and migration strategies
Labs and best practices
- Review
- Best practices
When to choose batch data pipelines
- Batch data pipelines and their use cases
- Processing and common challenges
Design and Build Scalable Batch Data Pipelines
- Design batch pipelines
- Large scale data transformations
- Dataflow and Serverless for Apache Spark
- Data connections and orchestration
- Execute an Apache Spark pipeline
- Optimize batch pipeline performance
Control Data Quality in Batch Data Pipelines
- Batch data validation and cleansing
- Log and analyze errors
- Schema evolution for batch pipelines
- Data integrity and duplication
- Deduplication with Serverless for Apache Spark
- Deduplication with Dataflow
Orchestrate and Monitor Batch Data Pipelines
- Orchestration for batch processing
- Cloud Composer
- Unified observability
- Alerts and troubleshooting
- Visual pipeline management
Course introduction
- Course learning objectives
- Course prerequisites
- The use case, company, challenge, and mission
Streaming use cases and reference architectures
- Introduction to streaming data pipelines on Google Cloud
- Streaming ETL
- Streaming AI/ML
- Streaming applications
- Reverse ETL
Product deep dives
- Architectural considerations for Pub/Sub and Managed Service for Apache Kafka
- Dataflow: The processing powerhouse
- BigQuery: The analytical engine
- Bigtable: The solution for operational data
Key takeaways
- What you've accomplished
- Next steps
Related courses
AlloyDB Essentials
This course explores the benefits of AlloyDB, especially compared to PostgreSQL on Cloud SQL. It will walk you through AlloyDB’s unique architecture, and explain how to configure deployments on Google Cloud.The course is divided into two parts: AlloyDB Administration Essentials (architecture and configuration) and AlloyDB Optimization Essentials (performance tuning).
BigQuery for Data Analysts
This course is designed for data analysts who want to learn about using BigQuery for their data analysis needs. Through a combination of videos, labs, and demos, we cover various topics that discuss how to ingest, transform, and query your data in BigQuery to derive insights that can help in business decision-making.
Data Integration with Cloud Data Fusion
This 2-day course introduces learners to Google Cloud’s data integration capability using Cloud Data Fusion. In this course, we discuss challenges with data integration and the need for a data integration platform (middleware). We then discuss how Cloud Data Fusion can help to effectively integrate data from a variety of sources and formats and generate insights. We take a look at Cloud Data Fusion’s main components and how they work, how to process batch data and real time streaming data with visual pipeline design, rich tracking of metadata and data lineage, and how to deploy data pipelines on various execution engines.
Not sure which course fits?
Book a 30-minute call. We'll look at where your team is, what they need to be able to do, and put together either a specific course date or a tailored learning plan.