Introduction to Data Engineering on Google Cloud
In this course, you learn about data engineering on Google Cloud, the roles and responsibilities of data engineers, and how those map to offerings provided by Google Cloud. You also learn about ways to address data engineering challenges.
- Duration: 1d 8h
- Languages: English, German
- Level: Introductory
- Group size: Up to 16
What you'll learn
- Understand the role of a data engineer.
- Identify data engineering tasks and core components used on Google Cloud.
- Understand how to create and deploy data pipelines of varying patterns on Google Cloud.
- Identify and utilize various automation techniques on Google Cloud.
Prerequisites
- Prior Google Cloud experience at the fundamental level using Cloud Shell and accessing products from the Google Cloud console.
- Basic proficiency with a common query language such as SQL.
- Experience with data modeling and ETL (extract, transform, load) activities.
- Experience developing applications using a common programming language such as Python.
Course outline
Data Engineering Tasks and Components
- The role of a data engineer
- Data sources versus data sinks
- Data formats
- Storage solution options on Google Cloud
- Metadata management options on Google Cloud
- Sharing datasets using Analytics Hub
Data Replication and Migration
- Replication and migration architecture
- The gcloud command-line tool
- Moving datasets
- Datastream
The Extract and Load Data Pipeline Pattern
- Extract and load architecture
- The bq command-line tool
- BigQuery Data Transfer Service
- BigLake
The Extract, Load, and Transform Data Pipeline Patter
- Extract, load, and transform (ELT) architecture
- SQL scripting and scheduling with BigQuery
- Dataform
The Extract, Transform, and Load Data Pipeline Pattern
- Extract, transform, and load (ETL) architecture
- Google Cloud GUI tools for ETL data pipelines
- Batch data processing using Dataproc
- Streaming data processing options
- Bigtable and data pipelines
Automation Techniques
- Automation patterns and options for pipelines
- Cloud Scheduler and Workflows
- Cloud Composer
- Cloud Run Functions
- Eventarc
Related courses
Analyzing and Visualizing Data in Looker
In this course, you learn how to do the kind of data exploration and analysis in Looker that would formerly be done primarily by SQL developers or analysts. You learn to leverage Looker's modern analytics platform to find and explore relevant content in your organization’s Looker instance. You also discover how to ask questions of your data, create new metrics as needed, and build and share visualizations and dashboards to facilitate data-driven decision-making.
Introduction to Data Analytics on Google Cloud
This course is an introduction to data analytics on Google Cloud. It is designed for learners who have no prior experience with data analytics or Google Cloud. The course covers the basics of data analysis, including collection, storage, exploration, visualization, and sharing. It also introduces learners to Google Cloud's data analytics tools and services. Through video lectures, demos, quizzes, and hands-on labs, the course demonstrates how to go from raw data to impactful visualizations and dashboards.
AlloyDB Essentials
This course explores the benefits of AlloyDB, especially compared to PostgreSQL on Cloud SQL. It will walk you through AlloyDB’s unique architecture, and explain how to configure deployments on Google Cloud.The course is divided into two parts: AlloyDB Administration Essentials (architecture and configuration) and AlloyDB Optimization Essentials (performance tuning).
Not sure which course fits?
Book a 30-minute call. We'll look at where your team is, what they need to be able to do, and put together either a specific course date or a tailored learning plan.