Data Integration with Cloud Data Fusion
This 2-day course introduces learners to Google Cloud’s data integration capability using Cloud Data Fusion. In this course, we discuss challenges with data integration and the need for a data integration platform (middleware). We then discuss how Cloud Data Fusion can help to effectively integrate data from a variety of sources and formats and generate insights. We take a look at Cloud Data Fusion’s main components and how they work, how to process batch data and real time streaming data with visual pipeline design, rich tracking of metadata and data lineage, and how to deploy data pipelines on various execution engines.
- Duration: 2 days
- Languages: English, German
- Level: Intermediate
- Group size: Up to 16
What you'll learn
- Identify the need for data integration.
- Understand the capabilities Cloud Data Fusion provides as a data integration platform.
- Identify use cases for possible implementation with Cloud Data Fusion.
- List the core components of Cloud Data Fusion.
- Design and execute batch and real-time data processing pipelines.
- Work with Wrangler to build data transformations.
- Use connectors to integrate data from various sources and formats.
- Configure execution environment; Monitor and Troubleshoot pipeline execution.
- Understand the relationship between metadata and data lineage.
Prerequisites
- Participants are encouraged to have completed "Introduction to Data Engineering".
Course outline
Introduction
- Course Introduction
Introduction to data integration and Cloud Data Fusion
- Data integration: what, why, challenges
- Data integration tools used in industry
- User personas
- Introduction to Cloud Data Fusion
- Data integration critical capabilities
- Cloud Data Fusion UI components
Building pipelines
- Cloud Data Fusion architecture
- Core concepts
- Data pipelines and directed acyclic graphs (DAG)
- Pipeline Lifecycle
- Designing pipelines in Pipeline Studio
Designing complex pipelines
- Branching, Merging and Joining
- Actions and Notifications
- Error handling and Macros
- Pipeline Configurations, Scheduling, Import and Export
Pipeline execution environment
- Schedules and triggers
- Execution environment: Compute profile and provisioners
- Monitoring pipelines
Building Transformations and Preparing Data with Wrangler
- Wrangler Directives
- User-defined directives
Connectors and streaming pipelines
- Understand the data integration architecture
- List various connectors
- Use the Cloud Data Loss Prevention (DLP) API
- Understand the reference architecture of streaming pipelines
- Build and execute a streaming pipeline
Metadata and data lineage
- Metadata
- Data lineage
Summary
- Course Summary
Related courses
AlloyDB Essentials
This course explores the benefits of AlloyDB, especially compared to PostgreSQL on Cloud SQL. It will walk you through AlloyDB’s unique architecture, and explain how to configure deployments on Google Cloud.The course is divided into two parts: AlloyDB Administration Essentials (architecture and configuration) and AlloyDB Optimization Essentials (performance tuning).
BigQuery for Data Analysts
This course is designed for data analysts who want to learn about using BigQuery for their data analysis needs. Through a combination of videos, labs, and demos, we cover various topics that discuss how to ingest, transform, and query your data in BigQuery to derive insights that can help in business decision-making.
Data Engineering on Google Cloud
Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hands-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data.
Not sure which course fits?
Book a 30-minute call. We'll look at where your team is, what they need to be able to do, and put together either a specific course date or a tailored learning plan.