Workflow Orchestration with Cloud Composer
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. Composer enables you to create, schedule, monitor, and manage workflow pipelines that span across clouds and on-premises data centers. In this course, you will learn about Apache Airflow and its implementation via Cloud Composer. You will learn how to provision Composer instances, create and manage Airflow DAGs on Composer, and perform tasks such as testing, debugging, and monitoring of Airflow DAGs.
- Duration: 1 day
- Languages: English, German
- Level: Intermediate
- Group size: Up to 16
What you'll learn
- Explore Apache Airflow and Cloud Composer as workflow orchestration solutions.
- Create and manage Airflow DAGs following best practices.
- Test and debug Airflow DAGS.
- Monitor and observe Airflow DAGs on Cloud Composer.
Prerequisites
- Completion of "Building Batch Data Pipelines on Google Cloud" or equivalent knowledge of data analytics and engineering on Google Cloud.
Course outline
Introduction to Cloud Composer
- Data Engineer's need for Workflow Orchestration
- Introduction to Apache Airflow
- Cloud Composer
- Environment Setup
- Using the Composer and Airflow
Creating and managing DAGS
- DAG structure and best practices
- Common operators
- Dependencies, trigger rules, and flow control
- Integration of Airflow and Google Cloud Services
Advanced Airflow techniques and best practices
- Advanced Airflow features
- Debugging DAGS
- Performance and scalability
- Security and Access Control
- Observability and monitoring
Related courses
AlloyDB Essentials
This course explores the benefits of AlloyDB, especially compared to PostgreSQL on Cloud SQL. It will walk you through AlloyDB’s unique architecture, and explain how to configure deployments on Google Cloud.The course is divided into two parts: AlloyDB Administration Essentials (architecture and configuration) and AlloyDB Optimization Essentials (performance tuning).
BigQuery for Data Analysts
This course is designed for data analysts who want to learn about using BigQuery for their data analysis needs. Through a combination of videos, labs, and demos, we cover various topics that discuss how to ingest, transform, and query your data in BigQuery to derive insights that can help in business decision-making.
Data Engineering on Google Cloud
Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hands-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning. This course covers structured, unstructured, and streaming data.
Not sure which course fits?
Book a 30-minute call. We'll look at where your team is, what they need to be able to do, and put together either a specific course date or a tailored learning plan.