Airflow Workflow Automation

This guide offers an introduction to some of Airflow's fundamental concepts (DAGs, tasks etc.) as well as two example of multi-node workflows on UCloud.

Contents

• Airflow Basics
  • DAGs
  • Tasks
  • Airflow Components
  • Creating a Workflow
  • Execution and Monitoring
• Airflow Connections: Integration with Spark
  • Setup
  • Spark Operators
  • Start Spark with BashOperator
• Airflow's Celery Executor: Distributed Task Execution
  • Celery and CeleryExecutor
  • General Setup
  • Main Node Configuration
  • Worker Node Configuration
  • Example Workflow