Airflow.
Workflow orchestration with Apache Airflow — DAGs, operators, sensors, XComs, and production scheduling patterns.
Beginner
Start here — no prior experience needed
Your First Airflow DAG
Install Airflow, write a three-task DAG, trigger it from the CLI, and read the scheduler's output to see what actually ran and in what order.
Passing Data Between Tasks: XComs and the TaskFlow API
Move a value from one task to the next with XComs, rewrite the same DAG with the TaskFlow API, and see why XComs are the wrong place for a DataFrame.
Scheduling, Logical Dates, and Backfill
Work out why a daily DAG for the 14th runs on the 15th, then use that same property to backfill a month of history correctly.
Intermediate
For developers with core concepts down
Dynamic Task Mapping in Airflow
Generate a variable number of parallel tasks at runtime with expand(), fan the results back in, and see why building tasks in a Python loop is not the same thing.
Sensors and Deferrable Operators
Wait for a file with a poke sensor, watch it hold a worker slot for an hour, then convert it to a deferrable operator that holds nothing.
Branching and Trigger Rules in Airflow
Pick a path at runtime with a branch operator, watch the skip propagate further than you expected, and use trigger rules to stop it.
Connections, Hooks, and Variables
Store credentials outside your DAG files, use hooks to talk to Postgres and S3, and see why a Variable read at parse time hammers your metadata database.
Advanced
Production-grade patterns for experienced engineers
Testing Airflow DAGs
Catch import errors and cycles with a validation test, unit-test task logic without a scheduler, and run a whole DAG in-process with dag.test().
Airflow in Production: Idempotency, Pools, and SLAs
Turn a DAG that works into one that survives — idempotent writes, pools that stop a backfill saturating a database, alerting, and the settings that keep the scheduler fast.