Scheduling, Logical Dates, and Backfill
Work out why a daily DAG for the 14th runs on the 15th, then use that same property to backfill a month of history correctly.
Airflow’s scheduling model confuses almost everyone once. The confusion has a single source, and once it clicks, backfill becomes free.
Intervals, not moments
A scheduled DAG run covers a data interval. The run is identified by the interval’s start, and it executes after the interval’s end.
from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator
with DAG(
dag_id="interval_demo",
start_date=datetime(2026, 2, 1),
schedule="@daily",
catchup=False,
) as dag:
show = BashOperator(
task_id="show",
bash_command=(
"echo 'logical date : {{ ds }}'; "
"echo 'interval start: {{ data_interval_start }}'; "
"echo 'interval end : {{ data_interval_end }}'; "
"echo 'actual now : {{ macros.datetime.utcnow() }}'"
),
)airflow tasks test interval_demo show 2026-02-14INFO - logical date : 2026-02-14
INFO - interval start: 2026-02-14T00:00:00+00:00
INFO - interval end : 2026-02-15T00:00:00+00:00
INFO - actual now : 2026-02-17 11:42:03.914722The run is for the 14th, covering midnight to midnight, and it is running on the 17th
because that is when the test was invoked. Real scheduled execution begins just after
data_interval_end — the 15th at 00:00.
This is deliberate. A job summarising the 14th cannot run at 00:00 on the 14th, because none of that day’s data exists yet.
Why this makes backfill work
Because a task derives everything from its logical date, running it for an old date produces exactly what it would have produced then:
extract = BashOperator(
task_id="extract",
bash_command="echo 'SELECT * FROM orders WHERE date = {{ ds }}' > /tmp/query_{{ ds_nodash }}.sql; cat /tmp/query_{{ ds_nodash }}.sql",
)airflow tasks test interval_demo extract 2026-01-05INFO - SELECT * FROM orders WHERE date = 2026-01-05The same task, run today, generated January’s query. Nothing in it reads the wall clock, so it is idempotent — re-running it is safe and produces the same result.
Contrast with a task that uses the clock:
bad = BashOperator(
task_id="bad",
bash_command="echo 'SELECT * FROM orders WHERE date = ' $(date +%F)",
)airflow tasks test interval_demo bad 2026-01-05INFO - SELECT * FROM orders WHERE date = 2026-02-17Asked for January 5th, queried February 17th. Any task that calls now() cannot be
backfilled or safely retried. Use the templated date, always.
Common date macros
airflow tasks test interval_demo macros 2026-02-14INFO - ds 2026-02-14
INFO - ds_nodash 20260214
INFO - data_interval_start 2026-02-14T00:00:00+00:00
INFO - data_interval_end 2026-02-15T00:00:00+00:00
INFO - prev_ds 2026-02-13
INFO - next_ds 2026-02-15
INFO - ts 2026-02-14T00:00:00+00:00
INFO - dag_run.run_id scheduled__2026-02-14T00:00:00+00:00{{ ds }} and {{ ds_nodash }} cover most needs — partition paths, query predicates, output
filenames.
Schedule syntax
schedule="@daily" # midnight
schedule="@hourly" # on the hour
schedule="0 6 * * *" # 06:00 daily
schedule="0 6 * * 1-5" # 06:00 weekdays
schedule="0 */4 * * *" # every 4 hours
schedule=timedelta(hours=6) # every 6h from start_date
schedule=None # manual trigger onlyA timedelta schedule differs subtly from cron: it counts from the previous run rather than
aligning to clock boundaries, so a 6-hour timedelta starting at 02:15 fires at 08:15, 14:15,
and so on.
Verify your interpretation before deploying:
airflow dags next-execution interval_demo2026-02-18T00:00:00+00:00Catchup
with DAG(
dag_id="catchup_on",
start_date=datetime(2026, 2, 1),
schedule="@daily",
catchup=True,
) as dag:
...Unpause this on the 17th:
airflow dags unpause catchup_on && sleep 10 && airflow dags list-runs -d catchup_ondag_id | run_id | state | execution_date
===========+=====================================+=========+===========================
catchup_on | scheduled__2026-02-16T00:00:00+00:00 | running | 2026-02-16T00:00:00+00:00
catchup_on | scheduled__2026-02-15T00:00:00+00:00 | success | 2026-02-15T00:00:00+00:00
catchup_on | scheduled__2026-02-14T00:00:00+00:00 | success | 2026-02-14T00:00:00+00:00
...
catchup_on | scheduled__2026-02-01T00:00:00+00:00 | success | 2026-02-01T00:00:00+00:00Sixteen runs queued instantly. Useful when you want history; alarming when the start_date is a year back and each run hits a rate-limited API.
Limit the damage with concurrency controls:
with DAG(
dag_id="catchup_safe",
start_date=datetime(2026, 2, 1),
schedule="@daily",
catchup=True,
max_active_runs=1, # one interval at a time, in order
) as dag:
...catchup_safe | scheduled__2026-02-02T00:00:00+00:00 | queued | ...
catchup_safe | scheduled__2026-02-01T00:00:00+00:00 | running | ...max_active_runs=1 is essential when runs share a resource — a table they append to, an API
quota, a warehouse connection pool.
Deliberate backfill
airflow dags backfill interval_demo \
--start-date 2026-01-01 --end-date 2026-01-07[2026-02-17 12:05:11] {backfill_job_runner.py:400} INFO - [backfill progress] |
finished run 0 of 7 | tasks waiting: 7 | succeeded: 0 | running: 1 | failed: 0
...
[2026-02-17 12:06:42] {backfill_job_runner.py:400} INFO - [backfill progress] |
finished run 7 of 7 | tasks waiting: 0 | succeeded: 7 | running: 0 | failed: 0
[2026-02-17 12:06:42] {backfill_job_runner.py:1050} INFO - Backfill done. Exiting.Seven runs, one per interval, all with their correct logical dates. Useful flags:
# Re-run only tasks that failed
airflow dags backfill my_dag --start-date 2026-01-01 --end-date 2026-01-07 --rerun-failed-tasks
# Overwrite runs that already succeeded
airflow dags backfill my_dag --start-date 2026-01-01 --end-date 2026-01-07 --reset-dagruns
# Only some tasks
airflow dags backfill my_dag --start-date 2026-01-01 --end-date 2026-01-07 --task-regex "extract.*"The moving start_date trap
with DAG(
dag_id="broken",
start_date=datetime.now() - timedelta(days=1), # never do this
schedule="@daily",
) as dag:
...Every time the scheduler parses the file, start_date moves forward. The first interval’s end
is therefore always in the future, so the interval is never complete and the DAG never runs.
It sits in the UI looking healthy, doing nothing.
Use a fixed date:
start_date=datetime(2026, 2, 1)Practice
1. What logical date does an hourly DAG use for the run covering 14:00-15:00?
ds: 2026-02-14
ts: 2026-02-14T14:00:00+00:00
data_interval_start: 2026-02-14T14:00:00+00:00
data_interval_end: 2026-02-14T15:00:00+00:00The interval start, 14:00 — and it executes just after 15:00. Note {{ ds }} is only the date
for an hourly DAG, so use {{ ts_nodash }} when you need hour granularity in a filename.
2. Set catchup=True with max_active_runs=1 and unpause. In what order do runs execute?
Oldest first, one at a time. This matters when runs are not independent — appending to a
partitioned table, or computing a running balance. Without max_active_runs, sixteen runs
execute concurrently in unpredictable order.
3. Backfill a range where some runs already succeeded, without --reset-dagruns.
INFO - Backfill done. Exiting.It completes almost instantly and skips the existing runs — Airflow will not re-run a
successful DagRun unless you clear it. Add --reset-dagruns to force re-execution, which is
what you want after fixing a bug in the task logic.
4. Use {{ ds }} and $(date +%F) in the same task, backfilled to a past date.
templated: 2026-01-05
shell: 2026-02-17Two different answers in one task. When a backfill produces today’s data instead of the target
date’s, a shell date call is nearly always the cause. Grep your DAGs for date +, now(),
and today() — each is a backfill bug waiting to happen.
Next: waiting for something to exist before proceeding.