Apache Airflow Main: Scheduler Isolation, a dag_run Index and airflowctl Fixes

Apache Airflow is the workflow orchestrator many data teams run in production. The main branch logged 234 commits in the last week. Most are small, but a few touch scheduler behavior, the metadata schema and operator tooling in ways worth reading before the next minor release.

The most operator visible fix is skipping scheduling for tasks whose Dag cannot be deserialized. Before this change, an exception from get_dag_for_run inside the scheduling loop could stop progress for unrelated Dags. That is a bad blast radius for one bad serialized row.

The patch in scheduler_job_runner.py adds a starved_dag_runs set. When loading a Dag fails, the scheduler logs the exception, records (dag_id, run_id) and returns False for that task instance. Later task instances from the same run are skipped without another load attempt.

One detail matters. DBAPIError is raised again on purpose. A database outage should not look like a pile of broken Dags, and the commit message says DB errors propagate so the scheduler does not silently skip task instances.

try:
    serialized_dag = self.scheduler_dag_bag.get_dag_for_run(
        dag_run=task_instance.dag_run, session=session
    )
except DBAPIError:
    raise
except Exception:
    starved_dag_runs.add((dag_id, run_id))
    return False

The tradeoff: a skipped run now fails quietly except for a log line. Alert on Failed to load Dag in scheduler logs, or the broken Dag will just sit there.

The schema change is an index on dag_run (dag_id, run_after). It ships as a new Alembic migration tagged airflow_version = "3.5.0", and it is declared on the DagRun model next to the existing single column indexes on dag_id and run_after.

The commit does not cite benchmark numbers, so treat the benefit as unmeasured. The shape is clear though. Queries that filter by Dag and order or range scan on run_after can use one index instead of picking between two. The winners are large metadata databases with many runs per Dag. The cost is extra write amplification on dag_run inserts and a longer migration on big tables.

Before upgrading, check how large your dag_run table is. On PostgreSQL you may want to plan the index build in a maintenance window. The migration uses batch_alter_table with a plain create_index, not a concurrent build.

Several commits extend observability. KubernetesPodOperator now emits resumable_job metrics, and a follow up adds dag_id and task_id tags. Without those tags the metric could not tell you which pipeline was resuming pods.

A separate change adds metrics for task and asset state store access. It adds 72 lines to metrics_template.yaml and instruments context.py in the task SDK. If you run StatsD or OpenTelemetry with an allow list, new metric names will not appear until you add them.

Tag cardinality is the thing to check. Per Dag and per task tags are fine on a small deployment and expensive in a large one.

airflowctl took a batch of bug fixes this week:

The --env bug is the one to audit. If you scripted against several environments with a version that ignored the flag, review what those scripts actually hit.

Two changes concern language SDK runtimes. One runs the runtime in its own process group. Previously, stopping a task signalled only the runtime pid, so child processes were reparented to PID 1 and kept running. The other reports launch failures in the task log, so a failed start is visible where you already look.

On the provider side, two BigQuery fixes cover durable recovery for regional jobs and cancellation with inferred job locations. Both matter if your datasets live outside the US multi region.

Much of the file churn is CI work. The top changed files are fork CI documentation and workflows such as ci-amd.yml, driven by a rule that waits for a maintainer when a fork PR changes CI defining files. That affects contributors, not runtime users.

  • The new migration for 3.5.0. Measure dag_run size and rehearse the upgrade on a copy.
  • Scheduler logs for Failed to load Dag. The fix hides the symptom, so you need your own alert.
  • The deprecation of airflow.utils.email in favor of the SMTP provider, landed in this commit. Search your Dags for imports now.