Apache Airflow 3.3.0 - Asset State And Partition Controls

Apache Airflow 3.3.0 was published on July 6, 2026, with asset partitioning as the change most likely to alter pipeline design. It expands partition mapping, adds runtime partition assignment, and puts task and asset state behind API managed stores instead of ad hoc task code. The GitHub release is not marked as a prerelease.

The full release notes and downloads are on the GitHub release page.

Airflow 3.3.0 extends the asset partitioning model that arrived in 3.2.0. A single upstream asset event can now map into partitioned downstream Dag runs through RollupMapper, FanOutMapper, and FixedKeyMapper with SegmentWindow.

Those mappers cover many to one rollups, one to many fan out, and categorical rollup patterns. For event driven datasets, that is the useful part. Producer cadence and consumer partitions often do not match cleanly, especially when daily tables feed weekly or monthly jobs.

The release also adds time windows for day, week, month, quarter, and year, plus wait_policy controls. WaitForAll waits for all expected upstream partitions. MinimumCount(n) lets a downstream run begin after enough upstream events arrive.

There is an operator guardrail here. Total fan out per upstream event is bounded by [scheduler] partition_mapper_max_downstream_keys, with per mapper control. That setting is worth reviewing before large partition graphs hit the scheduler. PartitionedAtRuntime also lets a Dag assign partition keys when the run starts, instead of deriving every key from an upstream event.

The new task and asset state store is the other large data pipeline change in this release. Tasks can persist arbitrary key value state through task_state_store, and assets can do the same through asset_state_store.

By default, state lives in the Airflow metadata database. Operators can point workers at a custom backend with [workers] state_store_backend. The release notes also call out retention per key, periodic garbage collection, optional clear_on_success, and management through the Core API and Execution API.

This is cleaner than hiding durable state in XComs, temp tables, or external files that only some tasks know how to read. It also means retention and cleanup become platform concerns, not just Dag author concerns.

Pluggable retry policies add a related control point. A task is no longer limited to a fixed retries count. A custom policy can retry only on selected exceptions or choose timing from custom logic. The tradeoff is that retry behavior is now code that operators need to test like other scheduling behavior.

Airflow 3.3.0 adds a Coordinator layer for task implementations in Java, JVM languages, and Go while the Dag and scheduler remain Python. A Dag declares the task with @task.stub(queue=...), then the worker routes it to JavaCoordinator or ExecutableCoordinator.

The coordinator proxies Variables, Connections, and XComs through the Execution API. That is the key platform detail. It means a non Python task can still use Airflow managed runtime context instead of carrying a separate integration path.

The release notes mark the Coordinator layer and the Java and Go SDKs as experimental. Treat this as a path for isolated tasks or early SDK evaluation. It is not a signal to rewrite working Python task fleets.

Dag bundle version selection now has an explicit control. rerun_with_latest_version decides whether a cleared, rerun, or backfilled Dag run uses the latest bundle version or the original bundle version. The precedence is request parameter or CLI flag, Dag setting, [core] rerun_with_latest_version, then default behavior.

Remote logging resolution moved into airflow_shared.logging.factory. Custom configs that export REMOTE_TASK_LOG or DEFAULT_REMOTE_CONN_ID still take precedence, then ProvidersManager scheme dispatch handles [logging] remote_base_log_folder. configure_logging now resolves the remote handler lazily, and direct calls to airflow.logging_config.load_logging_config() are deprecated.

Metrics also changed in ways that affect dashboards. OpenTelemetry timer and timing metrics are now Histograms instead of Gauges, preserving count, sum, and bucket distribution. The new dag_processing.last_run.seconds_ago metric is tagged with file_path, bundle_name, and file_name. The older dag_processing.last_run.seconds_ago.{dag_file} metric is still emitted by default and can be disabled with [metrics] legacy_names_on.

  • Check REST API clients that filter provider example Dags by bundle_name. The old dags-folder filter will miss provider example bundles now named apache-airflow-providers-<distribution>-example-dags or <distribution>-example-dags.
  • Replace direct use of airflow.logging_config.load_logging_config() with the new helper path named in the release notes. Provider remote log classes should expose from_config() and read airflow.providers.common.compat.sdk.conf.
  • Review metrics dashboards before disabling legacy names. The new dag_processing.last_run.seconds_ago series relies on file_path, bundle_name, and file_name tags.