Prefect Tightens Retry Leases And Event Persistence


Prefect is a Python workflow orchestrator, and its latest activity centers on execution correctness rather than a large new feature. Across 30 commits, the useful operator story is tighter retry accounting, less repeat work in server paths, and more reliable worker setup.

The most important fix is in deployment concurrency. During a retry in the same process, the engine continues to renew the concurrency lease while the flow waits. The previous orchestration path could release that lease when a failed transition was rejected into AwaitingRetry. The next renewal would then fail and the flow could be cancelled during its retry delay.

The retry lease correction changes two related decisions in core_policy.py. It copies the deployment lease identifier from the initial state into the retry state. It also avoids releasing the slot when the transition into AwaitingRetry is the rejected result of RetryFailedFlows.

The failure sequence is subtle. A running flow proposes FAILED, but the retry rule rejects that proposal and substitutes a scheduled retry state. The release policy then sees a state transition that resembles a departure, even though the same engine process remains responsible for renewal. Once the lease is revoked, the next renewal turns an ordinary task failure into a cancelled flow run. That symptom can send an operator toward worker health or infrastructure logs when the actual fault is orchestration state bookkeeping.

That exception is deliberately narrow. An AwaitingRetry state proposed directly, such as a reschedule after SIGTERM, still leaves the process and releases its slot. This distinction matters for deployments with a concurrency limit of one: an internal retry must keep its existing claim, while a run that exits must stop consuming capacity. The expanded test_core_policy.py coverage targets that state machine boundary.

Two server changes remove work from paths that can run for every event or message. The event persister change reduces CPU spent compiling event INSERT statements in database.py. The activity record does not include a benchmark, so there is no defensible percentage to attach to it. Still, avoiding repeated SQL compilation is the right target when event volume, rather than query complexity, dominates server load.

The Redis messaging change caches the messaging URL lookup. This is a smaller optimization, but it removes configuration resolution from a repeated path. Operators should expect lower incidental CPU work, not a change in Redis protocol behavior or delivery semantics.

These changes address separate costs. One concerns SQL statement construction before persistence. The other concerns discovery of a stable connection value before messaging work. Neither commit claims a database schema change or a new configuration contract, which keeps adoption risk lower than the size of the broader 158 file activity set might suggest.

There is also a correctness guard around event size. A run context resource limit fix keeps related resources within the maximum an event accepts. The change touches context.py. For flows with rich run context, this is preferable to building an event that later exceeds its own contract. The tradeoff is that consumers must not assume every related resource will always fit into one event.

Process workers can receive source through pull steps. Project detection performed before those steps has incomplete evidence because files such as pyproject.toml and uv.lock may not exist yet. The ProcessWorker correction moves automatic uv handling to the point after pull steps, when the fetched project is actually present.

The affected process.py and its tests are among the most frequently touched files in this activity window. A related documentation clarification now states that project detection runs after pull steps. For deployments that materialize code at run time, this makes environment selection follow the deployed files instead of the worker’s initial directory.

Ordering is the behavior change here. A worker that starts in a generic directory cannot infer the final project environment before source retrieval. Deferring detection lets the same deployment pattern work whether a pull step clones a repository, copies an artifact, or prepares another local source tree. Teams with custom pull steps should verify that the expected project files land in the directory inspected by the worker.

Quality gates also received a useful repair. The Pyright workflow fix addresses a CI check that silently analyzed zero files in the static analysis workflow. A subsequent cleanup fixed four reported errors. This is internal maintenance, but a green check that examines no code is worse than a noisy check because it hides the missing coverage.

Operators using deployment concurrency limits should watch retrying flows for unexpected cancellation and confirm that slots return after runs actually leave the process. The new branch is specific enough that both lease retention and final release deserve metrics.

High volume Prefect servers should compare event persister CPU before and after adopting these changes, since no result was published with the commit. Also watch event consumers that expect a complete related resource set. The new cap favors valid event envelopes over unlimited context, which is the safer default but may expose assumptions in downstream automation.