Temporal Replication And Queue Health Changes

Temporal is a durable workflow orchestration platform, so changes to replication cursors, task routing, and connection ownership belong in the data engineering review queue. The 28 recent commits touch 173 files, with 6,879 insertions and 628 deletions. The useful theme is operational control: safer replication rollout, bounded streams, cleaner resource shutdown, and better backlog evidence.

The newest change extracts replication reader state handling from the stream sender into replicationReaderGroup. The new reader group file owns reader ID calculation, catchup watermarks, and construction of persisted QueueReaderState. It also validates whether high and low priority state matches the current tiered processing mode.

This is mostly an internal refactor, but it has a useful rollout seam. The new global setting history.EnableReplicationReaderGroup defaults to false in dynamic configuration. Enabling it selects the refactored path, and changing it restarts replication streams. The excerpt states that wire protocol and persistence formats do not change, so the expected blast radius is stream restart rather than data migration.

The setting also appears in the central history service configuration, one of the most frequently changed files in the window. A related commit adds a client side maximum lifetime for replication streams. The review does not provide the lifetime value, but the intent is clear: streams can be recycled deliberately instead of surviving until an unrelated failure closes them.

Temporal also tightened ownership around internal clients. One change caches local and remote frontend gRPC connections in RPCFactory. The follow up releases connections and SDK clients owned by factories. Another commit stops the SDK workers and clients started by the worker service.

Those changes belong together. Connection reuse reduces repeated setup, while explicit close paths prevent cached resources from outliving the service that created them. The touched RPC implementation and SDK factory make the ownership boundary visible in code. The prominence of leak checks in the changed file list is consistent with that lifecycle focus.

There is smaller cleanup in the same category. The version checker now closes its response body on every path. None of this changes workflow semantics, but shutdown tests, rolling restarts, and repeated integration runs should see fewer resources left behind.

Matching received backlog aware task and poll load balancing. The partition scaler now avoids loading unloaded partitions during backlog checks, while still checking every versioned queue that could contain work. That distinction matters because an observation path should not activate idle partitions merely to ask whether work exists.

Operators also get an immediate queue backlog age metric, represented in the metric definitions. Queue depth alone can hide old work behind a modest count. Age gives a direct view of how long the oldest immediate task has waited. The wider scan across versioned queues may cost more work, although the activity summary cites no benchmark.

The fair task reader also moves outstanding tasks to a copy on write B tree. Treat that as an implementation change until measurements appear. The commit list does not provide latency, memory, or allocation numbers.

Several correctness fixes reduce useless queue activity. The batcher now stops retrying terminal state errors in batcher activities. A versioned transition cleanup that finds nothing to delete no longer sends the task to the DLQ. Both changes keep expected terminal or empty outcomes from looking like recoverable failures. For operators, the common symptom would be queue growth without useful forward progress. Removing those cases improves the signal quality of DLQ volume and retry counts, even if raw throughput stays unchanged.

Generation fencing was also added for Backfiller tasks. Separately, migrated schedule starts at the same time now receive unique identities. These are narrow fixes, but they protect the control plane from stale work and identity collisions that can otherwise surface as retry noise.

  • These commits are on main, not a named release in the supplied review. Confirm the target server release before changing production configuration.

  • Stage history.EnableReplicationReaderGroup carefully. It is false by default, and changing it restarts replication streams. Watch replication lag and reconnect counts during rollout.

  • Add immediate backlog age beside depth on matching dashboards. Compare it with stream churn after the client lifetime change so queue delay is not mistaken for normal connection recycling.