OpenTofu progress this week centers on the maturation of the new experimental engine runtime. Engineering efforts have pivoted toward hardening the variable evaluation path and refining how resource state is upgraded during plan operations.
The development of the 2024 engine runtime is a major architectural shift. While the existing engine has served the community well, the new implementation aims to provide better performance and more robust handling of complex graph evaluations. Recent activity confirms the team is tackling nuanced features like data sensitivity and the lifecycle of ephemeral values. This work is essential for reaching parity with the stable runtime.
Handling Sensitive and Ephemeral Variables ¶
One of the most critical updates is the explicit support for sensitive and ephemeral flags on variable declarations. Martin Atkins landed a change to handle sensitive and ephemeral on variable declarations which ensures data marking is respected early in the evaluation cycle.
This change is vital for security posture. In infrastructure as code, sensitive variables prevent secrets from leaking into logs. By implementing this at the variable declaration level, the project ensures security guarantees remain intact. The commit touches internal/lang/eval/internal/tofu2024/compile_input_variables.go, indicating flags are now integrated into the compilation phase.
Additionally, the introduction of ephemeral support suggests better management of short lived data that should not be persisted. This aligns with modern cloud native workflows where temporary credentials pass through the graph without leaving a footprint. The ability to allow arbitrary transforms of final input variable value further enhances flexibility before reaching resource providers.
Advancing State Migration Logic ¶
State management is the heart of OpenTofu, and migrating state between different schema versions is a high stakes operation. Larry Bordowitz recently submitted a first draft of upgradestate in the new engine.
The logic for state upgrading is found within internal/engine/planning/plan_managed.go. A key fix involved how provider instance addresses are tracked. The project found the provider instance address should be updated for refreshed state but not necessarily during the initial upgrade phase. This distinction is vital for maintaining a consistent view of provider responsibility.
The code now explicitly separates these concerns. When planning a managed resource instance, the engine decodes previous state and applies upgrades. The fix for updating provider in state ensures refreshed state reflects the current provider instance while the upgraded state remains a faithful representation of the transformed input. This prevents edge cases where resources might be incorrectly associated with the wrong provider instance.
Resolving Experimental Runtime Bugs ¶
The new engine has faced its share of bugs. This week saw the resolution of an issue where for each logic was not behaving correctly. Martin Atkins confirmed the ExperimentalBugForEach is now fixed, a significant milestone for the 2024 runtime.
The for each construct is widely used to manage collections of resources. Instability here would block wider adoption. Along with this fix, the team addressed a case where ExperimentalBugVariableSensitive was resolved. These fixes demonstrate a systematic approach to clearing known issues.
To maintain quality, the project refined its testing strategy. A recent commit introduced an alternate approach to enabled or disabled experimental tests. This allows developers to toggle tests depending on specific experimental flags, reducing noise in the integration pipeline.
Tooling and Runtime Updates ¶
Beyond core engine changes, OpenTofu is keeping its toolchain current. Andrei Ciobanu merged a change to upgrade the Go version to 1.26.4 as part of a govulncheck update. Staying on the latest Go runtime is standard practice to benefit from performance and security patches.
The Makefile also received updates to improve the developer experience. The test-experimental target was adjusted to focus on the internal/tofu directory. This speeds up the feedback loop for engineers working on the runtime by avoiding the need to run the entire test suite.
Furthermore, the project is identifying obsolete features that will not be carried forward. For example, flatattrs tests are now skipped because flat mapped attributes are considered a legacy relic. This pruning ensures the 2024 engine focusing on a clean and maintainable future.
What to watch ¶
Operators should track the progress of the TOFU_X_EXPERIMENTAL_RUNTIME flag. As more core bugs are squashed, the path toward making this the default engine becomes clearer.
First, watch for refinements in the tofu2024 package. Recent work on input variables and state upgrades suggests a move toward language implementation specifics. We expect more commits targeting complex expressions and function calls soon.
Second, track the deprecation of legacy state formats. The decision to skip flat mapped attribute tests indicates the new engine will enforce a modern state schema. Users with old state files may need a migration using the stable engine first.
Finally, keep an eye on performance benchmarks. While the current focus is on correctness, the primary driver for the new engine is efficiency. Once the bug count drops, we anticipate data regarding speed and memory usage compared to the current implementation.