Toil 9.6.0 - Slurm Walltime Controls, a Single Run Directory and Cache Fixes


Toil 9.6.0 was published on October 20, 2026 by DataBiosphere. It is a stable release, not a prerelease. The headline change is walltime control for Slurm jobs through --defaultWalltime, alongside a new --runDir option and a set of fixes to the file cache.

The full release notes and downloads are on the GitHub release page.

Slurm jobs now take their runtime limit from --defaultWalltime (#3037, #5502). The documentation now recommends it over --slurmTime (#5576). If you have scripts that pass --slurmTime, read the docs before you assume they still do what you expect.

CWL workflows get a related change. Time limits declared with ToolTimeLimit are now used in Slurm batch submissions. A workflow author can set a limit per tool, and the batch system will see it.

There is also a new --doubleTime=True flag (#5575). It doubles the walltime of jobs that fail. The trade is simple. Jobs killed by a too tight limit get another chance with more room. Jobs that fail for other reasons will also be retried with a larger reservation, so watch your queue accounting if you turn it on.

A Slurm parsing fix is worth reading in full (#5583). The leader no longer crashes when scontrol show job output contains a job with newlines in its fields. The notes add a warning. On the oldest Slurm version Toil supports, the output cannot be parsed reliably. A malicious user on the same cluster can build jobs that mislead Toil about the state of its own jobs. On a shared cluster, treat that as a trust boundary issue and not only a bug.

--runDir and the TOIL_RUN_DIR environment variable are new (#5516). Toil derives the job store, work dir, coordination dir and container image cache locations from the one directory. That removes four paths you had to keep consistent by hand.

Related behavior: --workDir and --coordinationDir are now created if they are missing. A typo in a path used to fail early. Now it may quietly create a directory in the wrong place, so check what you pass.

Most of the operational fixes touch the caching file store. All of them cite #5084 or nearby issues.

  • Toil should no longer crash on “missing” files in the cache on nodes where the cache was previously shut down (#5084).
  • The cache database is now accessed correctly when Toil reports missing files, and it logs more about cache database state when files appear to be gone.
  • Files uploaded with writeGlobalFile are no longer deleted when a job finishes with the caching file store (#5562).
  • --cleanWorkDir=always now cleans more aggressively. If anything of yours lives in a work dir, move it first.
  • Restarting a workflow from a different machine no longer crashes (#5544).
  • A lock versus unlink race in safe_lock() is fixed (#5537).
  • The history database now uses transactions properly for schema changes on Python older than 3.12 (#5579).

The writeGlobalFile item is the one to check against your own pipelines. If a job ever lost an output that it had just written, this is the likely cause.

On the CWL side, Toil now evaluates when conditionals on the leader before scheduling (#3990). Steps that evaluate false are never submitted to the real batch system. That saves queue time and scheduler noise on workflows with many conditional steps.

Other CWL items:

  • Support for the cwltool:Loop extension (#4227).
  • toil-cwl-runner --stats now records resource usage from Docker containers, so toil stats can read it (#5477).
  • Failing jobs now get their CWL error logs reported even when standard error is captured to a file (#2994).
  • Several loadContents: true fixes for step inputs, including File arrays (#5568).

For WDL, MiniWDL is upgraded to 1.15, which improves WDL 1.2 compatibility (#5377). The rest is test coverage, including exporting files after a restart (#5247) and a WDL quickstart example that now pins a version (#5534).

The Toil Docker container is now based on Ubuntu 26.04 (#5524). The appliance image again includes the rev binary needed on the host to pass CWL conformance tests (#5548). Mesos debs are now installed individually (#5586). The EC2 instance type table and node descriptions are updated (#5651).

The rest is mostly internal: type stubs for Kubernetes (#5650), CI workarounds for a multi arch container issue, a bump of actions/checkout, and test stabilization. None of it should change runtime behavior for you.

The “Breaking Changes” heading in the notes is empty, so nothing is declared as breaking. Still check these before rolling out:

  • Replace --slurmTime with --defaultWalltime where you can.
  • Review any use of --cleanWorkDir=always.
  • Decide if --runDir fits your layout before mixing it with explicit path flags.
  • Rebuild any custom image that inherits from the appliance, since the base moved to Ubuntu 26.04.