Toil is a Python workflow engine that runs CWL, WDL, and native Python pipelines on laptops, clouds, and HPC clusters. On 1 September 2026 the DataBiosphere/toil tree gained --doubleTime, an opt in flag that retries a timed out job with twice the walltime. The same change teaches Slurm to report a timeout as TIMELIMIT instead of a generic kill, which is the signal the retry path actually needs.
The doubleTime flag mirrors doubleMem for wall clock limits ¶
HPC operators already know --doubleMem. After a MEMLIMIT death it doubles memory and spends one retry. The new flag does the same for walltime.
The option is registered in src/toil/options/common.py. It is a boolean, default false, parsed with strtobool. Pass --doubleTime True. Help text says the batch system must kill the job for exceeding its walltime, Toil doubles that requirement, and the remaining retry count drops by one.
Retry logic lives in JobDescription.setupJobAfterFailure. On BatchJobExitReason.TIMELIMIT with the flag set, walltime is multiplied by two. A walltime of 0 means unlimited and stays unlimited. A plain FAILED exit does not bump the clock.
if exit_reason == BatchJobExitReason.TIMELIMIT and self._config.doubleTime:
if self.walltime == 0:
logger.warning(
"The walltime of the failed job %s is already unlimited",
self,
)
else:
self.walltime = self.walltime * 2
This is not a free extra attempt. Each timeout still consumes a retry. A workflow that needs one doubling must set --retryCount to at least 1.
Winners are jobs whose declared walltime is a bit short for the long tail. Losers are queues that charge by reservation: a doubled --time can bounce the retry onto a longer partition. CLI help for --doubleMem still says LSF only. --doubleTime help names no scheduler, but only Slurm emits TIMELIMIT in this change.
Slurm now reports TIMELIMIT instead of a generic kill ¶
Doubling only happens if the leader can see a timeout. Before this change, Slurm TIMEOUT mapped to BatchJobExitReason.KILLED. BatchJobExitReason now has TIMELIMIT = 10, and src/toil/batchSystems/slurm.py maps TIMEOUT to that reason.
That mapping is not enough on its own. Toil asks Slurm to signal the worker 30 seconds before the limit so cleanup can run. A worker that exits on that warning is FAILED to Slurm, not TIMEOUT. The old --signal=B:INT@30 made it worse: SIGINT became a Python KeyboardInterrupt, which third party code already treats as a normal interrupt.
The worker path in the same commit now installs a handler for SIGUSR2 and exits with code 77 (WALLTIME_EXIT_CODE). Slurm is asked for --signal=B:USR2@30. If sacct shows exit code 77, the Slurm backend still sets TIMELIMIT. Those constants now live in src/toil/worker.py next to NO_JOB_STORE_EXIT_CODE (76).
Tradeoff: code that used the old SIGINT warning for container cleanup no longer gets SIGINT from Toil’s sbatch line. It gets SIGUSR2, which the worker turns into KeyboardInterrupt inside the handler. Stray Ctrl+C on a login node is no longer mistaken for a scheduler timeout. Anything that inspected KILLED for Slurm timeouts now needs to look at TIMELIMIT instead.
Docs prefer defaultWalltime over slurmTime ¶
The HPC page docs/running/hpcEnvironments.rst stopped telling people to pass --slurmTime 4:00:00. Examples now use --defaultWalltime in seconds: 14400 for four hours, 600 in the WDL tutorial. --slurmTime still exists in the Slurm backend (slurm_time or walltime), but the documented path is the generic walltime field on the job.
That page is also the first public recipe for --doubleTime=True: set --defaultWalltime for the typical case, then let timeouts promote the long jobs onto a longer partition. That only works if partition selection already keys off walltime, which Toil’s Slurm backend does.
docs/running/cliOptions.rst also clarifies that --maxJobDuration is a kill issued by Toil for the whole workflow, not the per job walltime sent to the scheduler. Mixing those two knobs remains a footgun.
The cluster test in contrib/slurm-test/slurm_test.sh dropped --slurmTime 2:00 in favor of --defaultWalltime 120. The new toil_doubletime_workflow.py sleeps 120 seconds with walltime=100, which is meant to miss the first attempt and pass after the doubling. The test greps the log for doubled the walltime. First attempt: walltime 100, warning near second 70, sleep of 120 dies. Retry: walltime 200, warning near second 170, sleep of 120 finishes. Toil still does not pad the time it asks the batch system for.
Tests catch both timeout paths ¶
src/toil/test/batchSystems/test_slurm.py now has two unit cases. Job 754725 is a true Slurm TIMEOUT and must return TIMELIMIT. Job 789457 is FAILED with exit code 77 and must return the same reason. The bulk status test that used to expect KILLED for timeout now expects TIMELIMIT. Job description tests cover the doubling itself: 600 seconds becomes 1200 then 2400, stays 600 when the flag is off, and stays 0 when walltime is unlimited.
The CWL helper waste_cpu_memory.cwl switched from time.monotonic() to time.process_time() and busy loops instead of sleeping. On a loaded runner, wall clock can pass without burning the CPU seconds the test claimed to waste. The Docker Slurm suite now runs the basic, sort, and doubleTime workflows in parallel, each with its own job store.
What to watch ¶
--doubleTime is off unless you pass it. It only fires on TIMELIMIT. Today that reason is wired for Slurm TIMEOUT and for worker exit 77. LSF, Kubernetes, HTCondor, and Torque do not gain a timeout reason in this commit.
Plan retries. One doubling costs one retry. A job that times out twice needs two retries and will request 4x the original walltime on the third attempt.
Treat the 30 second warning as real. A job with walltime=100 can be told to stop near second 70. If the work cannot finish in requested time minus that warning window, either raise the requirement or wait for the padding the source still marks as TODO.