job submission: No such file or directory .../job

[oliver-sanders]

Another bug associated with triggering tasks whilst the workflow is paused :(

This bug leads to some really bizarre behaviours:

• Multiple job submission commands get logged into a single job.activity.log file.
• job files are missing from some submissions causing them to fail erroneously.

Reproducible Example

(credit Dave)

We must simulate job submission failure in order to reproduce, the easiest way to do this is to jigger the .bashrc file of a remote platform.

exit 1

Or for a remote platform on a shared filesystem:

[[ $HOSTNAME == myremote ]] && exit 1

Then run this workflow:

[scheduling]
    [[graph]]
        R1 = """
            remote
            operator
        """

[runtime]
    [[remote]]
        platform = myremote

    [[operator]]
        script = """
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote/02:submit-failed"
            cylc pause "${CYLC_WORKFLOW_ID}"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote"

            sleep 1
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote/02:submit-failed"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote"

            sleep 1
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote/02:submit-failed"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote"
        """

Submission 01 goes as expected, but things start going terribly wrong afterwards.

Most critically, this exception appears in the scheduler log:

ERROR - [Errno 2] No such file or directory: '.../job/1/remote/03/job'
    Traceback (most recent call last):
      File ".../cylc/flow/scheduler_cli.py", line 719, in cylc_play
        asyncio.get_running_loop()
    RuntimeError: no running event loop
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File ".../cylc/flow/subprocpool.py", line 495, in _run_command_init
        stdin_file = open(  # noqa: SIM115
                     ^^^^^^^^^^^^^^^^^^^^^

Curiously, change remote to remote:submit-failed? and this exception goes away, so an interaction involving TaskProxy state is likely.

[oliver-sanders]

Associated with this bug we have spotted another, rather worrying symptom.

If two remote tasks are triggered at around the same time, the job submit commands seem to get jumbled up.

Adapting the above example to add a second task (expand to reveal config)

[scheduling]
    [[graph]]
        R1 = """
            remote1
            remote2
            operator
        """

[runtime]
    [[remote1, remote2]]
        platform = cazcron1

    [[operator]]
        script = """
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote1/02:submit-failed"
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote2/02:submit-failed"
            cylc pause "${CYLC_WORKFLOW_ID}"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote1"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote2"

            sleep 1
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote1/02:submit-failed"
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote2/02:submit-failed"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote1"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote2"

            sleep 1
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote1/02:submit-failed"
            cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote2/02:submit-failed"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote1"
            cylc trigger "${CYLC_WORKFLOW_ID}//1/remote2"
        """

We get log messages like this:

[jobs-submit cmd] ssh -oBatchMode=yes -oConnectTimeout=8 -oStrictHostKeyChecking=no <host> env CYLC_VERSION=8.6.1.dev
    CYLC_ENV_NAME=cylc-dev bash --login -c ''"'"'exec "$0" "$@"'"'"'' cylc jobs-submit --remote-mode --clean-env --path=/bin --path=/usr/bin
    --path=/usr/local/bin --path=/sbin --path=/usr/sbin --path=/usr/local/sbin -- '$HOME/cylc-run/tmp.GKycdvUOUT/run5/log/job' 1/remote2/04
    [jobs-submit ret_code] 1
    [jobs-submit out] 2025-10-03T16:13:55+01:00|1/remote1/04|1

Pulling out the relevant bits:

[jobs-submit cmd] ssh <host> cylc jobs-submit -... -- '.../log/job' 1/remote2/04
    [jobs-submit ret_code] 1
    [jobs-submit out] 2025...|1/remote1/04|1

The cylc jobs-submit command for remote2 returned a submission for remote1? Originally observed with a PBS submission!

[hjoliver]

Submission 01 goes as expected, but things start going terribly wrong afterwards.

• reproduced
• noticed the 1/remote/03 log directory does not exist when the error occurs
• weirdly (possibly only with longer sleeps?) job 04 goes on to submit without the error

cylc workflow-state "${CYLC_WORKFLOW_ID}//1/remote/02:submit-failed"

BTW, the job number (02) is just ignored there.

[luuqh]

Do you have any idea whether or not this bug is known to 8.6.0 only - or it affects older versions as well, say 8.5.4 or 8.5.1?

[oliver-sanders]

I think this will probably affect Cylc 8.5 too.

[oliver-sanders]

The issue appears to have been caused by #6768 which appears to cause jobs to be run through job preparation (including job file writing) twice in this situation.

This change went in at Cylc 8.5.2