tests/f: fix remote tests & improve portability #6882

oliver-sanders · 2025-07-25T10:56:28Z

Pre-release testing for 8.5.0 flagged several test issues.

Some tests haven't been run for a while (PBS testing was very difficult for us on the old system, recently retired).
Some tests needed modification to run on the new system.

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
Changelog entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

oliver-sanders · 2025-07-25T10:56:51Z

cylc/flow/task_remote_cmd.py

+        if cylc.flow.flags.verbosity > 1:
+            print(f'$ ln -s "{target}" "{path}"')


Make symlink dir issues easier to debug.

oliver-sanders · 2025-07-25T10:58:32Z

tests/functional/events/11-cycle-task-event-job-logs-retrieve.t

    [[_retrieve]]
        $(cylc config -i "[platforms][$CYLC_TEST_PLATFORM]")
+    [[_retrieve]]
        retrieve job logs = True
-        install target = $CYLC_TEST_PLATFORM
    [[_no_retrieve]]
        $(cylc config -i "[platforms][$CYLC_TEST_PLATFORM]")
+    [[_no_retrieve]]
        retrieve job logs = False
-        install target = $CYLC_TEST_PLATFORM
 "


Allow default [directives] to be specified in the test config.

install target should already be set correctly.

oliver-sanders · 2025-07-25T10:59:26Z

tests/functional/events/17-task-event-job-logs-retrieve-command.t

+sed -i -E 's/--max-size=[^ ]* //' 'my-rsync.log.edited'  # strip "retrieve job logs max size" arg
+sort -u 'my-rsync.log.edited'  # stip out duplicates (can result from PBS log file spooling)


Strip --max-size options which may or may not be added based on config.

Tolerate job log retrieval retries (needed when PBS output spooling is in play).

oliver-sanders · 2025-07-25T11:00:00Z

tests/functional/intelligent-host-selection/04-kill.t

    [[goodhostplatform]]
        hosts = ${CYLC_TEST_HOST}
-        install target = ${CYLC_TEST_INSTALL_TARGET}


We can't define platforms like this because they don't inherit any config from $CYLC_TEST_PLATFORM.

oliver-sanders · 2025-07-25T11:00:35Z

tests/functional/intelligent-host-selection/04-kill.t

-named_grep_ok "job kill retries & succeeds" \
-    "\[jobs-kill out\] \[TASK JOB SUMMARY\].*1/mixedhosttask/01" \
-    "${LOGFILE}"


This test turned out to be fragile. Replaced with more targetted checks.

oliver-sanders · 2025-07-25T11:01:02Z

tests/functional/intelligent-host-selection/04-kill/flow.cylc

+    [[goodhosttask]]
+        script = sleep 60
+        platform = goodhostplatform
+
+    [[mixedhosttask]]
+        script = sleep 60
+        platform = mixedhostplatform


Ensure both tasks are running when the kill command is sent.

oliver-sanders · 2025-07-25T11:02:35Z

tests/functional/remote/04-symlink-dirs.t

            share/cycle = \$TMPDIR/\$USER/cylctb_tmp_share_dir
            work = \$TMPDIR/\$USER
        [[[$CYLC_TEST_INSTALL_TARGET]]]
-            run = \$TMPDIR/\$USER/test_cylc_symlink/ctb_tmp_run_dir


$TMPDIR/$USER does not exist on our new machine and $TMPDIR path changes from node to node :/

Switched to a permanent directory $HOME/cylctb-symlinks with one subdir for each test.

oliver-sanders · 2025-07-25T11:02:53Z

tests/functional/remote/08-symlink-dir-target-exist.t

    [[symlink dirs]]
        [[[${CYLC_TEST_INSTALL_TARGET}]]]
-            run = \$TMPDIR/\$USER/sym-run
+            run = \$HOME/cylctb-symlinks/$TEST_NAME/


oliver-sanders · 2025-07-25T15:27:01Z

tests/functional/cylc-cat-log/09-cat-running/flow.cylc

+            cylc message -- 'echo done'
+
+            # wait up to PT1M for the cat-log task to succeed
+            # (the workflow will shut down if cat-log fails)
+            sleep 60
+
+            # fail if the task was not orphaned by this point
+            false


This test was supposed to be testing cat-log against running tasks:

Test "cylc cat-log" of currently-running local and remote jobs.

However, there was nothing to keep the task running, so it was actually testing succeeded jobs.

I've changed the approach, the tasks now sleep for 60s, then fail.

If the cat-log works, it will cylc set the task, orphaning the sleep and triggering fin which is used to diagnose test success.

I went back to an early version of this test, and it used to have sleep 60 there. Maybe got borked during migration to Cylc 8.

Matt did purposefully try to pull out sleeps from the tests in order to speed them up reduce flakyness. It might have vanished then.

In this particular case, the sleep is ok, but it could have easily been mistaken.

oliver-sanders · 2025-09-30T15:54:42Z

tests/functional/lib/bash/test_header

    local OPTS="$*"
    local TEST_NAME
-    TEST_NAME="grep-ok: ${NAME}"
+    TEST_NAME="$(basename "${FILE}")-grep-ok"


Revert an erroneous change made in 15ac2fb which broke test reporting.

oliver-sanders · 2025-09-30T15:55:46Z

All tests now passing on _remote_background_indep_pbs

tests/functional/cylc-poll/04-poll-multi-hosts/flow.cylc

tests/functional/events/17-task-event-job-logs-retrieve-command.t

tests/functional/intelligent-host-selection/04-kill.t

hjoliver · 2025-10-01T01:12:11Z

(I had a quick run through this; it generally LGTM, mysterious polling result notwithstanding).

wxtim · 2025-10-01T08:24:13Z

tests/functional/platforms/13-ping-pong-host.t

I should have made this _remote_background_indep_* only. The consequences of using the _remote_pbs_indep_tcp don't make sense.

tests/functional/intelligent-host-selection/04-kill.t

oliver-sanders · 2025-10-01T10:43:07Z

tests/functional/intelligent-host-selection/04-kill.t

 # Test job kill will retry on a different host if there is a connection failure

-export REQUIRE_PLATFORM='loc:remote fs:indep comms:tcp'
+export REQUIRE_PLATFORM='loc:remote fs:indep comms:tcp runner:background'


Test relies on mocking background submission (my_background) so can only be run with background platforms.

oliver-sanders · 2025-10-01T11:23:02Z

tests/functional/cylc-poll/04-poll-multi-hosts/flow.cylc

+            # give a bit of grace time for the job to leave the job runner queue
+            sleep 5


Unavoidable sleep :(

Give PBS a grace period to remove the finished job from the queue before attempting to poll.

This failed reliably yesterday, but passed flakily today. Looks like the required grace period is fairly small.

oliver-sanders · 2025-10-01T12:27:40Z

Have added a bunch of fixes for tests which were using spaces, colons, slashes, etc in test IDs. These IDs are used as filename prefixes.

* Cylc platforms defined in tests must inherit from the test platform config. * Reference tests only capture triggering, not task outcome (add a downstream task if you need to check outputs). * Fix some timing issues. * Use the platform configured SSH command for any add-hoc SSH'es. * Remove `TMPDIR` dependence (ain't always there!).

* Test requires the background job runner. * Test names cannot contain spaces. * Revert an erroneous change made in 15ac2fb which broke test reporting.

* Tasks needed to wait for their started messages. * Give the job runner a few seconds grace time for the job to exit the queue. * Document the test.

* Test works by overriding the background job runner so is not compatible with other job runners.

* Test is only compatible with background platforms.

* Test IDs should be prefixed with "$TEST_NAME_BASE". * Test IDs should be valid for use as file names (no ":", "/", " ", etc). * `named_grep_ok` was broken by default :(

oliver-sanders · 2025-10-01T15:05:46Z

All tests now passing on:

_local_slurm_indep_tcp
_remote_background_indep_tcp
_remote_pbs_indep_tcp

(and the other remote platforms covered by CI)

oliver-sanders · 2025-10-14T09:52:19Z

@hjoliver, @wxtim, gentle poke, would really like to have the tests passing on master.

tests/functional/cylc-cat-log/09-cat-running/flow.cylc

wxtim · 2025-10-15T12:10:10Z

Over 8 runs against _remote_pbs_indep_tcp I have failures

1 tests/functional/events/17-task-event-job-logs-retrieve-command.t
1 tests/functional/events/11-cycle-task-event-job-logs-retrieve.t
3 tests/functional/remote/08-symlink-dir-target-exist.t
5 tests/functional/cylc-cat-log/09-cat-running.t
1 tests/functional/remote/04-symlink-dirs.t

oliver-sanders · 2025-10-15T13:44:57Z

All pass for me. Have you synced your branch to the test platform?

wxtim · 2025-10-16T10:21:25Z

Now have all tests passing. Some of these seem a little sensitive to global config settings.

oliver-sanders · 2025-10-16T10:27:59Z

(for info, the failure(s) Tim was seeing were related to timing issues around PBS job log spooling, we have to configure log retrieval retries to make this work)

oliver-sanders · 2025-10-23T12:49:59Z

Ping @hjoliver

oliver-sanders added this to the 8.5.x milestone Jul 25, 2025

oliver-sanders self-assigned this Jul 25, 2025

oliver-sanders commented Jul 25, 2025

View reviewed changes

oliver-sanders marked this pull request as draft July 25, 2025 12:08

oliver-sanders force-pushed the remote-tests++ branch from f1ba4c3 to 1d0563e Compare July 25, 2025 15:24

oliver-sanders commented Jul 25, 2025

View reviewed changes

MetRonnie changed the base branch from master to 8.5.x July 29, 2025 11:47

oliver-sanders changed the base branch from 8.5.x to master September 30, 2025 14:58

oliver-sanders commented Sep 30, 2025

View reviewed changes

oliver-sanders modified the milestones: 8.5.x, 8.6.0 Sep 30, 2025

oliver-sanders commented Sep 30, 2025

View reviewed changes

tests/functional/cylc-poll/04-poll-multi-hosts/flow.cylc Outdated Show resolved Hide resolved

hjoliver reviewed Oct 1, 2025

View reviewed changes

tests/functional/events/17-task-event-job-logs-retrieve-command.t Outdated Show resolved Hide resolved

hjoliver reviewed Oct 1, 2025

View reviewed changes

tests/functional/intelligent-host-selection/04-kill.t Outdated Show resolved Hide resolved

wxtim reviewed Oct 1, 2025

View reviewed changes

tests/functional/intelligent-host-selection/04-kill.t Show resolved Hide resolved

oliver-sanders commented Oct 1, 2025

View reviewed changes

remote-init: add debug logging for symlinks

ec80eb7

oliver-sanders force-pushed the remote-tests++ branch from 65df52a to be94b23 Compare October 1, 2025 10:52

oliver-sanders commented Oct 1, 2025

View reviewed changes

oliver-sanders force-pushed the remote-tests++ branch from 4eeca35 to 36f98ea Compare October 1, 2025 12:26

oliver-sanders force-pushed the remote-tests++ branch from 36f98ea to 8715bcc Compare October 1, 2025 12:57

oliver-sanders added 6 commits October 1, 2025 15:42

tests/f: fix platforms/13

afe6ff3

* Test requires the background job runner. * Test names cannot contain spaces. * Revert an erroneous change made in 15ac2fb which broke test reporting.

tests/f: fix cylc-poll/04 for PBS

4ba34dd

* Tasks needed to wait for their started messages. * Give the job runner a few seconds grace time for the job to exit the queue. * Document the test.

test/f: fix intelligent-host-selection/04

e9457ed

* Test works by overriding the background job runner so is not compatible with other job runners.

tests/f: fix platforms/13

883f0e1

* Test is only compatible with background platforms.

tests/f: fix test IDs

5860116

* Test IDs should be prefixed with "$TEST_NAME_BASE". * Test IDs should be valid for use as file names (no ":", "/", " ", etc). * `named_grep_ok` was broken by default :(

oliver-sanders force-pushed the remote-tests++ branch from 8715bcc to 5860116 Compare October 1, 2025 14:42

oliver-sanders marked this pull request as ready for review October 1, 2025 15:03

oliver-sanders added this to the 8.6.x milestone Oct 1, 2025

oliver-sanders requested review from hjoliver and wxtim October 1, 2025 15:03

wxtim reviewed Oct 15, 2025

View reviewed changes

tests/functional/cylc-cat-log/09-cat-running/flow.cylc Show resolved Hide resolved

wxtim approved these changes Oct 16, 2025

View reviewed changes

		if cylc.flow.flags.verbosity > 1:
		print(f'$ ln -s "{target}" "{path}"')

		sed -i -E 's/--max-size=[^ ]* //' 'my-rsync.log.edited' # strip "retrieve job logs max size" arg
		sort -u 'my-rsync.log.edited' # stip out duplicates (can result from PBS log file spooling)

		# give a bit of grace time for the job to leave the job runner queue
		sleep 5

tests/f: fix remote tests & improve portability #6882

Are you sure you want to change the base?

tests/f: fix remote tests & improve portability #6882

Uh oh!

Conversation

oliver-sanders commented Jul 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oliver-sanders commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hjoliver commented Oct 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oliver-sanders commented Oct 1, 2025

Uh oh!

oliver-sanders commented Oct 1, 2025

Uh oh!

oliver-sanders commented Oct 14, 2025

Uh oh!

Uh oh!

wxtim commented Oct 15, 2025

Uh oh!

oliver-sanders commented Oct 15, 2025

Uh oh!

wxtim commented Oct 16, 2025

Uh oh!

oliver-sanders commented Oct 16, 2025

Uh oh!

oliver-sanders commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

oliver-sanders Oct 1, 2025 •

edited

Loading

oliver-sanders Sep 30, 2025 •

edited

Loading