Resume not working using cloud cache and launching from different folders

[FabianDK]

Dear authors,

We are working with nextflow in the Azure cloud using docker containers and all input/output files and workDir in the blob storage as well.

The issue we encountered is that when the same nextflow run command is launched in different folders, the resume function is not working. This is despite everything being equal (main.nf, config, modules, cloud files) apart from the directory where the nf command is relaunched.

The command we use is:

	NXF_VER=24.02.0-edge \
	NXF_CLOUDCACHE_PATH="az://blob_storage/.nextflow/cache" \
	nextflow -log nextflow_logs/log \
	run nextflow_library/nf_template/main.nf \
	-c nextflow_library/nf_template/nextflow.config \
	-profile run_azure_batch,azure_batch_resources,with_docker

It is a simple test script that successfully executes when run in ~/nextflow_lib, but when relaunched in a new folder ~test/nextflow_lib it does not understand the resume (resume = true) defined in the config.

The only way to preserve the cache seems to be to copy the local .nextflow folder in ~/nextflow_lib to the folder where it should be re-executed (~test/nextflow_lib).
Also tried downloading the cache from the cloud to the new folder, but this did not solve the issue.

Isn't the purpose of the cloudcache to enable this?

Thanks,
Daniel

[bentsherman]

Hi @FabianDK , first, I would go through the docs page on caching and resuming, particularly the "tips" at the end. There is a method to compare the task hashes across two runs in a nice diff, it should help us figure out where the resume is failing

[FabianDK]

It would be good if the developers can describe the expected behavior.

The first question is if resume is supposed to work if the same workflow is launched with Azure batch from a different folder/VM/etc, with all input files, cacheDir and workDir in the cloud blob storage.

When I rerun a successfully executed cloud workflow from another VM or folder (not copying over the local .nextflow folder), then I get "WARN: It appears you have never run this project before -- Option -resume is ignored".

I have run the hash compare method that you suggested:

Initial run from Folder 1:

• Using -dump-hashes json

[RUN_TEST_MODULE (1)] cache hash: 2ee913d586bbf1002766ab57aee503e0; mode: STANDARD; entries: [

[TEST_PROCESS (1)] cache hash: 4f22060064d812e4e9763b5fb4a5aff0; mode: STANDARD; entries: [

Resumed run from Folder 2 (without the .nextflow folder):

• Using -dump-hashes json -resume
• Prints: "WARN: It appears you have never run this project before -- Option -resume is ignored"

[RUN_TEST_MODULE (1)] cache hash: be3cb833886ba2b3392af77ba116e89e; mode: STANDARD; entries: [

[TEST_PROCESS (1)] cache hash: c59aa6046ce0e1195243200dc4eb6954; mode: STANDARD; entries: [

It does not seem to contain any relevant info?

There is a similar issue here: #1747

[mearatjames]

@bentsherman
My team is having the exact same issue.
We have executed the nextflow run as a container in k8s in the cloud (aws) and we use aws batch as the executor. We have the NFX_CLOUDCACHE_PATH set to an s3 path. We have confirmed that we can see the cache files written to that path.
However, whenever we're trying to execute nextflow run with the -resume flag, we kept seeing this in the log "WARN: It appears you have never run this project before -- Option -resume is ignored". .

I looked at the source code in

nextflow/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy

Line 527 in c713ad5

log.warn "It appears you have never run this project before -- Option `-resume` is ignored"

and it seems like it is looking for the last session id in the history file.
Since we executed the nextflow run in a container, obviously the subsequent nextflow run with -resume flag in the new container does not know any thing about the previous .nextflow/history. Only the .nextflow/cache is persisted in the cloud (s3) but I don't think the .nextflow/history is persisted along with that cache.

I was under the impression that as long as we have the cloudcache persisted, we should be able to resume the nextflow run without relying on the local .nextflow/history file. Did I miss something here?

Thanks!

[bentsherman]

That's a good point. I think you'll have to specify the session id explicitly i.e. -resume <session-id>. This will allow Nextflow to query the cloudcache without needing to first check the history file.

This is actually how Seqera Platform supports resume, since the history file is also lost in that case.

[mearatjames]

I specified the session id explicitly as you suggested, and now I'm getting a different error "Can't find a run with the specified id: my-session-id-here -- Execution can't be resumed".

It is coming from this ScriptRunner.groovy.

nextflow/modules/nextflow/src/main/groovy/nextflow/script/ScriptRunner.groovy

Line 280 in 935bb1e

throw new AbortOperationException("Can't find a run with the specified id: ${session.uniqueId} -- Execution can't be resumed")

If I read the code correctly, it seems like it is still trying to check the history file to see if that session id exist even when I explicitly pass in the session id to the -resume. Maybe there is something else that I am missing.

[bentsherman]

You are right, it actually checks the history file either way. I didn't think it would be needed, but there is a hidden environment variable that seqera platform uses to disable the history file: NXF_IGNORE_RESUME_HISTORY. Set that to true and it should just ignore the history file. But you do have to specify the resume id since it can't be inferred from the history file.

Seems like this hidden environment variable should actually just default to true when using the cloud cache.

[mearatjames]

Thanks @bentsherman for the suggestion. I am now able to resume the run after setting the NXF_IGNORE_RESUME_HISTORY to true.

One additional thing that I need to do for this to work is to also provide the run name via the -name option. If I don't provide the name while setting NXF_IGNORE_RESUME_HISTORY to true, I will get the error: Missing workflow run name.

[sabrinacamp2]

Would it be possible to have the .nextflow/history file also be stored in the NXF_CLOUDCACHE_PATH location?

Since the NXF_CACHE_DIR variable determines where the local .nextflow folder will be located (including the history file), it would make sense that NXF_CLOUDCACHE_PATH would place the entire .nextflow folder in that cloud location.