CI | NSFS | Versioning | Tests That Sometimes Fail

[shirady]

Environment info

• NooBaa Version: 5.19.0
• Platform: on the CI (can be on NC and Containerized)

Actual behavior

1. Currently the test "test versioning concurrency › nested key - concurrent puts & deletes" sometimes fail - probably related to the fact that it runs on POSIX.

Expected behavior

1. The test should always pass.

Steps to reproduce

1. Run in the CI or run it on your local machine with: sudo npx jest test_nsfs_concurrency.test.js

More information - Screenshots / Logs / Other output

Might be related to issue #8820

[nadavMiz]

looking at the logs the issue is that link fails from tmp file to latest dir with ENOENT:

Feb-26 14:54:58.091 [/1162248]  [WARN] core.sdk.namespace_fs:: NamespaceFS: Retrying failed move to dest retries=19 source_path=/home/nadav/tmp/test_versioning_concurrency/.noobaa-nsfs_1/uploads/26dd5d97-11a4-4fc8-9893-c874c3065dbc dest_path=/home/nadav/tmp/test_versioning_concurrency/dir3/key3 [Error: No such file or directory] { code: 'ENOENT', context: 'SafeLink _link_from.c_str()=/home/nadav/tmp/test_versioning_concurrency/.noobaa-nsfs_1/uploads/26dd5d97-11a4-4fc8-9893-c874c3065dbc _link_to.c_str()=/home/nadav/tmp/test_versioning_concurrency/dir3/key3 _link_expected_mtime=1740574496253000448 _link_expected_inode=268032 ' }

in this test we are concurrently putting and deleting nested objects, the issue is that when we delete a nested object we also delete its directories. so when we put a new object it fails because the base directory doesn't exist (it was deleted). the fix is to recreate the path to the key in case it was deleted inside move_to_dest_versions. in case the path was deleted after the call to the function, we will retry and recreate the paths again

[nadavMiz]

tested locally 15 times. test passed each time. before the fix test failed around one in three times