Artifact folder could not be made on Windows.

[KnutAM]

When running Pkg.add("CairoMakie") in an empty project I'm getting this, Error: "C:\\Users\\meyer\\.julia\\artifacts\\a8244d6d23cbb895fcd39dd3eddb859a0c05d1c6" could not be made, which I seems to occur as the atomic rename fails after #3768.

However, before getting there, I hit JuliaLang/julia#34700, but that could be due to a failure related to above, but the rm is in a final block...

To "fix" the problem, I first hacked Base.rm

function Base.rm(path::String; force::Bool=false, recursive::Bool=false)
   if !(force && recursive)
       return invoke(Base.rm, Tuple{AbstractString}, path; force, recursive)
   else
       max_attempts = 3
       attempts = 0
       while attempts < max_attempts
           attempts += 1
           try
               invoke(Base.rm, Tuple{AbstractString}, path; recursive=true, force=true)
           catch err
               if isa(err, Base.IOError) && attempts < max_attempts
                   println("Trying again for \"", path, "\"")
                   attempts == (max_attempts - 1) && sleep(1.0)
                   continue
               else
                   println("Failed for \"", path, "\"")
                   rethrow(err)
               end
           end
       end
   end
end

(but this could just as well be done in create_artifact)

And modified _mv_temp_artifact_dir (which I couldn't hack since the dispatch is concrete...)

function _mv_temp_artifact_dir(temp_dir::String, new_path::String)::Nothing
    if !isdir(new_path)
        # This next step is like
        # `mv(temp_dir, new_path)`.
        # However, `mv` defaults to `cp` if `rename` returns an error.
        # `cp` is not atomic, so avoid the potential of calling it.
        err = ccall(:jl_fs_rename, Int32, (Cstring, Cstring), temp_dir, new_path)
        # Ignore rename error, but ensure `new_path` exists.
        if !isdir(new_path)
            println("Just do cp 💀")
            mv(temp_dir, new_path)
        end
        if !isdir(new_path)
            error("$(repr(new_path)) could not be made")
        end
        chmod(new_path, filemode(dirname(new_path)))
        set_readonly(new_path)
    end
    nothing
end

Just do cp 💀 is printed twice, along with a couple of Trying again for ....
In fact, it seems like the x264_jll and Pixman_jll are the ones that needs retries, not sure if by chance or anything special about these.

Strangely, trying many times, it seemed to work, but fails again if I empty the artifacts folder...

Versioninfo

Julia Version 1.10.2
Commit bd47eca2c8 (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, tigerlake)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS =

[KnutAM]

Update, perhaps not unsuprisingly, it seems to have been an anti-virus issue (using a different, non-centrally administrated anti-virus solved the problem).
Even so, perhaps adding some fix with retry (or even if using Base.mv could be made safe) could make sense? Having these issues could be a reason for first-time users to give up on julia?

[KristofferC]

Having these issues could be a reason for first-time users to give up on julia?

Perhaps, but at the same time it feels fruitless to fight anti viruses since they sit at a different privilege level and can just block anything they feel like.

[KnutAM]

What about providing a hint to windows users to check if their anti-virus could be interfering then?

[nhz2]

This post seems to be about a similar issue https://discourse.julialang.org/t/installation-of-nodejs-fails-directory-not-empty/111117

It might be good to put the rm at src\Artifacts.jl:376 in a try block. If the rm fails there will be some extra junk in the artifact directory, but no serious problems. I initially didn't do that because I thought rm failing was a very rare event.

If you run Pkg.add("CairoMakie") again after it fails it should hopefully make progress. Even if there is an error, the artifact folder might still be in a valid state.

[nhz2]

Oh, I see, the real issue is jl_fs_rename failing. Would sleeping and multiple attempts of calling jl_fs_rename fix the issue?

[IanButterworth]

As @vtjnash points out, it may be possible to change libuv to using posix rename on Windows. Posix rm on windows is ready and tested with julia libuv/libuv#4318

[vtjnash]

This seems a common misconception, so just note that that comment does not reflect what atomic rename is documented to mean on Unix or anywhere else either. The atomic only refers to the new file name and contents, but does not require the old name to be deleted simultaneously

   If newpath already exists, it will be atomically replaced, so

   that there is no point at which another process attempting to
   access newpath will find it missing.  However, there will
   probably be a window in which both oldpath and newpath refer to
   the file being renamed.

[nhz2]

When renaming directories instead of files, does that mean oldpath and newpath can refer to the same directory shortly after rename succeeds?

[vtjnash]

No, most file systems usually won't permit a hard link to be created to a directory

[KnutAM]

The atomic only refers to the new file name and contents, but does not require the old name to be deleted simultaneously

So in that case, it should be fine/equivalent to call cp instead of using jl_fs_rename, since this would only occur if new_path doesn't exists?

And in that case, most likely this problem would be solved? Unfortunately, I don't have time to re-install the old antivirus to test that now...

[mzaffalon]

This post seems to be about a similar issue https://discourse.julialang.org/t/installation-of-nodejs-fails-directory-not-empty/111117

The fix suggested in #3822 (comment) (the rm part only) did not work for me. I deleted the artifacts directory and reinstalled without problems.

EDIT: after rereading the OP, the error I got with the hacked rm was the same.

[nhz2]

The issue with using cp is that it is not atomic for directories. It makes the destination directory and then copies the files one at a time (if I understand the code in base/file.jl correctly). If you have a huge artifact and quit julia while it is doing the cp you will end up with an invalid artifact directory.

[vtjnash]

Yeah, it feels like someone really needs to rewrite a lot of that code in Filesystem better. It is currently intended to replicate the mv command with force instead of --clobber / -n, which has this non-atomic behavior, as documented in that man page.

For example, we could try to be inspired more by the rsync API, which is a general purpose tool for both mv, copy, and delete, atomically, given various combinations of --delay-updates, -a, and --remove-source-files options. The Filesystem module doesn't need most of the options--that can be left to a package--but better care with these options would be quite welcome for being able to deal with cases like this better.

[nhz2]

@mzaffalon @KnutAM
Do you think the example error message in the PR #3827 will help others workaround similar issues?

I don't know how to test this in a real-world scenario because I don't have a problematic anti-virus installed.

But if you want to test this out https://github.com/JuliaLang/Pkg.jl?tab=readme-ov-file#using-the-development-version-of-pkgjl has instructions for using the development version of Pkg.jl. You will also need the nightly version of julia https://julialang.org/downloads/nightlies/