Update LibGit2 to 1.9.0 #56955

fxcoudert · 2025-01-05T10:06:18Z

New version: https://github.com/libgit2/libgit2/releases/tag/v1.9.0

fxcoudert · 2025-01-05T11:26:51Z

I have zero idea where the problem is…

Failed to precompile LazyArtifacts [4af54fe1-eca0-43a8-85a7-787d91b784e3] to "[buildroot]/usr/share/julia/compiled/v1.12/LazyArtifacts/jl_leacuO".
[14072] signal 11 (2): Segmentation fault
in expression starting at [buildroot]/usr/share/julia/stdlib/v1.12/Pkg/src/precompile.jl:101
ERROR: LoadError: Failed to precompile Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] to "[buildroot]/usr/share/julia/compiled/v1.12/Pkg/jl_2cUeCV".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:44
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int32}, loadable_exts::Nothing)
    @ Base ./loading.jl:3279
  [3] (::Base.var"#__require_prelocked##0#__require_prelocked##1"{Base.PkgId})()
    @ Base ./loading.jl:2673
  [4] mkpidlock(f::Base.var"#__require_prelocked##0#__require_prelocked##1"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int32, wait::Bool})
    @ FileWatching.Pidfile [buildroot]/usr/share/julia/stdlib/v1.12/FileWatching/src/pidfile.jl:93
  [5] #mkpidlock#7
    @ [buildroot]/usr/share/julia/stdlib/v1.12/FileWatching/src/pidfile.jl:88 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int32})
    @ FileWatching.Pidfile [buildroot]/usr/share/julia/stdlib/v1.12/FileWatching/src/pidfile.jl:114
  [7] #invokelatest#1
    @ ./essentials.jl:1058 [inlined]
  [8] invokelatest
    @ ./essentials.jl:1052 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#__require_prelocked##0#__require_prelocked##1"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int32)
    @ Base ./loading.jl:3850
 [10] maybe_cachefile_lock
    @ ./loading.jl:3847 [inlined]
 [11] __require_prelocked(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2659
 [12] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2484
 [13] macro expansion
    @ ./loading.jl:2412 [inlined]
 [14] macro expansion
    @ ./lock.jl:376 [inlined]
 [15] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2369
 [16] #invoke_in_world#2
    @ ./essentials.jl:1091 [inlined]
 [17] invoke_in_world
    @ ./essentials.jl:1087 [inlined]
 [18] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2361
 [19] top-level scope
    @ [buildroot]/usr/share/julia/stdlib/v1.12/LazyArtifacts/src/LazyArtifacts.jl:13
 [20] include(mod::Module, _path::String)
    @ Base ./Base.jl:300
 [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2995
 [22] top-level scope
    @ stdin:6
 [23] eval(m::Module, e::Any)
    @ Core ./boot.jl:485
 [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:2846
 [25] include_string
    @ ./loading.jl:2856 [inlined]
 [26] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:327
 [27] _start()
    @ Base ./client.jl:558
in expression starting at [buildroot]/usr/share/julia/stdlib/v1.12/LazyArtifacts/src/LazyArtifacts.jl:3
in expression starting at stdin:6

giordano · 2025-01-05T11:52:05Z

Since this is happening during the compilation of the package manager, which uses libgit2 internally for git operations, I suspect some more changes are needed to adapt to new libgit2

fxcoudert · 2025-01-05T17:21:39Z

I've gone through the list of breaking changes, I don't see anymore. Any idea how I can obtain a proper backtrace of the error?

giordano · 2025-01-05T18:30:17Z

I was able to compile Julia locally, but I realised I had JULIA_PKG_USE_CLI_GIT=true, which was avoiding using libgit2 entirely 😅 but that's good, because then I can get a working Julia build and then look at proper stacktrace at runtime after changing JULIA_PKG_USE_CLI_GIT=false:

julia> Pkg.activate(; temp=true)
  Activating new project at `/var/folders/v2/hmy3kzgj4tb3xsy8qkltxd0r0000gn/T/jl_EAw7FL`

julia> Pkg.add(; name="Example", rev="master")
    Updating git-repo `https://github.com/JuliaLang/Example.jl.git`
ERROR: failed to fetch from https://github.com/JuliaLang/Example.jl.git, error: GitError(Code:ERROR, Class:Invalid, invalid version 0 on git_proxy_options)
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/Types.jl:68
  [2] fetch(io::IOContext{IO}, repo::LibGit2.GitRepo, remoteurl::String; header::Nothing, credentials::Nothing, refspecs::Vector{String}, kwargs::@Kwargs{})
    @ Pkg.GitTools ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/GitTools.jl:189
  [3] (::Pkg.Types.var"#handle_repo_add!##0#handle_repo_add!##1"{Pkg.Types.Context, Pkg.Types.PackageSpec, String})(repo::LibGit2.GitRepo)
    @ Pkg.Types ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/Types.jl:909
  [4] with(f::Pkg.Types.var"#handle_repo_add!##0#handle_repo_add!##1"{Pkg.Types.Context, Pkg.Types.PackageSpec, String}, obj::LibGit2.GitRepo)
    @ LibGit2 ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/LibGit2/src/types.jl:1205
  [5] handle_repo_add!(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec)
    @ Pkg.Types ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/Types.jl:884
  [6] handle_repos_add!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.Types ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/Types.jl:954
  [7] add(ctx::Pkg.Types.Context, pkgs::Vector{…}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, target::Symbol, allow_autoprecomp::Bool, kwargs::@Kwargs{…})
    @ Pkg.API ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/API.jl:296
  [8] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{})
    @ Pkg.API ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/API.jl:164
  [9] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/API.jl:153
 [10] add(; name::String, uuid::Nothing, version::Nothing, url::Nothing, rev::String, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::@Kwargs{})
    @ Pkg.API ./array.jl:0
 [11] top-level scope
    @ REPL[9]:1

caused by: GitError(Code:ERROR, Class:Invalid, invalid version 0 on git_proxy_options)
Stacktrace:
  [1] macro expansion
    @ ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/LibGit2/src/error.jl:124 [inlined]
  [2] fetch(rmt::LibGit2.GitRemote, refspecs::Vector{String}; options::LibGit2.FetchOptions, msg::String)
    @ LibGit2 ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/LibGit2/src/remote.jl:318
  [3] fetch
    @ ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/LibGit2/src/remote.jl:313 [inlined]
  [4] fetch(repo::LibGit2.GitRepo; remote::String, remoteurl::SubString{String}, refspecs::Vector{String}, credentials::Nothing, callbacks::Dict{Symbol, Tuple{Ptr{Nothing}, Any}})
    @ LibGit2 ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/LibGit2/src/LibGit2.jl:294
  [5] fetch
    @ ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/LibGit2/src/LibGit2.jl:271 [inlined]
  [6] fetch(io::IOContext{IO}, repo::LibGit2.GitRepo, remoteurl::String; header::Nothing, credentials::Nothing, refspecs::Vector{String}, kwargs::@Kwargs{})
    @ Pkg.GitTools ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/GitTools.jl:182
  [7] (::Pkg.Types.var"#handle_repo_add!##0#handle_repo_add!##1"{Pkg.Types.Context, Pkg.Types.PackageSpec, String})(repo::LibGit2.GitRepo)
    @ Pkg.Types ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/Types.jl:909
  [8] with(f::Pkg.Types.var"#handle_repo_add!##0#handle_repo_add!##1"{Pkg.Types.Context, Pkg.Types.PackageSpec, String}, obj::LibGit2.GitRepo)
    @ LibGit2 ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/LibGit2/src/types.jl:1205
  [9] handle_repo_add!(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec)
    @ Pkg.Types ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/Types.jl:884
 [10] handle_repos_add!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.Types ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/Types.jl:954
 [11] add(ctx::Pkg.Types.Context, pkgs::Vector{…}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, target::Symbol, allow_autoprecomp::Bool, kwargs::@Kwargs{…})
    @ Pkg.API ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/API.jl:296
 [12] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{})
    @ Pkg.API ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/API.jl:164
 [13] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/repo/julia/libgit2-1.9-56955/usr/share/julia/stdlib/v1.12/Pkg/src/API.jl:153
 [14] add(; name::String, uuid::Nothing, version::Nothing, url::Nothing, rev::String, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::@Kwargs{})
    @ Pkg.API ./array.jl:0
 [15] top-level scope
    @ REPL[9]:1
Some type information was truncated. Use `show(err)` to see complete types.

This isn't the segmentation fault reported above, but another indication of code path which apparently changed.

IanButterworth · 2025-01-05T19:48:45Z

It's erroring at

julia/stdlib/LibGit2/src/repository.jl

Line 470 in bee56f4

@check ccall((:git_clone, libgit2), Cint,

with

repo_ptr_ptr = Base.RefValue{Ptr{Nothing}}(Ptr{Nothing}(0x0000000000000000))
repo_url = "/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_gKLmlR/TestPkg.jl"
repo_path = "/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_gKLmlR/clones/bca5e194-8749-4d4a-8ffb-6c970e4b6e0d"
clone_opts_ref = Base.RefValue{LibGit2.CloneOptions}( #redacted# )

within https://github.com/JuliaLang/Pkg.jl/blob/1eb09be2ae808f5c4a50a421710df5881513bf7e/src/Operations.jl#L760

Failed to precompile Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] to "/Users/ian/.julia/compiled/v1.12/Pkg/jl_uBcoGw".
┌ Debug: tiered_resolve: trying PRESERVE_ALL_INSTALLED
└ @ Pkg.Operations ~/Documents/GitHub/Pkg.jl/src/Operations.jl:1619
┌ Debug: tiered_resolve: trying PRESERVE_ALL
└ @ Pkg.Operations ~/Documents/GitHub/Pkg.jl/src/Operations.jl:1626
[ Info: LibGit2 Downloading package TestPkg from Set(["/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_qZXljl/TestPkg.jl"])
┌ Debug: Starting install_git for package 'TestPkg' (43e4a733-3593-4a6b-9653-c20770ff97dc).
└ @ Pkg.Operations ~/Documents/GitHub/Pkg.jl/src/Operations.jl:756
┌ Debug: Ensuring clones directory exists at '/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_qZXljl/clones'.
└ @ Pkg.Operations ~/Documents/GitHub/Pkg.jl/src/Operations.jl:760
┌ Debug: Repository path determined: '/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_qZXljl/clones/43e4a733-3593-4a6b-9653-c20770ff97dc'.
└ @ Pkg.Operations ~/Documents/GitHub/Pkg.jl/src/Operations.jl:765
┌ Debug: Cloning/ensuring repository from '/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_qZXljl/TestPkg.jl' to '/var/folders/1z/jf841bdj73bdj3vk7kc7f_3w0000gn/T/jl_qZXljl/clones/43e4a733-3593-4a6b-9653-c20770ff97dc' as a bare repo.
└ @ Pkg.Operations ~/Documents/GitHub/Pkg.jl/src/Operations.jl:768

[82786] signal 10 (1): Bus error: 10
in expression starting at /Users/ian/Documents/GitHub/Pkg.jl/src/precompile.jl:101
unknown function (ip: 0x103d10008) at (unknown file)
Allocations: 90163933 (Pool: 90162426; Big: 1507); GC: 2

That's with some debug logs JuliaLang/Pkg.jl@master...IanButterworth:Pkg.jl:ib/libgit_debug

antonio-rojas · 2025-01-05T21:52:11Z

RemoteCallbacks needs to be synced too, see https://gitlab.archlinux.org/archlinux/packaging/packages/julia/-/blob/main/julia-libgit2-1.9.patch?ref_type=heads#L9-11

giordano · 2025-01-05T22:14:37Z

Ah, yeah, that was changed in libgit2/libgit2#6559 but not mentioned in the breaking changes, thanks for the tip!

Suggested-by: Antonio Rojas <[email protected]>

giordano · 2025-01-05T23:37:35Z

https://buildkite.com/julialang/julia-master/builds/43471#019438a0-94af-4de2-bbc3-fac683e44db8/853-1698

The global RNG seed was 0x4c69563434439e722bdc7b4a972274cc.
Error in testset LibGit2/libgit2:
Error During Test at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-master/julia-cb6f8c7de2/share/julia/stdlib/v1.12/LibGit2/test/libgit2-tests.jl:1170
  Got exception outside of a @test
  BoundsError: attempt to access 0-element Vector{LibGit2.FetchHead} at index [1]
  Stacktrace:
    [1] throw_boundserror(A::Vector{LibGit2.FetchHead}, I::Tuple{Int64})
      @ Base ./essentials.jl:15
    [2] getindex
      @ ./essentials.jl:916 [inlined]
    [3] (::Main.Test21Main_LibGit2_libgit2.LibGit2Tests.var"#210#211"{Main.Test21Main_LibGit2_libgit2.LibGit2Tests.var"#add_and_commit_file#207"})(repo::LibGit2.GitRepo)
      @ Main.Test21Main_LibGit2_libgit2.LibGit2Tests /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-master/julia-cb6f8c7de2/share/julia/stdlib/v1.12/LibGit2/test/libgit2-tests.jl:1222
    [4] with(f::Main.Test21Main_LibGit2_libgit2.LibGit2Tests.var"#210#211"{Main.Test21Main_LibGit2_libgit2.LibGit2Tests.var"#add_and_commit_file#207"}, obj::LibGit2.GitRepo)
      @ LibGit2 /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-master/julia-cb6f8c7de2/share/julia/stdlib/v1.12/LibGit2/src/types.jl:1208
    [5] macro expansion
      @ /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-master/julia-cb6f8c7de2/share/julia/stdlib/v1.12/LibGit2/test/libgit2-tests.jl:1171 [inlined]

@antonio-rojas is this why you added https://gitlab.archlinux.org/archlinux/packaging/packages/julia/-/blob/7a544826d25e5d13231946a660f4649545718b97/julia-libgit2-1.9.patch#L71?

Suggested-by: Antonio Rojas <[email protected]>

giordano · 2025-01-05T23:47:03Z

I guess so, because with that change I can run that test file successfully:

julia> Base.runtests("LibGit2/libgit2")
Running parallel tests with:
  getpid() = 29424
  nworkers() = 1
  nthreads() = 1
  Sys.CPU_THREADS = 2
  Sys.total_memory() = 16.000 GiB
  Sys.free_memory() = 2.630 GiB

Test        (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
LibGit2/libgit2  (1) |        started at 2025-01-06T00:43:24.624
LibGit2/libgit2  (1) |   114.30 |   0.44 |  0.4 |    1488.88 |  1163.80

Test Summary: | Pass  Total     Time
  Overall     |  762    762  1m55.0s
    SUCCESS

We should be good to go with that one!

antonio-rojas · 2025-01-06T09:28:10Z

@antonio-rojas is this why you added https://gitlab.archlinux.org/archlinux/packaging/packages/julia/-/blob/7a544826d25e5d13231946a660f4649545718b97/julia-libgit2-1.9.patch#L71?

Right, clone_repo no longer does automatically fetch

https://github.com/libgit2/libgit2/blob/69555048fd471934296e9fe7d8711170429cc904/src/libgit2/clone.c#L620

inkydragon added the external dependencies Involves LLVM, OpenBLAS, or other linked libraries label Jan 5, 2025

fxcoudert force-pushed the libgit2 branch from 385c232 to c8a8b7e Compare January 5, 2025 10:45

nsajko added stdlib Julia's standard library JLLs labels Jan 5, 2025

giordano added the libgit2 The libgit2 library or the LibGit2 stdlib module label Jan 5, 2025

fxcoudert force-pushed the libgit2 branch from 75f658c to 2e9ae99 Compare January 5, 2025 17:20

fxcoudert and others added 6 commits January 5, 2025 23:33

Update LibGit2 to 1.9.0

fa70a24

Refresh checksums

07d002e

Update shared lib name

82960e7

Add ConfigBackendEntry

4a6dacc

[deps] Add riscv64 checksums of libgit2

903923e

[LibGit2] Adapt RemoteCallbacks

cb6f8c7

Suggested-by: Antonio Rojas <[email protected]>

giordano force-pushed the libgit2 branch from 2e9ae99 to cb6f8c7 Compare January 5, 2025 22:39

[LibGit2] Fix cloning function in libgit2 tests

a0d6de3

Suggested-by: Antonio Rojas <[email protected]>

giordano merged commit c4825b0 into JuliaLang:master Jan 6, 2025
7 checks passed

fxcoudert deleted the libgit2 branch January 6, 2025 11:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update LibGit2 to 1.9.0 #56955

Update LibGit2 to 1.9.0 #56955

fxcoudert commented Jan 5, 2025

fxcoudert commented Jan 5, 2025

giordano commented Jan 5, 2025

fxcoudert commented Jan 5, 2025

giordano commented Jan 5, 2025

IanButterworth commented Jan 5, 2025 •

edited

Loading

antonio-rojas commented Jan 5, 2025 •

edited

Loading

giordano commented Jan 5, 2025

giordano commented Jan 5, 2025

giordano commented Jan 5, 2025

antonio-rojas commented Jan 6, 2025

Update LibGit2 to 1.9.0 #56955

Update LibGit2 to 1.9.0 #56955

Conversation

fxcoudert commented Jan 5, 2025

fxcoudert commented Jan 5, 2025

giordano commented Jan 5, 2025

fxcoudert commented Jan 5, 2025

giordano commented Jan 5, 2025

IanButterworth commented Jan 5, 2025 • edited Loading

antonio-rojas commented Jan 5, 2025 • edited Loading

giordano commented Jan 5, 2025

giordano commented Jan 5, 2025

giordano commented Jan 5, 2025

antonio-rojas commented Jan 6, 2025

IanButterworth commented Jan 5, 2025 •

edited

Loading

antonio-rojas commented Jan 5, 2025 •

edited

Loading