Skip to content

refactor discovery, ship therock tarballs as artifacts (2) - #1028

Draft
simeonschaub wants to merge 17 commits into
mainfrom
sds/rocm_libs
Draft

refactor discovery, ship therock tarballs as artifacts (2)#1028
simeonschaub wants to merge 17 commits into
mainfrom
sds/rocm_libs

Conversation

@simeonschaub

Copy link
Copy Markdown
Member

(Same as #1024, reopened from the upstream branch to run MI300 CI)

Some of the AI comments still need to be cleaned up but the logic looks reasonable from a first review, so I thought I'd put this up as a draft for feedback.

It turns out MIOpen needs all of libllvm and libclang, so unlike discussed in #919 (comment), I went back to shipping therock artifacts directly as done in that PR, since we don't gain much by repackaging anymore.

Otherwise, do a split similar to CUDA.jl of having ROCm_Runtime be a pseudo-jll providing automatic downloads through artifacts and having manual discovery live in ROCm_Runtime_Discovery. Either one of those is then loaded depending on a preference

@simeonschaub

Copy link
Copy Markdown
Member Author

Hmm, looks like an out of memory error. @luraess have you seen that before? The question is whether it's just an intermittent issue, whether ROCm 7.14 is really using more memory, or if this surfaced some other issue with the 7.14 vendor libs

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

cscs-ci run

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

Did something change from #1024 as this was passing Buildkite CI - like we're pulling in some newer jll or so? Else these failure may be MI300 specific.

@simeonschaub

Copy link
Copy Markdown
Member Author

No, nothing should have changed. I think this might be MI300 specific

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

The PR amends the Buildkite yml, and we may need to port those changes to the CSCS CI one. I am checking it rn.

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

Ok, so the issue on MI300 was the fact we did not update the yml file. I am working on a fix and possibly testing twice, once using local ROCm install, once using the local install. Is running not only from artifact but also testing one local ROCm install something we want to have also in Buildkite?

@simeonschaub

Copy link
Copy Markdown
Member Author

Those buildkite changes should only really matter on 1.10 though, on 1.12 Julia should respect the sources entry in the Project.toml file.

We should definitely test both configurations, but not sure we need to do this in buildkite too, seeing how the runners are already struggling too keep up

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

Ok. Yeah so using the new preference way I can now select back to use local ROCm install on MI300 and CI passes. However, it fails when using the new approach at the first HIP stream creation which may require investigation.

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

With the ROCm 7.14 TheRock artifact bundle, the first hipStreamCreateWithPriority makes ROCclr JIT-compile its internal blit kernels, and that link fails on undefined __amd_streamOpsIncrement/__amd_streamOpsDecrement. Those are present in the bundle only as weak symbols in opencl.bc, which isn't in the HIP device-library link set. Then ROCclr reports the failure misleadingly as hipErrorOutOfMemory. I'll dig in further.

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

On gfx942 (MI300A) the downloaded ROCm 7.14 artifacts don't seem usable as every HIP application we tried fails at its first hipStreamCreateWithPriority with hipErrorOutOfMemory. It doesn't look like memory exhaustion: ROCclr builds its blit kernels on first stream creation, that link fails on undefined __amd_streamOpsIncrement/__amd_streamOpsDecrement, and the error appears to surface as OOM. Reproducible in plain C++ against the bundle with no Julia involved. Filed upstream as ROCm/TheRock#7426.

RDNA looks unaffected as therock-dist-linux-gfx110X-all-7.14.0.tar.gz works, with the same device-library layout, so it may be specific to the gfx94X bundle.

For this PR it mainly matters that an MI300 user taking the new default would get an error pointing at memory rather than at the cause. Setting the local preference works fine, but that isn't discoverable from the message. Maybe we should decide what to do is such cases as well in the future, make a clearer error message or possibly fall back to system install if one is discovered?

Note that JULIA_AMDGPU_DISABLE_ARTIFACTS has no more effect.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMDGPU.jl Benchmarks

Details
Benchmark suite Current: 4044922 Previous: a144289 Ratio
amdgpu/synchronization/context/device 495 ns 557.5 ns 0.89
amdgpu/synchronization/stream/blocking 210 ns 237.5 ns 0.88
amdgpu/synchronization/stream/nonblocking 272.5 ns 315 ns 0.87
array/accumulate/Float32/1d 79073.5 ns 75158.75 ns 1.05
array/accumulate/Float32/dims=1 267381.25 ns 274737 ns 0.97
array/accumulate/Float32/dims=1L 93661.25 ns 80966.25 ns 1.16
array/accumulate/Float32/dims=2 95941.25 ns 71113.5 ns 1.35
array/accumulate/Float32/dims=2L 2614305.25 ns 2757050.75 ns 0.95
array/accumulate/Int64/1d 89048.75 ns 78761 ns 1.13
array/accumulate/Int64/dims=1 248803.75 ns 245961 ns 1.01
array/accumulate/Int64/dims=1L 100539 ns 84621.25 ns 1.19
array/accumulate/Int64/dims=2 95601.25 ns 85651.25 ns 1.12
array/accumulate/Int64/dims=2L 2896379.75 ns 2891608 ns 1.00
array/broadcast 49895.75 ns 69148.5 ns 0.72
array/construct 2367.5 ns 2125 ns 1.11
array/copy 36045.5 ns 37400.5 ns 0.96
array/copyto!/cpu_to_gpu 89606.25 ns 111151.5 ns 0.81
array/copyto!/gpu_to_cpu 89856.25 ns 110589 ns 0.81
array/copyto!/gpu_to_gpu 35330.5 ns 59623.25 ns 0.59
array/iteration/findall/bool 145599.5 ns 139224.5 ns 1.05
array/iteration/findall/int 150267.25 ns 151857.25 ns 0.99
array/iteration/findfirst/bool 152199.5 ns 146044.5 ns 1.04
array/iteration/findfirst/int 151427.25 ns 145224.5 ns 1.04
array/iteration/findmin/1d 133194.5 ns 120931.75 ns 1.10
array/iteration/findmin/2d 117539.25 ns 109801.5 ns 1.07
array/iteration/logical 246368.5 ns 250738.75 ns 0.98
array/iteration/scalar 309707 ns 305062 ns 1.02
array/permutedims/2d 70841.25 ns 72388.75 ns 0.98
array/permutedims/3d 70813.5 ns 71686 ns 0.99
array/permutedims/4d 73796 ns 74573.5 ns 0.99
array/random/rand/Float32 44648.25 ns 44683.25 ns 1.00
array/random/rand/Int64 53858.25 ns 54945.75 ns 0.98
array/random/rand!/Float32 41770.5 ns 66086 ns 0.63
array/random/rand!/Int64 48458.25 ns 72483.5 ns 0.67
array/random/randn/Float32 73746.25 ns 81451.25 ns 0.91
array/random/randn!/Float32 53128.25 ns 81703.75 ns 0.65
array/reductions/mapreduce/Float32/1d 111206.75 ns 97053.75 ns 1.15
array/reductions/mapreduce/Float32/dims=1 93906.25 ns 88963.75 ns 1.06
array/reductions/mapreduce/Float32/dims=1L 841169.75 ns 837089 ns 1.00
array/reductions/mapreduce/Float32/dims=2 98561.25 ns 84591.25 ns 1.17
array/reductions/mapreduce/Float32/dims=2L 142839.5 ns 143217 ns 1.00
array/reductions/mapreduce/Int64/1d 110689.25 ns 98171.5 ns 1.13
array/reductions/mapreduce/Int64/dims=1 94164 ns 83696.25 ns 1.13
array/reductions/mapreduce/Int64/dims=1L 842907.5 ns 832586.75 ns 1.01
array/reductions/mapreduce/Int64/dims=2 96968.75 ns 84578.5 ns 1.15
array/reductions/mapreduce/Int64/dims=2L 144304.75 ns 146112 ns 0.99
array/reductions/reduce/Float32/1d 111094 ns 98861.5 ns 1.12
array/reductions/reduce/Float32/dims=1 93901.25 ns 90131.25 ns 1.04
array/reductions/reduce/Float32/dims=1L 841347.25 ns 830326.75 ns 1.01
array/reductions/reduce/Float32/dims=2 96741.5 ns 85723.75 ns 1.13
array/reductions/reduce/Float32/dims=2L 143304.5 ns 145169.5 ns 0.99
array/reductions/reduce/Int64/1d 110556.5 ns 98446.25 ns 1.12
array/reductions/reduce/Int64/dims=1 93753.75 ns 85671 ns 1.09
array/reductions/reduce/Int64/dims=1L 840887.5 ns 832411.5 ns 1.01
array/reductions/reduce/Int64/dims=2 95329 ns 84691 ns 1.13
array/reductions/reduce/Int64/dims=2L 144264.5 ns 145602 ns 0.99
array/reverse/1d 45588 ns 46438.25 ns 0.98
array/reverse/1dL 75613.5 ns 75023.75 ns 1.01
array/reverse/1dL_inplace 57756 ns 81443.75 ns 0.71
array/reverse/1d_inplace 38830.5 ns 61513.5 ns 0.63
array/reverse/2d 49998.25 ns 50780.75 ns 0.98
array/reverse/2dL 93809 ns 83493.5 ns 1.12
array/reverse/2dL_inplace 69841 ns 92721.25 ns 0.75
array/reverse/2d_inplace 41325.5 ns 43518 ns 0.95
array/sorting/1d 334464.75 ns 334237.5 ns 1.00
integration/byval/reference 38780 ns 39990 ns 0.97
integration/byval/slices=1 40331 ns 40521 ns 1.00
integration/byval/slices=2 150882 ns 149302 ns 1.01
integration/byval/slices=3 251424 ns 238314 ns 1.06
integration/volumerhs 5004363 ns 4992017 ns 1.00
kernel/indexing 35403 ns 48705.5 ns 0.73
kernel/indexing_checked 37050.5 ns 59730.75 ns 0.62
kernel/launch 1575 ns 1432.5 ns 1.10
kernel/rand 68986 ns 98344 ns 0.70
latency/import 1820629161 ns 1711844646 ns 1.06
latency/precompile 39503494228 ns 39815647264 ns 0.99
latency/ttfp 2400329001 ns 2332633440 ns 1.03

This comment was automatically generated by workflow using github-action-benchmark.

@evetion

evetion commented Aug 20, 2026

Copy link
Copy Markdown

Tested on my 6900XT. Only one failure, which is fixed in #1042, but some interesting warnings, like

Output generated during execution of 'hip_dnn/conv':
┌ MIOpen(HIP): Warning [ParseAndLoadDb] File is unreadable: "/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/share/miopen/db/gfx1030_40.HIP.fdb.txt"
│ MIOpen(HIP): Warning [OpenRuntimeLibraryForDevice] CK grouped conv library not found for device gfx1030: libMIOpenCKGroupedConv_gfx1030.so: cannot open shared object file: No such file or directory
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ MIOpen(HIP): Warning [IsEnoughWorkspace] [EvaluateInvokers] Solver <GemmFwdRest>, workspace required: 4032, provided ptr: 0 size: 0
└ MIOpen Error: localhost.localdomain:/__w/rockrel/rockrel/rocm-libraries/projects/miopen/src/convolution.cpp:249: Channels do not match for the filter

Full run:

Details

(AMDGPU) pkg> test
      Status `~/.julia/dev/AMDGPU/test/Project.toml`
  [21141c5a] AMDGPU v2.7.3 `.`
  [79e6a3ab] Adapt v4.7.0
  [ab4f0b2a] BFloat16s v0.6.1
  [7a1cc6ca] FFTW v1.10.0
  [1a297f60] FillArrays v1.17.0
  [f6369f11] ForwardDiff v1.4.5
  [0c68f7d7] GPUArrays v11.5.12
  [61eb1bfa] GPUCompiler v2.2.1
  [033835bb] JLD2 v0.6.5
  [63c18a36] KernelAbstractions v0.9.42
  [929cbde3] LLVM v9.13.0
  [d3525ed8] ParallelTestRunner v2.8.1
  [08abe8d2] PrettyTables v3.4.8
  [0bf8d700] ROCm_Runtime_Discovery v0.1.0 `ROCm_Runtime_Discovery`
  [a0a7dd2c] SparseMatricesCSR v0.6.12
  [276daf66] SpecialFunctions v2.9.0
  [90137ffa] StaticArrays v1.9.19
  [10745b16] Statistics v1.11.1
  [8ba89e20] Distributed v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [44cfe95a] Pkg v1.12.1
  [9a3f8284] Random v1.11.0
  [2f01184e] SparseArrays v1.12.0
  [fa267f1f] TOML v1.0.3
  [8dfed614] Test v1.11.0
      Status `~/.julia/dev/AMDGPU/Manifest.toml`
  [21141c5a] AMDGPU v2.7.3 `.`
  [621f4979] AbstractFFTs v1.5.0
  [6a4ca0a5] AcceleratedKernels v0.4.3
  [79e6a3ab] Adapt v4.7.0
  [dce04be8] ArgCheck v2.5.0
  [4fba245c] ArrayInterface v7.30.0
  [a9b6321e] Atomix v1.1.3
  [ab4f0b2a] BFloat16s v0.6.1
  [62783981] BitTwiddlingConvenienceFunctions v0.1.6
  [fa961155] CEnum v0.5.0
  [2a0fbf3d] CPUSummary v0.2.7
  [0b6fb165] ChunkCodecCore v1.0.1
  [4c0bbee4] ChunkCodecLibZlib v1.1.0
  [55437552] ChunkCodecLibZstd v1.0.0
  [fb6a15b2] CloseOpenIntervals v0.1.13
  [bbf7d656] CommonSubexpressions v0.3.1
  [f70d9fcc] CommonWorldInvalidations v1.1.2
  [34da2185] Compat v4.18.1
  [9db33cc3] CompilerCaching v0.4.2
  [adafc99b] CpuId v0.3.1
  [a8cc5b0e] Crayons v4.2.0
  [9a962f9c] DataAPI v1.16.0
  [e2d170a0] DataValueInterfaces v1.0.0
  [163ba53b] DiffResults v1.1.0
  [b552c78f] DiffRules v1.16.0
  [ffbed154] DocStringExtensions v0.9.5
  [e2ba6199] ExprTools v0.1.11
  [7a1cc6ca] FFTW v1.10.0
  [5789e2e9] FileIO v1.20.0
  [1a297f60] FillArrays v1.17.0
  [f6369f11] ForwardDiff v1.4.5
  [0c68f7d7] GPUArrays v11.5.12
  [46192b85] GPUArraysCore v0.2.0
  [61eb1bfa] GPUCompiler v2.2.1
  [096a3bc2] GPUToolbox v3.0.0
  [076d061b] HashArrayMappedTries v0.2.0
  [b5f81e59] IOCapture v1.0.0
  [615f187c] IfElse v0.1.1
  [92d709cd] IrrationalConstants v0.2.6
  [82899510] IteratorInterfaceExtensions v1.0.0
  [033835bb] JLD2 v0.6.5
  [692b3bcd] JLLWrappers v1.8.0
  [63c18a36] KernelAbstractions v0.9.42
  [929cbde3] LLVM v9.13.0
  [b964fa9f] LaTeXStrings v1.4.1
  [10f19ff3] LayoutPointers v0.1.17
  [2ab3a3ac] LogExpFunctions v1.0.1
  [1914dd2f] MacroTools v0.5.16
  [36869731] Malt v1.4.1
  [d125e4d3] ManualMemory v0.1.8
  [77ba4419] NaNMath v1.1.4
  [bac558e1] OrderedCollections v2.0.1
  [d3525ed8] ParallelTestRunner v2.8.1
  [f517fe37] Polyester v0.7.19
  [1d0040c9] PolyesterWeave v0.2.2
  [aea7be01] PrecompileTools v1.3.4
  [21216c6a] Preferences v1.5.2
  [08abe8d2] PrettyTables v3.4.8
  [3129f4d2] ROCm_Runtime v7.14.0 `ROCm_Runtime`
  [0bf8d700] ROCm_Runtime_Discovery v0.1.0 `ROCm_Runtime_Discovery`
  [74087812] Random123 v1.7.1
  [e6cf234a] RandomNumbers v1.6.0
  [189a3867] Reexport v1.2.2
  [05181044] RelocatableFolders v1.0.1
  [ae029012] Requires v1.3.1
  [94e857df] SIMDTypes v0.1.0
  [431bcebd] SciMLPublic v1.2.4
  [7e506255] ScopedValues v1.6.2
  [6c6a2e73] Scratch v1.3.0
  [a0a7dd2c] SparseMatricesCSR v0.6.12
  [276daf66] SpecialFunctions v2.9.0
  [aedffcd0] Static v1.4.6
  [0d7ed370] StaticArrayInterface v1.10.0
  [90137ffa] StaticArrays v1.9.19
  [1e83bf80] StaticArraysCore v1.4.4
  [10745b16] Statistics v1.11.1
  [7792a7ef] StrideArraysCore v0.5.9
  [892a3eda] StringManipulation v0.5.0
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.13.0
  [8290d209] ThreadingUtilities v0.5.6
  [e689c965] Tracy v0.1.6
  [013be700] UnsafeAtomics v0.3.2
  [cc5c0156] AMDGPU_LLVM_Backend_jll v22.1.8+2
  [f5851436] FFTW_jll v3.3.12+0
  [1d5cc7b8] IntelOpenMP_jll v2025.2.0+0
  [dad2f222] LLVMExtra_jll v0.0.46+0
  [ad6e5548] LibTracyClient_jll v0.13.1+0
  [856f044c] MKL_jll v2025.2.0+0
  [efe28fd5] OpenSpecFun_jll v0.5.6+0
⌅ [873c0968] ROCmDeviceLibs_jll v6.2.1+2
  [3161d3a3] Zstd_jll v1.5.7+1
  [1317d2d5] oneTBB_jll v2022.3.0+0
  [8b5cbfcf] tree_sitter_gcn_jll v0.1.0+0
  [44208993] tree_sitter_llvm_jll v1.1.0+0
  [71e3f6e6] tree_sitter_ptx_jll v0.1.0+0
  [f0e86581] tree_sitter_spirv_jll v0.1.0+0
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [4af54fe1] LazyArtifacts v1.11.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [a63ad114] Mmap v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [2f01184e] SparseArrays v1.12.0
  [f489334b] StyledStrings v1.11.0
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [4536629a] OpenBLAS_jll v0.3.29+0
  [05823500] OpenLibm_jll v0.8.7+0
  [458c3c95] OpenSSL_jll v3.5.4+0
  [bea87d4a] SuiteSparse_jll v7.8.3+2
  [83775a58] Zlib_jll v1.3.1+2
  [8e850b90] libblastrampoline_jll v5.15.0+0
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading.
Precompiling for configuration --code-coverage=none --color=yes --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none
Precompiling packages finished.
  3 dependencies successfully precompiled in 29 seconds. 89 already precompiled.
     Testing Running tests...
[ Info: System information:
Julia Version 1.12.6
Commit 15346901f00 (2026-04-09 19:20 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 24 × AMD Ryzen 9 5900X 12-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, znver3)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 24 virtual cores)
Environment:
  JULIA_LOAD_PATH = @:/home/evetion/.julia/dev/AMDGPU/test
AMDGPU versioninfo
ROCm provider: downloaded artifacts
┌───────────┬──────────────────┬────────────┬─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Available │ Name             │ Version    │ Path                                                                                        │
├───────────┼──────────────────┼────────────┼─────────────────────────────────────────────────────────────────────────────────────────────┤
│     +     │ LLD              │ -/home/evetion/.julia/artifacts/e21d5c5990876ecba40cb3d23b0b6fc6126709ec/bin/lld             │
│     +     │ Device Libraries │ -/home/evetion/.julia/artifacts/0a0cad46b7dacc66387ed19e8c0f45b097f46691/amdgcn/bitcode      │
│     +     │ HIP              │ 7.14.60850/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/libamdhip64.so  │
│     +     │ rocBLAS          │ 5.5.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocblas.so   │
│     +     │ rocSOLVER        │ 3.35.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocsolver.so │
│     +     │ rocSPARSE        │ 4.7.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocsparse.so │
│     +     │ rocRAND          │ 4.5.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocrand.so   │
│     +     │ rocFFT           │ 1.0.38/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocfft.so    │
│     +     │ MIOpen           │ 3.5.2/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/libMIOpen.so    │
└───────────┴──────────────────┴────────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘

AMDGPU devices
┌────┬───────────────────────┬──────────┬───────────┬────────────┬───────────────┐
│ Id │                  Name │ GCN arch │ Wavefront │     Memory │ Shared Memory │
├────┼───────────────────────┼──────────┼───────────┼────────────┼───────────────┤
│  1 │ AMD Radeon RX 6900 XT │  gfx1030 │        3215.984 GiB │    64.000 KiB │
└────┴───────────────────────┴──────────┴───────────┴────────────┴───────────────┘

┌ Info: Available tests:
│ core/codegen
│ core/core_tests
│ core/discovery_tests
│ core/graph_tests
│ core/rocarray_base
│ core/rocarray_broadcast
│ core/tls
│ device/array
│ device/exceptions
│ device/execution_control
│ device/hostcall
│ device/indexing
│ device/launch
│ device/math
│ device/memory
│ device/output
│ device/random
│ device/regression
│ device/synchronization
│ device/vadd
│ device/wavefront
│ enzyme_tests
│ external/forwarddiff
│ gpuarrays/alloc cache
│ gpuarrays/base
│ gpuarrays/broadcasting
│ gpuarrays/constructors
│ gpuarrays/ext/jld2
│ gpuarrays/indexing find
│ gpuarrays/indexing multidimensional
│ gpuarrays/indexing reshaped wrappers
│ gpuarrays/indexing scalar
│ gpuarrays/linalg/NaN_false
│ gpuarrays/linalg/core
│ gpuarrays/linalg/diagonal
│ gpuarrays/linalg/kron
│ gpuarrays/linalg/kron_diagonal
│ gpuarrays/linalg/mul!/integer-accumulate
│ gpuarrays/linalg/mul!/matrix-matrix
│ gpuarrays/linalg/mul!/mixed-eltype
│ gpuarrays/linalg/mul!/strided-views
│ gpuarrays/linalg/mul!/vector-matrix
│ gpuarrays/linalg/norm
│ gpuarrays/math/intrinsics
│ gpuarrays/math/power
│ gpuarrays/random
│ gpuarrays/reductions/== isequal
│ gpuarrays/reductions/and or xor
│ gpuarrays/reductions/any all count
│ gpuarrays/reductions/mapreduce
│ gpuarrays/reductions/mapreducedim!
│ gpuarrays/reductions/mapreducedim!_large
│ gpuarrays/reductions/minimum maximum extrema
│ gpuarrays/reductions/sum prod
│ gpuarrays/sparse
│ gpuarrays/statistics
│ gpuarrays/uniformscaling
│ gpuarrays/vectors
│ hip_core_tests
│ hip_dnn/activations
│ hip_dnn/batchnorm
│ hip_dnn/conv
│ hip_dnn/descriptors
│ hip_dnn/pool
│ hip_rocarray/blas
│ hip_rocarray/fft
│ hip_rocarray/indexing
│ hip_rocarray/multi_gpu
│ hip_rocarray/random
│ hip_rocarray/reverse
│ hip_rocarray/solver
│ hip_rocarray/sorting
│ hip_rocsparse/broadcast
│ hip_rocsparse/conversions
│ hip_rocsparse/device
│ hip_rocsparse/generic
│ hip_rocsparse/interfaces
│ hip_rocsparse/linalg
│ hip_rocsparse/preconditioners
│ hip_rocsparse/sparse_matrices_csr
│ kernelabstractions_tests
│ wmma_rdna3_tests
└ wmma_rdna4_tests
Running 80 tests using 24 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
                                                   │   Test   │ ──────────────── CPU ──────────────── │
Test                                      (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
core/codegen                                  (10) │    27.94 │   failed at 2026-08-20T14:48:47.469
gpuarrays/indexing find                       (23) │   116.572.001.75019.561403.18 │
gpuarrays/linalg/diagonal                     (20) │   142.702.231.65127.821750.55 │
gpuarrays/ext/jld2                            (21) │   162.812.161.35452.501330.11 │
gpuarrays/math/power                          (10) │   137.852.952.16862.151720.43 │
gpuarrays/reductions/mapreducedim!_large      (12) │   192.534.322.27224.991571.34 │
hip_rocarray/fft                              (18) │   193.712.921.54827.511706.62 │
device/wavefront                              (22) │   198.422.031.04563.271496.95 │
gpuarrays/constructors                        (24) │   216.902.821.36425.591405.97 │
gpuarrays/statistics                          (19) │   238.233.541.59063.441969.02 │
hip_rocarray/sorting                          (17) │   253.053.271.37987.501696.25 │
gpuarrays/reductions/== isequal               (16) │   256.343.451.38920.671576.09 │
hip_rocsparse/broadcast                       (23) │   134.701.210.95681.462215.89 │
gpuarrays/indexing multidimensional           (11) │   278.193.691.39617.091806.81 │
hip_rocsparse/generic                         (20) │   142.761.371.03640.912205.06 │
gpuarrays/base                                (21) │   130.612.471.95643.061905.07 │
hip_rocsparse/interfaces                      (13) │   304.453.981.38886.391916.57 │
kernelabstractions_tests                      (10) │   133.572.632.05293.042209.82 │
core/graph_tests                              (21) │    11.620.080.7472.331957.14 │
gpuarrays/indexing reshaped wrappers          (22) │   118.151.651.45139.431823.52 │
gpuarrays/linalg/mul!/strided-views           (19) │    80.150.891.13870.192283.23 │
gpuarrays/random                              (24) │   102.710.730.73069.241755.69 │
device/random                                 (18) │   147.031.451.03695.302031.11 │
hip_rocsparse/linalg                          (17) │    99.700.720.73100.952351.42 │
hip_rocarray/reverse                          (24) │    30.280.351.21297.201888.52 │
gpuarrays/reductions/and or xor               (16) │   100.600.780.83133.901845.16 │
core/rocarray_broadcast                       (13) │    53.420.490.92077.332135.75 │
external/forwarddiff                          (22) │    37.080.962.61595.591969.10 │
hip_rocarray/random                           (19) │    41.020.280.71218.512530.88 │
gpuarrays/linalg/mul!/vector-matrix           (12) │   178.581.821.05721.892236.48 │
core/rocarray_base                            (21) │    52.220.340.61802.452369.13 │
gpuarrays/linalg/NaN_false                    (20) │    81.870.610.72726.322340.18 │
hip_rocsparse/preconditioners                 (23) │   117.180.510.42539.152353.46 │
hip_rocsparse/conversions                     (13) │    15.650.070.5519.122284.49 │
core/core_tests                               (22) │    15.500.000.018.541969.29 │
gpuarrays/linalg/mul!/mixed-eltype            (11) │   107.091.791.75095.982074.33 │
gpuarrays/reductions/any all count            (18) │    40.230.320.81473.762152.79 │
hip_rocsparse/sparse_matrices_csr             (24) │    25.730.110.4496.552273.71 │
gpuarrays/indexing scalar                     (10) │    78.970.370.51766.172371.20 │
gpuarrays/linalg/kron_diagonal                (12) │    15.540.080.5398.312410.23 │
gpuarrays/linalg/mul!/integer-accumulate      (19) │    21.870.160.7818.132614.25 │
device/math                                   (17) │    35.370.210.61077.952463.01 │
gpuarrays/math/intrinsics                     (20) │    16.200.070.5572.882480.91 │
hip_dnn/batchnorm                             (21) │    18.790.000.0232.182758.63 │
device/launch                                 (13) │    12.100.070.6413.112343.81 │
hip_rocarray/indexing                         (23) │    16.000.100.6570.912493.31 │
device/exceptions                             (18) │     8.660.000.0314.282236.98 │
gpuarrays/uniformscaling                      (16) │    39.230.250.61063.512017.14 │
device/memory                                 (11) │    12.460.000.0267.542111.79 │
gpuarrays/alloc cache                         (21) │     1.880.000.092.692758.63 │
device/synchronization                        (12) │     7.300.000.0185.972410.23 │
hip_rocsparse/device                          (19) │     7.500.000.074.282708.81 │
device/execution_control                      (13) │     4.470.000.0134.852343.81 │
device/array                                  (20) │     5.850.000.0173.092480.91 │
core/discovery_tests                          (18) │     1.180.000.045.872236.98 │
hip_dnn/descriptors                           (10) │    10.820.000.0192.492371.20 │
device/indexing                               (23) │     5.870.000.0105.312493.31 │
device/vadd                                   (16) │     3.960.000.0115.752017.14 │
core/tls                                      (11) │     2.870.000.0132.392239.95 │
hip_core_tests                                (21) │     2.100.000.073.392758.63 │
wmma_rdna4_tests                              (13) │     0.120.000.01.542343.81 │
gpuarrays/vectors                             (19) │     0.820.000.023.552721.56 │
hip_rocarray/multi_gpu                        (20) │     0.210.000.01.542480.91 │
device/regression                             (24) │    20.950.100.5515.742427.00 │
wmma_rdna3_tests                              (18) │     0.550.000.023.322236.98 │
gpuarrays/sparse                              (10) │     0.020.000.01.062371.20 │
gpuarrays/linalg/mul!/matrix-matrix            (6) │   428.644.921.112205.982186.73 │
hip_rocarray/solver                            (4) │   428.785.971.414613.912703.00 │
hip_dnn/activations                           (17) │    23.190.000.0306.902933.34 │
hip_dnn/conv                                  (22) │    41.560.070.2464.752917.51 │
gpuarrays/linalg/kron                         (14) │   438.369.092.114433.012193.75 │
hip_dnn/pool                                  (12) │    23.440.000.096.362723.57 │
gpuarrays/reductions/mapreduce                 (5) │   449.505.521.213483.011973.34 │
gpuarrays/reductions/sum prod                  (2) │   488.366.541.316708.232367.82 │
gpuarrays/reductions/mapreducedim!             (1) │   497.156.921.418441.332449.36 │
hip_rocarray/blas                              (3) │   514.969.051.825136.482838.43 │
gpuarrays/reductions/minimum maximum extrema   (7) │   533.728.041.523260.852736.15 │
gpuarrays/linalg/core                         (15) │   535.899.391.826925.573194.47 │
gpuarrays/linalg/norm                          (8) │   583.768.381.428219.413381.32 │
gpuarrays/broadcasting                         (9) │   645.0410.691.735856.753771.40 │

Output generated during execution of 'gpuarrays/linalg/diagonal':
┌ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::Tuple{Int64, Int64}, ::Int64, ::UnitRange{Int64}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
└ └ Otherwise, performance might degrade if they keep running in the background.

Output generated during execution of 'gpuarrays/indexing multidimensional':
┌ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Int16, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Int16, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Int16, 1}, ::AMDGPU.Device.ROCDeviceArray{Int16, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Int32, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Int32, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Int32, 1}, ::AMDGPU.Device.ROCDeviceArray{Int32, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Int64, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Int64, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Int64, 1}, ::AMDGPU.Device.ROCDeviceArray{Int64, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Float16, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float16, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float16, 1}, ::AMDGPU.Device.ROCDeviceArray{Float16, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Float32, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::AMDGPU.Device.ROCDeviceArray{Float32, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Float64, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::AMDGPU.Device.ROCDeviceArray{Float64, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{ComplexF16, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF16, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF16, 1}, ::AMDGPU.Device.ROCDeviceArray{ComplexF16, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{ComplexF32, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF32, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF32, 1}, ::AMDGPU.Device.ROCDeviceArray{ComplexF32, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{ComplexF64, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF64, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF64, 1}, ::AMDGPU.Device.ROCDeviceArray{ComplexF64, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int16}, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int16}, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int16}, 1}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int16}, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int32}, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int32}, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int32}, 1}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int32}, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int64}, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int64}, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int64}, 1}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int64}, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::Tuple{Int64, Int64}, ::Int64, ::Int64, ::Base.Slice{Base.OneTo{Int64}})
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::LinearAlgebra.Adjoint{Float32, AMDGPU.Device.ROCDeviceMatrix{Float32, 1}}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
└ └ Otherwise, performance might degrade if they keep running in the background.

Output generated during execution of 'kernelabstractions_tests':
┌ Precompiling packages...6826.4 ms  ✓ AMDGPU  AMDGPUSpecialFunctionsExt
│   1 dependency successfully precompiled in 7 seconds. 98 already precompiled.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float64, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
└ [ Info: Stopped global hostcall: `malloc_hostcall`.

Output generated during execution of 'hip_rocsparse/sparse_matrices_csr':
┌ Precompiling packages...7155.6 ms  ✓ ArrayInterface  ArrayInterfaceAMDGPUExt
│   1 dependency successfully precompiled in 7 seconds. 96 already precompiled.
│ Precompiling packages...4694.5 ms  ✓ AMDGPU  AMDGPUSparseMatricesCSRExt
└   1 dependency successfully precompiled in 5 seconds. 118 already precompiled.

Output generated during execution of 'device/launch':
┌ ┌ Warning: Only 1 GPU detected; skipping multi-GPU tests
└ └ @ Main.var"##device/launch#1208" ~/.julia/dev/AMDGPU/test/device/launch.jl:91

Output generated during execution of 'core/discovery_tests':
┌ Warning: HIP_PLATFORM=hcc is deprecated.Please use HIP_PLATFORM=amd.
└ Warning: HIP_PLATFORM=hcc is deprecated.Please use HIP_PLATFORM=amd.

Output generated during execution of 'wmma_rdna4_tests':
[ [ Info: Skipping WMMA_RDNA4 tests (requires RDNA4+ / gfx1200+)

Output generated during execution of 'hip_rocarray/multi_gpu':
[ [ Info: Skipping Multi-GPU tests (requires more than 1 GPU)

Output generated during execution of 'wmma_rdna3_tests':
[ [ Info: Skipping WMMA_RDNA3 tests (requires RDNA3+)

Output generated during execution of 'hip_rocarray/solver':
┌ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float64, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{ComplexF32, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF32, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{ComplexF64, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF64, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
└ └ Otherwise, performance might degrade if they keep running in the background.

Output generated during execution of 'hip_dnn/conv':
┌ MIOpen(HIP): Warning [ParseAndLoadDb] File is unreadable: "/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/share/miopen/db/gfx1030_40.HIP.fdb.txt"MIOpen(HIP): Warning [OpenRuntimeLibraryForDevice] CK grouped conv library not found for device gfx1030: libMIOpenCKGroupedConv_gfx1030.so: cannot open shared object file: No such file or directory
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ MIOpen(HIP): Warning [IsEnoughWorkspace] [EvaluateInvokers] Solver <GemmFwdRest>, workspace required: 4032, provided ptr: 0 size: 0
└ MIOpen Error: localhost.localdomain:/__w/rockrel/rockrel/rocm-libraries/projects/miopen/src/convolution.cpp:249: Channels do not match for the filter

Test Summary:                                    |  Pass  Fail  Broken  Total      Time
  Overall                                        | 16857     1      16  16874  11m07.9s
    core/codegen                                 |     3     1              4     27.8s
      Synchronization                            |     1                    1     24.5s
      Trapping                                   |     2                    2      1.7s
      Hardware FP atomics                        |           1              1      1.4s
    gpuarrays/indexing find                      |    67                   67   1m56.4s
    gpuarrays/linalg/diagonal                    |    82                   82   2m22.5s
    gpuarrays/ext/jld2                           |   108                  108   2m42.6s
    gpuarrays/math/power                         |    72                   72   2m17.8s
    gpuarrays/reductions/mapreducedim!_large     |    50                   50   3m12.4s
    hip_rocarray/fft                             |   261                  261   3m13.5s
    device/wavefront                             |    83             9     92   3m18.2s
    gpuarrays/constructors                       |   992                  992   3m36.7s
    gpuarrays/statistics                         |    78                   78   3m58.0s
    hip_rocarray/sorting                         |   384                  384   4m12.9s
    gpuarrays/reductions/== isequal              |   290                  290   4m16.2s
    hip_rocsparse/broadcast                      |    65                   65   2m14.7s
    gpuarrays/indexing multidimensional          |   101                  101   4m38.0s
    hip_rocsparse/generic                        |   596                  596   2m22.8s
    gpuarrays/base                               |   151                  151   2m10.6s
    hip_rocsparse/interfaces                     |   625                  625   5m04.3s
    kernelabstractions_tests                     |  2236             6   2242   2m13.6s
    core/graph_tests                             |    12                   12     11.6s
    gpuarrays/indexing reshaped wrappers         |    42                   42   1m58.2s
    gpuarrays/linalg/mul!/strided-views          |    36                   36   1m20.1s
    gpuarrays/random                             |    62                   62   1m42.7s
    device/random                                |   312                  312   2m27.0s
    hip_rocsparse/linalg                         |   168                  168   1m39.7s
    hip_rocarray/reverse                         |    88                   88     30.3s
    gpuarrays/reductions/and or xor              |    99                   99   1m40.6s
    core/rocarray_broadcast                      |    23                   23     53.4s
    external/forwarddiff                         |    18                   18     37.1s
    hip_rocarray/random                          |   141                  141     41.0s
    gpuarrays/linalg/mul!/vector-matrix          |   168                  168   2m58.6s
    core/rocarray_base                           |    74                   74     52.2s
    gpuarrays/linalg/NaN_false                   |    54                   54   1m21.9s
    hip_rocsparse/preconditioners                |   312                  312   1m57.2s
    hip_rocsparse/conversions                    |    34                   34     15.6s
    core/core_tests                              |    27                   27     15.5s
    gpuarrays/linalg/mul!/mixed-eltype           |    60                   60   1m47.1s
    gpuarrays/reductions/any all count           |   101                  101     40.2s
    hip_rocsparse/sparse_matrices_csr            |    53                   53     25.7s
    gpuarrays/indexing scalar                    |   477                  477   1m19.0s
    gpuarrays/linalg/kron_diagonal               |    16                   16     15.5s
    gpuarrays/linalg/mul!/integer-accumulate     |     6                    6     21.9s
    device/math                                  |    27                   27     35.4s
    gpuarrays/math/intrinsics                    |    12                   12     16.2s
    hip_dnn/batchnorm                            |    12                   12     18.8s
    device/launch                                |    11             1     12     12.1s
    hip_rocarray/indexing                        |     3                    3     16.0s
    device/exceptions                            |     6                    6      8.7s
    gpuarrays/uniformscaling                     |    56                   56     39.2s
    device/memory                                |     7                    7     12.5s
    gpuarrays/alloc cache                        |    37                   37      1.9s
    device/synchronization                       |   176                  176      7.3s
    hip_rocsparse/device                         |    10                   10      7.5s
    device/execution_control                     |     5                    5      4.5s
    device/array                                 |    22                   22      5.9s
    core/discovery_tests                         |    14                   14      1.2s
    hip_dnn/descriptors                          |   125                  125     10.8s
    device/indexing                              |     2                    2      5.9s
    device/vadd                                  |     1                    1      4.0s
    core/tls                                     |    13                   13      2.9s
    hip_core_tests                               |     4                    4      2.1s
    wmma_rdna4_tests                             |                          0      0.1s
    gpuarrays/vectors                            |    10                   10      0.8s
    hip_rocarray/multi_gpu                       |                          0      0.2s
    device/regression                            |     3                    3     20.9s
    wmma_rdna3_tests                             |                          0      0.6s
    gpuarrays/sparse                             |                          0      0.0s
    gpuarrays/linalg/mul!/matrix-matrix          |   444                  444   7m08.5s
    hip_rocarray/solver                          |  1089                 1089   7m08.6s
    hip_dnn/activations                          |    10                   10     23.2s
    hip_dnn/conv                                 |    27                   27     41.6s
    gpuarrays/linalg/kron                        |   192                  192   7m18.2s
    hip_dnn/pool                                 |                          0     23.4s
    gpuarrays/reductions/mapreduce               |   900                  900   7m29.3s
    gpuarrays/reductions/sum prod                |   526                  526   8m08.2s
    gpuarrays/reductions/mapreducedim!           |   576                  576   8m17.0s
    hip_rocarray/blas                            |  1040                 1040   8m34.8s
    gpuarrays/reductions/minimum maximum extrema |   414                  414   8m53.6s
    gpuarrays/linalg/core                        |  1120                 1120   8m55.7s
    gpuarrays/linalg/norm                        |   876                  876   9m43.6s
    gpuarrays/broadcasting                       |   460                  460  10m44.9s
    FAILURE

Error in testset Hardware FP atomics:
Test Failed at /home/evetion/.julia/dev/AMDGPU/test/core/codegen.jl:47
  Expression: occursin("global_atomic_add_$(fp)", String(take!(iob)))
   Evaluated: occursin("global_atomic_add_f32", "\t.amdgcn_target \"amdgcn-amd-amdhsa--gfx1030\"\n\t.amdhsa_code_object_version 6\n\t.text\n\t.globl\t_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE ; -- Begin function _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n\t.p2align\t8\n\t.type\t_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE,@function\n_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE: ; @_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n; %bb.0:                                ; %conversion\n\ts_add_u32 s12, s12, s17\n\ts_addc_u32 s13, s13, 0\n\ts_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12\n\ts_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13\n\ts_mov_b64 s[22:23], s[2:3]\n\ts_mov_b64 s[20:21], s[0:1]\n\ts_load_dwordx2 s[0:1], s[8:9], 0x28\n\ts_add_u32 s20, s20, s17\n\ts_addc_u32 s21, s21, 0\n\ts_waitcnt lgkmcnt(0)\n\ts_cmp_lg_u64 s[0:1], 0\n\ts_cbranch_scc0 .LBB0_6\n; %bb.1:                                ; %L14\n\ts_mov_b32 s4, exec_lo\n\ts_mov_b32 s2, 0\n\tv_mbcnt_lo_u32_b32 v0, s4, 0\n\ts_mov_b32 s3, exec_lo\n\tv_cmpx_eq_u32_e32 0, v0\n\ts_cbranch_execz .LBB0_4\n; %bb.2:\n\ts_load_dwordx2 s[0:1], s[8:9], 0x30\n\ts_bcnt1_i32_b32 s4, s4\n\tv_mov_b32_e32 v3, 0\n\tv_cvt_f32_ubyte0_e32 v2, s4\n\ts_mov_b32 s4, 0\n\ts_waitcnt lgkmcnt(0)\n\ts_load_dword s5, s[0:1], 0x0\n\ts_waitcnt lgkmcnt(0)\n\tv_mov_b32_e32 v1, s5\n.LBB0_3:                                ; %atomicrmw.start\n                                        ; =>This Inner Loop Header: Depth=1\n\tv_add_f32_e32 v0, v1, v2\n\tglobal_atomic_cmpswap v0, v3, v[0:1], s[0:1] glc\n\ts_waitcnt vmcnt(0)\n\tbuffer_gl1_inv\n\tbuffer_gl0_inv\n\tv_cmp_eq_u32_e32 vcc_lo, v0, v1\n\tv_mov_b32_e32 v1, v0\n\ts_or_b32 s4, vcc_lo, s4\n\ts_andn2_b32 exec_lo, exec_lo, s4\n\ts_cbranch_execnz .LBB0_3\n.LBB0_4:                                ; %Flow3\n\ts_or_b32 exec_lo, exec_lo, s3\n\ts_andn2_b32 vcc_lo, exec_lo, s2\n\ts_cbranch_vccz .LBB0_7\n; %bb.5:                                ; %UnifiedReturnBlock\n\ts_endpgm\n.LBB0_6:\n.LBB0_7:                                ; %L10\n\ts_load_dwordx2 s[0:1], s[8:9], 0x0\n\ts_lshl_b32 s3, s14, 16\n\ts_mov_b32 s2, 0\n\ts_add_i32 s15, s15, 1\n\ts_mov_b64 s[4:5], src_private_base\n\ts_lshl_b32 s4, s16, 16\n\ts_add_i32 s3, s3, 0x10000\n\ts_and_b32 s7, s15, 0xffff\n\ts_mov_b32 s6, s2\n\ts_add_i32 s4, s4, 0x10001\n\ts_or_b64 s[2:3], s[6:7], s[2:3]\n\ts_waitcnt lgkmcnt(0)\n\ts_cmp_eq_u32 s1, s5\n\ts_cselect_b32 s2, -1, 0\n\ts_andn2_b32 vcc_lo, exec_lo, s2\n\ts_mov_b32 s2, -1\n\ts_cbranch_vccz .LBB0_9\n; %bb.8:                                ; %atomicrmw.global\n\tv_mov_b32_e32 v2, 0\n\tv_mov_b32_e32 v5, s1\n\tv_mov_b32_e32 v0, s4\n\tv_mov_b32_e32 v1, s3\n\tv_mov_b32_e32 v4, s0\n\tv_mov_b32_e32 v3, v2\n\ts_mov_b32 s2, 0\n\tflat_atomic_cmpswap_x2 v[4:5], v[0:3]\n\ts_waitcnt lgkmcnt(0)\n\ts_waitcnt_vscnt null, 0x0\n\tbuffer_gl1_inv\n\tbuffer_gl0_inv\n.LBB0_9:                                ; %Flow4\n\ts_andn2_b32 vcc_lo, exec_lo, s2\n\ts_cbranch_vccnz .LBB0_11\n; %bb.10:                               ; %atomicrmw.private\n\ts_cmp_lg_u64 s[0:1], 0\n\ts_cselect_b32 s0, s0, -1\n\tv_mov_b32_e32 v2, s0\n\ts_clause 0x1\n\tbuffer_load_dword v0, v2, s[20:23], 0 offen\n\tbuffer_load_dword v1, v2, s[20:23], 0 offen offset:4\n\ts_waitcnt vmcnt(0)\n\tv_cmp_eq_u64_e32 vcc_lo, 0, v[0:1]\n\tv_cndmask_b32_e64 v0, v0, s4, vcc_lo\n\tv_cndmask_b32_e64 v1, v1, s3, vcc_lo\n\tbuffer_store_dword v0, v2, s[20:23], 0 offen\n\tbuffer_store_dword v1, v2, s[20:23], 0 offen offset:4\n.LBB0_11:                               ; %atomicrmw.phi\n\ts_endpgm\n\t; divergent unreachable\n\ts_endpgm\n\t.section\t.rodata,\"a\",@progbits\n\t.p2align\t6, 0x0\n\t.amdhsa_kernel _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n\t\t.amdhsa_group_segment_fixed_size 0\n\t\t.amdhsa_private_segment_fixed_size 0\n\t\t.amdhsa_kernarg_size 320\n\t\t.amdhsa_user_sgpr_count 14\n\t\t.amdhsa_user_sgpr_private_segment_buffer 1\n\t\t.amdhsa_user_sgpr_dispatch_ptr 1\n\t\t.amdhsa_user_sgpr_queue_ptr 1\n\t\t.amdhsa_user_sgpr_kernarg_segment_ptr 1\n\t\t.amdhsa_user_sgpr_dispatch_id 1\n\t\t.amdhsa_user_sgpr_flat_scratch_init 1\n\t\t.amdhsa_user_sgpr_private_segment_size 0\n\t\t.amdhsa_wavefront_size32 1\n\t\t.amdhsa_uses_dynamic_stack 0\n\t\t.amdhsa_system_sgpr_private_segment_wavefront_offset 0\n\t\t.amdhsa_system_sgpr_workgroup_id_x 1\n\t\t.amdhsa_system_sgpr_workgroup_id_y 1\n\t\t.amdhsa_system_sgpr_workgroup_id_z 1\n\t\t.amdhsa_system_sgpr_workgroup_info 0\n\t\t.amdhsa_system_vgpr_workitem_id 2\n\t\t.amdhsa_next_free_vgpr 6\n\t\t.amdhsa_next_free_sgpr 24\n\t\t.amdhsa_reserve_vcc 1\n\t\t.amdhsa_reserve_flat_scratch 1\n\t\t.amdhsa_float_round_mode_32 0\n\t\t.amdhsa_float_round_mode_16_64 0\n\t\t.amdhsa_float_denorm_mode_32 3\n\t\t.amdhsa_float_denorm_mode_16_64 3\n\t\t.amdhsa_dx10_clamp 1\n\t\t.amdhsa_ieee_mode 1\n\t\t.amdhsa_fp16_overflow 0\n\t\t.amdhsa_workgroup_processor_mode 1\n\t\t.amdhsa_memory_ordered 1\n\t\t.amdhsa_forward_progress 1\n\t\t.amdhsa_shared_vgpr_count 0\n\t\t.amdhsa_exception_fp_ieee_invalid_op 0\n\t\t.amdhsa_exception_fp_denorm_src 0\n\t\t.amdhsa_exception_fp_ieee_div_zero 0\n\t\t.amdhsa_exception_fp_ieee_overflow 0\n\t\t.amdhsa_exception_fp_ieee_underflow 0\n\t\t.amdhsa_exception_fp_ieee_inexact 0\n\t\t.amdhsa_exception_int_div_zero 0\n\t.end_amdhsa_kernel\n\t.text\n.Lfunc_end0:\n\t.size\t_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE, .Lfunc_end0-_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n                                        ; -- End function\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.num_vgpr, 6\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.num_agpr, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.numbered_sgpr, 24\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.num_named_barrier, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.private_seg_size, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.uses_vcc, 1\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.uses_flat_scratch, 1\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.has_dyn_sized_stack, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.has_recursion, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.has_indirect_call, 0\n\t.section\t.AMDGPU.csdata,\"\",@progbits\n; Kernel info:\n; codeLenInByte = 424\n; TotalNumSgprs: 26\n; NumVgprs: 6\n; ScratchSize: 0\n; MemoryBound: 0\n; FloatMode: 240\n; IeeeMode: 1\n; LDSByteSize: 0 bytes/workgroup (compile time only)\n; SGPRBlocks: 0\n; VGPRBlocks: 0\n; NumSGPRsForWavesPerEU: 26\n; NumVGPRsForWavesPerEU: 6\n; Occupancy: 16\n; WaveLimiterHint : 0\n; COMPUTE_PGM_RSRC2:SCRATCH_EN: 0\n; COMPUTE_PGM_RSRC2:USER_SGPR: 14\n; COMPUTE_PGM_RSRC2:TRAP_HANDLER: 0\n; COMPUTE_PGM_RSRC2:TGID_X_EN: 1\n; COMPUTE_PGM_RSRC2:TGID_Y_EN: 1\n; COMPUTE_PGM_RSRC2:TGID_Z_EN: 1\n; COMPUTE_PGM_RSRC2:TIDIG_COMP_CNT: 2\n\t.text\n\t.p2alignl 6, 3214868480\n\t.fill 48, 4, 3214868480\n\t.section\t.AMDGPU.gpr_maximums,\"\",@progbits\n\t.set amdgpu.max_num_vgpr, 0\n\t.set amdgpu.max_num_agpr, 0\n\t.set amdgpu.max_num_sgpr, 0\n\t.set amdgpu.max_num_named_barrier, 0\n\t.text\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.section\t\".note.GNU-stack\",\"\",@progbits\n\t.amdgpu_metadata\n---\namdhsa.kernels:\n  - .args:\n      - .name:           state\n        .offset:         0\n        .size:           40\n        .value_kind:     by_value\n      - .name:           'x::ROCDeviceArray'\n        .offset:         40\n        .size:           24\n        .value_kind:     by_value\n      - .offset:         64\n        .size:           4\n        .value_kind:     hidden_block_count_x\n      - .offset:         68\n        .size:           4\n        .value_kind:     hidden_block_count_y\n      - .offset:         72\n        .size:           4\n        .value_kind:     hidden_block_count_z\n      - .offset:         76\n        .size:           2\n        .value_kind:     hidden_group_size_x\n      - .offset:         78\n        .size:           2\n        .value_kind:     hidden_group_size_y\n      - .offset:         80\n        .size:           2\n        .value_kind:     hidden_group_size_z\n      - .offset:         82\n        .size:           2\n        .value_kind:     hidden_remainder_x\n      - .offset:         84\n        .size:           2\n        .value_kind:     hidden_remainder_y\n      - .offset:         86\n        .size:           2\n        .value_kind:     hidden_remainder_z\n      - .offset:         104\n        .size:           8\n        .value_kind:     hidden_global_offset_x\n      - .offset:         112\n        .size:           8\n        .value_kind:     hidden_global_offset_y\n      - .offset:         120\n        .size:           8\n        .value_kind:     hidden_global_offset_z\n      - .offset:         128\n        .size:           2\n        .value_kind:     hidden_grid_dims\n      - .offset:         144\n        .size:           8\n        .value_kind:     hidden_hostcall_buffer\n      - .offset:         152\n        .size:           8\n        .value_kind:     hidden_multigrid_sync_arg\n      - .offset:         160\n        .size:           8\n        .value_kind:     hidden_heap_v1\n      - .offset:         168\n        .size:           8\n        .value_kind:     hidden_default_queue\n      - .offset:         176\n        .size:           8\n        .value_kind:     hidden_completion_action\n      - .offset:         264\n        .size:           8\n        .value_kind:     hidden_queue_ptr\n    .group_segment_fixed_size: 0\n    .kernarg_segment_align: 8\n    .kernarg_segment_size: 320\n    .language:       OpenCL C\n    .language_version:\n      - 2\n      - 0\n    .max_flat_workgroup_size: 1024\n    .name:           _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n    .private_segment_fixed_size: 0\n    .sgpr_count:     26\n    .sgpr_spill_count: 0\n    .symbol:         _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.kd\n    .uses_dynamic_stack: false\n    .vgpr_count:     6\n    .vgpr_spill_count: 0\n    .wavefront_size: 32\n    .workgroup_processor_mode: 1\namdhsa.target:   amdgcn-amd-amdhsa--gfx1030\namdhsa.version:\n  - 1\n  - 2\n...\n\n\t.end_amdgpu_metadata\n")

ERROR: LoadError: Test run finished with errors
in expression starting at /home/evetion/.julia/dev/AMDGPU/test/runtests.jl:103
ERROR: Package AMDGPU errored during testing

@simeonschaub

Copy link
Copy Markdown
Member Author

Thanks for the report! These warnings should all be harmless, MIOpen just has quite a verbose log level by default and the tests still pass, so I don't think there's anything for us to do here. Some of these warnings are already tracked in upstream. I'll attach the robot's analysis in case you're curious:

Breakdown 🤖
  1. MIOpen Error: … convolution.cpp:249: Channels do not match for the filter

Not a failure. It comes from the last testset in test/hip_dnn/conv.jl:86-91 ("Check MIOpenException"), which deliberately passes a 1‑channel filter against a 3‑channel input and asserts MIOpenException is thrown. MIOpen logs the
MIOPEN_THROW message to stderr before returning the bad-param status. The test passes; the log line is cosmetic. Could be worth a comment in the test so reporters don't mistake it for a failure.

  1. File is unreadable: …/share/miopen/db/gfx1030_40.HIP.fdb.txt

Upstream/expected. MIOpen ships no system find-db for RDNA parts in the TheRock builds; it JIT-tunes and writes a user db to ~/.config/miopen. An AMD maintainer confirmed this in ROCm/TheRock#5105 (closed): "The Warning [ParseAndLoadDb]
File is unreadable warning is a little misleading here as we don't expect the db files to be there." First-run convs will be slow; subsequent runs use the cached user db.

  1. CK grouped conv library not found for device gfx1030: libMIOpenCKGroupedConv_gfx1030.so

Upstream TheRock packaging/MIOpen. MIOpen 7.x dlopens a per-arch CK grouped-conv library; TheRock doesn't build one for gfx103x (CK's grouped conv needs MFMA/WMMA), so it falls back to GemmFwdRest. Same warning is tracked for other
archs:

No gfx1030-specific issue exists; #7409 is the closest to "please ship/suppress this for archs without CK support".

  1. warning: xnack 'Off' was requested for a processor that does not support it! (×4)

Upstream MIOpen. In src/comgr.cpp, under WORKAROUND_ISSUE_3001, MIOpen appends -mno-xnack when assembling its .s kernels (the 4 repeats = 4 asm kernel builds) whenever the target doesn't have xnack enabled — including gfx1030, which
doesn't support xnack at all, hence LLVM's warning. Tracked in ROCm/rocm-libraries#6812 (open, "MIOpen still sends old/removed -mno-xnack argument to clang"), continuation of ROCm/MIOpen#2851. Harmless for us (it's a warning, not an
error).

  1. [IsEnoughWorkspace] Solver , workspace required: 4032, provided ptr: 0 size: 0

Upstream MIOpen behavior. We do allocate miopenConvolutionForwardGetWorkSpaceSize bytes before Find (src/dnn/convolution.jl:103-104), but with no find-db the size query and the Find-time solver list don't agree, so a solver gets skipped
with this warning. Identical warnings (same solver, ptr 0) are reported from PyTorch in ROCm/rocm-libraries#4071 (open) and ROCm/MIOpen#3957 (open); MIOPEN_FIND_MODE=FAST suppresses them.

Caveat: I couldn't reproduce (no GPU in this sandbox); the analysis is from the MIOpen source in rocm-libraries/develop and the issue trackers.

If you want to quiet this for users, the cheap option is setting MIOPEN_LOG_LEVEL=2 (errors only; default release level is 3 = warnings) in the test runner or documenting it — the xnack line is an LLVM warning and wouldn't be affected
by that, though. I can add a comment to the Check MIOpenException test and/or an env default if you'd like.

@luraess

luraess commented Aug 22, 2026

Copy link
Copy Markdown
Member

I'll need to rework slightly the Bk pipeline now that #1045 landed.

@simeonschaub

Copy link
Copy Markdown
Member Author

I can also do it, I need to add hiptensor to this anyways

simeonschaub and others added 12 commits August 22, 2026 17:31
…ry subpackages

By default the ROCm runtime (HIP, HSA) and vendor libraries (rocBLAS,
rocSPARSE, rocSOLVER, rocRAND, rocFFT, hipTENSOR, MIOpen) now come from the
new ROCm_Runtime subpackage, which ships AMD's TheRock distribution tarballs
directly as lazy artifacts, selected by GPU architecture and ROCm version
through platform augmentation. The former src/discovery/ code moves into the
ROCm_Runtime_Discovery subpackage, which replicates ROCm_Runtime's API for a
local ROCm installation; it is only loaded when that is requested through
ROCm_Runtime's "local" preference, settable via the new
AMDGPU.set_rocm_version!([version]; local_rocm) (mirroring
CUDA.set_runtime_version!).

hipTENSOR is part of the same TheRock tarballs (only for the CDNA bundles,
gfx908/gfx90a/gfx94X/gfx950; TheRock excludes it on RDNA targets), so
libhiptensor is provided by both packages alongside the other libraries.

ld.lld now always comes from AMDGPU_LLVM_Backend_jll (it has to match the
LLVM version device code is generated with), and the device bitcode libraries
keep coming from ROCmDeviceLibs_jll until local ones can be downgraded.
libMIOpen_path is renamed to libMIOpen for consistency with the provider
packages.

Also adds a manually-dispatched workflow to regenerate the Artifacts.toml
hashes from repo.amd.com.

Assisted-by: Claude Code (claude-fable-5)
Both ROCm_Runtime and ROCm_Runtime_Discovery export the same library path
globals, so AMDGPU can simply `using` whichever the "local" preference
selects: the imported bindings alias the provider's globals and pick up the
values assigned by its `__init__`, which runs before AMDGPU loads. AMDGPU's
`libhip`/`libhsaruntime` names are renamed to the providers'
`libamdhip64`/`libhsa_runtime64` so no aliasing layer is needed, and
`lld_path`/`libdevice_libs` are imported directly from their JLLs, which are
assumed to always be available. The discovery package gains libhiprtc and
libamd_comgr lookups for API parity with ROCm_Runtime.

As a side effect the device libraries are now available while AMDGPU itself
precompiles (the JLL's `__init__` runs then), so the warming compile links
them for real; the DEVICE_LIBS reset stays to keep absolute artifact paths
and bitcode blobs out of the package image.

Assisted-by: Claude Code (claude-fable-5)
The subpackages are wired up through [sources], which only the active
project's Project.toml provides and which Pkg supports only on Julia 1.11+.
Steps running on Julia 1.10 (like the GPU-less environment check) or
resolving from another project (docs, perf) therefore failed with
"expected package ROCm_Runtime to be registered". Explicitly Pkg.develop'ing
the subpackages records path entries in the manifest, which works everywhere.

Verified by simulating the CI resolve in a fresh clone on Julia 1.10 and
nightly, including a GPU-less load (arch = "none") on 1.10.

Assisted-by: Claude Code (claude-fable-5)
The previous fix dev'ed the subpackages in the step command, but the
julia-test plugin instantiates the project in its pre-command hook, which
runs before the step's command -- so Julia 1.10 (no [sources] support) still
failed with "expected package ROCm_Runtime to be registered". Follow
CUDA.jl's approach instead: on 1.10 (tests, Enzyme, GPU-less check) drop the
julia-test plugin and run Pkg.develop + Pkg.test manually; on 1.11+ the
plugin works as-is since Pkg honors [sources] there, so those steps go back
to their plain Pkg.update() command.

Verified on a fresh clone with Julia 1.10: develop + update resolves, and
Pkg.test constructs its sandbox with the dev'ed subpackages from the parent
manifest.

Assisted-by: Claude Code (claude-fable-5)
The MI300 job set JULIA_AMDGPU_DISABLE_ARTIFACTS=1 to pick up the uenv's
ROCm. That variable no longer exists after the discovery refactor, so the
job silently resolved to the TheRock artifacts instead, and every test
died with hipErrorOutOfMemory out of hipStreamCreateWithPriority (3167
occurrences; no hipMalloc ever failed). The .rocm-miopen merge and
MIOPEN_PREFIX symlinking the script does were inert as a result.

Select the provider through ROCm_Runtime's "local" preference instead,
set before Pkg.instantiate() since it is a compile-time preference that
also gates artifact resolution, and dev the two workspace packages the
way .buildkite/pipeline.yml now does. The resolved provider is asserted
against the requested one afterwards, so a future change of default
fails loudly instead of quietly swapping the ROCm underneath the job.

The script is parameterised by ROCM_LOCAL so both providers can be run.
Only the uenv variant is enabled: this is the project's only gfx942 CI,
so covering the default artifact path here is worthwhile, but TheRock
ships a ROCm 7.14 userspace that cannot create a HIP stream at all
against the 6.12.12 amdgpu driver on this machine, so that job is left
commented out until the mismatch is resolved.

Verified on beverin (2x MI300A, gfx942, Julia 1.12.6) at c644fc5:

  uenv ROCm, full suite, --jobs=32:  16162 pass, 0 OOM  (was 3167 OOM)
  A/B in one allocation, only the preference differing:
    local=true   HIP 7.2.53211   hipStreamCreateWithPriority OK
    local=false  HIP 7.14.60850  hipErrorOutOfMemory

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment blamed a ROCm 7.14 userspace / 6.12.12 amdgpu driver mismatch.
That was wrong: the driver side is fine (rocminfo enumerates all agents,
hipMalloc succeeds). What actually fails is ROCclr's blit-kernel build on
first stream creation, which cannot resolve __amd_streamOpsIncrement and
__amd_streamOpsDecrement, and is then reported as hipErrorOutOfMemory --
which is what made this look like memory exhaustion for so long.

Reported upstream as ROCm/TheRock#7426, so the
comment now just points there rather than restating the analysis.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
simeonschaub and others added 3 commits August 26, 2026 14:38
Ship the ROCm 10.0 TheRock bundles from stable.repo.amd.com: regenerate
Artifacts.toml, bump ROCm_Runtime to 10.0.0 and the selectable toolkit
version along with AMDGPU's compat bound, and update the docs. The CSCS
MI300 artifact job is re-enabled with allow_failure, since the
hipErrorOutOfMemory failure on gfx942 was seen with the 7.14 bundle.

Assisted-by: Claude Code (claude-fable-5)
Adapts the upstream changes to the ROCm_Runtime provider split:

- #1044 (device libs from AMDGPU_LLVM_Backend_jll, downgraded with
  LLVMDowngrader_jll) is ported from the removed src/discovery/discovery.jl
  into src/libs.jl. `libdevice_libs` is now set in `__init__` from the
  downgraded scratch copy, falling back to the discovered local device
  libraries only when a local ROCm is used; ROCmDeviceLibs_jll is dropped.
- #1052 CSCS CI updates (new uenv/Julia paths, rocFFT trace layer, Julia
  1.13 job) are folded into the system-ROCm/artifact job layout.

Assisted-by: Claude Code (claude-fable-5)
@simeonschaub

This comment was marked as duplicate.

2 similar comments
@simeonschaub

This comment was marked as duplicate.

@simeonschaub

Copy link
Copy Markdown
Member Author

cscs-ci run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants