Forward Compatibility CUDA bug with CMake 3.17+ #110
Description
In HiOp, we link against CUDA in the following fashion:
find_package(CUDAToolkit REQUIRED)
if(HIOP_BUILD_SHARED)
target_link_libraries(hiop_cuda INTERFACE
CUDA::cusolver
CUDA::cusparse
CUDA::cudart
CUDA::cublasLt
)
endif()
if(HIOP_BUILD_STATIC)
target_link_libraries(hiop_cuda INTERFACE
CUDA::cusolver_static
CUDA::cusparse_static
CUDA::cudart_static
CUDA::cublasLt_static
)
endif()However, when building camp@2022.03.0 with RAJA@2022.03.0 and Umpire@2022.03.1, the camp CMake exported configuration ends up creating the camp target as follows (in campTargets.cmake):
# Create imported target camp
add_library(camp STATIC IMPORTED)
set_target_properties(camp PROPERTIES
INTERFACE_COMPILE_FEATURES "cxx_std_14"
INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include"
INTERFACE_LINK_LIBRARIES "cuda_runtime"
)Per CMake documentation https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html#module:FindCUDAToolkit, this breaks forward compatibility for including and linking CUDA libraries.
When building and running with this configuration of camp, RAJA and Umpire, I end up getting the following adding to my link line:
.../libumpire.a .../libRAJA.so .../libcamp.a -lcuda_runtime /share/apps/cuda/11.4/lib64/libcudart_static.a /usr/lib64/librt.soWhich generates an ld error:
/usr/bin/ld: cannot find -lcuda_runtimeSimply commenting out the line INTERFACE_LINK_LIBRARIES "cuda_runtime" in campTargets.cmake temporarily resolves the issue, but this should probably be updated for forward compatibility.
Activity
trws commentedon Jul 1, 2022
Do you not see this issue with RAJA or Umpire otherwise? That dependency comes through BLT, so I'm trying to track down how best to solve this.
Also, where in that documentation does it mention forward compatibility or this problem? Mislink?
cameronrutherford commentedon Jul 1, 2022
https://cmake.org/cmake/help/latest/module/FindCUDA.html - this is the link to the old way of finding cuda that is deprecated in favor of the link that I sent through. The old targets here are called
CUDA_cudart_static_LIBRARY, while the newer targets configured throughfind_package(CUDAToolkit)are insteadCUDA::cudartandCUDA::cudart_static. I still am not sure why/when linking againstcuda_runtimeworked, but I vaguely remember having that in HiOp initially.I didn't see this issue with RAJA 0.14.0 and Umpire 6.0.0, and I suspect that it is related to some of the C++ 14 changes that were made. That is largely a hunch though. Changing back to those versions with C++11 lets everything build just fine.
In terms of where this dependency comes from, I narrowed it down to this section of camp CMake https://github.com/LLNL/camp/blob/main/CMakeLists.txt#L69:
Parsing this manually, it seems like the string literal
cuda_runtimeis appended to the list passed toDEPENDS_ONinblt_add_library, and so I assume that this is where the dependency is being introduced. Perhaps the targetblt::cuda_runtimeshould exist, but isn't being found for some reason?I was tempted to make this sort of change myself, but I am not as familiar with camp as I would like, but I would imagine something like the following would acheive the same thing? It is a bit difficult (as we have seen in HiOp) to pick the correct static/shared library target to use:
Hopefully that clarifies.
trws commentedon Jul 1, 2022
rhornung67 commentedon Jul 1, 2022
@trws FWIW, Chris is working on the BLT updates for CUDA support to align what is done for HIP. He's run into some issues ripping out the object library stuff, which has issues with CUDA. So it's taking longer than we expected.
trws commentedon Jul 1, 2022
Thanks @rhornung67, I'm glad to hear that's going forward, but thankfully I don't think we're necessarily dependent on that rework to fix this. I can put in the stopgap in camp for now at least, and we may be able to do something in BLT to either select the static runtime or allow the runtime interface deps to not propagate.
cameronrutherford commentedon Jul 1, 2022
I'm satisfied with whatever is the most practical solution. I think a CMake
CAMP_NO_INTERFACE_RUNTIMEScould work, but I don't fully understand a lot of the implications of these decisions.If there is a fix coming through BLT that we are just waiting on, then I think we can wait for that. We are fine using the older RAJA and Umpire versions for the moment, but a temporary stopgap would be fine. I think this would have to be added to the camp spack package in the end, or I can use a specific camp branch for temporary development.
Thanks for the quick updates!
rhornung67 commentedon Jul 1, 2022
@cameronrutherford we'll keep you posted when the BLT updates and related changes in RAJA, Umpire, Camp, etc. are available.
trws commentedon Jul 12, 2022
@cameronrutherford, in working on some updates to the cmake here and in RAJA (partly spurred by this but also to fix other issues) I think we may already support this by use of BLT_EXPORT_THIRDPARTY=Off in the cmake configure. I'm not 100% sure that will remove the library dependency, but it will remove the target from the export. If that doesn't do it, we'll have to add something to completely prevent camp from linking to the library at all, which is a bit more complicated.
jaelynlitz commentedon Sep 6, 2022
I've been working with @cameronrutherford on testing out this suggested fix and am still running into some issues.
These are the changes I made to the RAJA spack package to add a toggle for the BLT_EXPORT_THIRDPARTY
With the suggested BLT_EXPORT_THIRDPARTY=Off, we run into an OpenMP compiler compatibility error in RAJA. We ran into this error with gcc8.5.0 and gcc10.2.0. Error snippet:
(full output below the cut)
Details
cameronrutherford commentedon Sep 6, 2022
Thanks for debugging this @jaelynlitz. We can probably disable OpenMP to avoid this issue for the moment, but I think this is worth debugging.
We tried to debug the OpenMP compatibility with our compiler by looking at docs here, but struggled to interpret what version we should use.
It is also unclear through spack and this build configuration what version of OpenMP we are pulling in in the first place.
rhornung67 commentedon Sep 6, 2022
When CMake is invoked, is ENABLE_OPENMP (no RAJA_ prefix) being set to on?
jaelynlitz commentedon Sep 6, 2022
@rhornung67 How do I check for that? I don't see it at hiop's cmake level
rhornung67 commentedon Sep 6, 2022
This may be a red herring, but I thought it may spark a thought. It looks like ENABLE_OPENMP must be on for RAJA_ENABLE_OPENMP to be on since RAJA_ENABLE_OPENMP is a CMake dependent option in RAJA. It doesn't make sense to me that gcc would not support OpenMP, or at least define the variable _OPENMP when OpenMP is enabled, which is standard. The logic here indicates that _OPENMP is not defined https://github.com/LLNL/RAJA/blob/develop/include/RAJA/config.hpp.in#L244
trws commentedon Sep 7, 2022
I'm not sure if it will be helpful or harmful for this issue, but the target export setup and various issues around it have been substantially reworked in the most recent releases of camp and umpire, and on develop in RAJA. It may be worth trying with those, and if you are using spack ensuring that you use the packages for those three from the most recent develop version of spack, to see if the rework helped or not.
For the OpenMP issue, what compiler and version is this with? It's extremely strange that the build would be able to get to that error, since cmake should have prevented it before that point, unless it's happening in a project including RAJA that is building without the OpenMP flags somehow.
jaelynlitz commentedon Sep 8, 2022
I updated spack to grab raja@develop, camp@2022.03.2, and umpire@2022.03.1 and am still running into the issue. I've tested with compilers gcc8.5.0 and gcc10.2.0 with no change - but I cannot find what version of OpenMP is being used.
Error output below the cut for reference:
Details
cameronrutherford commentedon Sep 8, 2022
I'm still noticing
camp@mainin your build, but apart from that, it looks like the same behaviour. Thankfully this isn't really a blocker for us, but frustrating that this is persisting with the newest versions.I realize that the original issue has been resolved, but I think continuing discussion is reasonable.
trws commentedon Sep 8, 2022
This error should only come up when compiling an application without OpenMP against a RAJA that was built with OpenMP. I also notice the camp in that spec does not list a +openmp variant, which tells me you aren't using the version of the camp package from spack's develop branch, so camp isn't getting the correct flags to add the OpenMP target correctly to the output.
pelesh commentedon Sep 22, 2022
Sorry, it took me long to test this. I cloned recursively RAJA repo and checked out the latest RAJA release. These are versions I got:
I also used
I built RAJA as a shared library with CUDA backend enabled and with
BLT_EXPORT_THIRDPARTY=Offas @trws suggested. I did not enable OpenMP backend. All RAJA tests passed.Turning off BLT exporting third party libraries helped me pass the CMake configuration for HiOp, but the "empty" flag
-lcuda_runtimeremained in the linking command and caused failure:[ 37%] Built target hiopLinAlg [ 38%] Linking CUDA device code CMakeFiles/hiop_shared.dir/cmake_device_link.o [ 38%] Linking CXX shared library libhiop.so /usr/bin/ld: cannot find -lcuda_runtime collect2: error: ld returned 1 exit status@cameronrutherford's brute force workaround to comment out any mentioning of
cuda_runtimeincampTargets.cmakethat gets me to build HiOp with RAJA built as a shared library.The target
RAJA::cuda_runtimeis empty, i.e. it contains no CUDA or other targets.pelesh commentedon Sep 23, 2022
The OpenMP issue @jaelynlitz found is unrelated to the forward compatibility with CMake 3.17, so I suggest we discuss it elsewhere.
balos1 commentedon Sep 28, 2022
@pelesh We do not use blt in SUNDIALS, so we run into this issue of the target
cuda_runtimenot being around with camp@2022.03.2 and RAJA@2022.03.1 (this was not a problem with versions before 2022.x.x). For now, I am working around it by creating a targetcuda_runtimeafterfind_package(RAJA)(if the target does not exist, see here). I would think that targets that blt produces should get defined somewhere in the exported.cmakefiles by libraries that use blt (so libraries that do not use blt can find them), so I am hoping once blt refactors the CUDA support we won't need to do this anymore.