Linking static library with device functions depends on argument order

[Snektron]

After creating a static library containing device functions using -fgpu-rdc and ar, the library is linked with a command like:

hipcc libdevicelibrary.a main.cpp -fgpu-rdc -o program

This works fine, however, when the argument order is reversed and main.cpp is passed before libdevicelibrary.a, the compiler tries to interpret the files from libdevicelibrary.a as source file:

$ hipcc main.cpp libdevicelibrary.a -fgpu-rdc -o program
/tmp/library.o:1:1: error: expected unqualified-id
<U+007F>ELF<U+0002><U+0001><U+0001><U+0000>.... many more lines

This is caused by that placing main.cpp on the command line causes hipcc to emit -x hip before it, which causes clang++ to interpret library.o as a .hip source file:

$ HIPCC_VERBOSE=1 hipcc main.cpp libdevicelibrary.a -fgpu-rdc -o program
hipcc-cmd: /opt/rocm/llvm/bin/clang++ [...extra options omitted for brevity] -x hip main.hip """/tmp/library.o""" -std=c++17 -fgpu-rdc -o "program" 

[Snektron]

It seems like the issue can be resolved by "resetting" the language before the object file is passed by using -x none. Manually invoking

/opt/rocm/llvm/bin/clang++ [...extra options omitted for brevity] -x hip main.hip -x none """/tmp/library.o""" -std=c++17 -fgpu-rdc -o "program" 

seems to work fine.

[kzhuravl]

cc @yxsamliu

[yxsamliu]

That is a hipcc issue. hipcc by default adds -x hip before .cpp files. -x hip applies to all input files after it.

hipcc is supposed to be a thin wrapper for clang and not intended for complicated command arguments handling. It is recommended to use clang++ instead of hipcc. Also it is recommended to use .hip as extension of HIP program instead of .cpp. If .cpp has to be used and .a file has to be after .cpp file, use '-x hip' before .cpp and '-x none' before .a file.

[ppanchad-amd]

@Snektron Internal ticket is created to investigate this issue. Thanks!

[schung-amd]

@Snektron Does the comment by @yxsamliu address your issue? It appears you've found a workaround, and based on the comment by @yxsamliu this is working as intended; i.e. hipcc is not intended to have this functionality.

[Snektron]

Typically this is not the way that libraries are placed on the command line, as they are usually placed last. The workaround works, yes, but in my opinion it would be nice if there was a proper solution. If hipcc does it's own language detection, would it not be relatively straight forward to detect the .a type and automatically pass -x none? I'm fairly sure that nvcc let's me pass the device library as last argument.

Note c that this came up during development of the static_device_library ROCm sample, see here.