Possible Wrong Search Path for C Runtime objects

[dslm4515]

When Clang(llvm) is built under chroot and tested, LLD searches for the C Runtime objects in this directory:

ld.lld: /usr/bin/../lib/Scrt1.o
ld.lld: /usr/bin/../lib/crti.o
ld.lld: /usr/bin/../lib/crtn.o

Which resolves to /usr/lib/*crt[1in].o

Once GCC is built under chroot after or before LLVM, LLD now searches for C runtime objects in:

ld. lld: /usr/lib/gcc/x86_64-linux-musl/10.3.1/../../../Scrt1.o
ld.lld: /usr/lib/gcc/x86_64-linux-musl/10.3.1/../../../crti.o
ld.lld: /usr/lib/gcc/x86_64-linux-musl/10.3.1/../../../crtn.o

Which resolves to /usr/lib/*crt[1in].o

Why does the search path change (even though both paths point to the same directory)?

And, can it be set to just /usr/lib ?

[andzai1995]

I have a version about this behaviour. LLVM has some specific code for GNU/Linux-Unix targets, and this specific code has conditional code that selects GCC pathes for Libc *crt[1in].o. This code (it is just my guess) should be in $LLVMSRC/tools/ld directory. But this part of code is very big. Maybe somehow, you can set the easiest way for this pathes - by modifying the code or setting options for cmake - anyway, you should find info about making of cross-compiler from clang/llvm (Something like how to make cross-compiler for your new unix-like OS from osdev.org for gcc, but for clang/llvm).

[dslm4515]

Looks like patching clang/lib/Driver/ToolChains/Linux.cpp to set the linux loader path fixes this issue.

For example, I patched the clang source to change the linux loader path from /lib to /llvmtools/lib:
commit d3709f5
... and now the search paths are clean:

ld.lld: /usr/lib/Scrt1.o
ld.lld: /usr/lib/crti.o
ld.lld: /usr/lib/crtn.o

I'll see if this is still valid for stage2 clang, the compiler for the final system

[dslm4515]

this specific code has conditional code that selects GCC pathes for Libc *crt[1in].o.

@andzai1995, thanks to your comment, I actually ran a grep command on the LLVM project source to see how the path was set. Sure enough, you were right: clang/lib/Driver/ToolChains/Linux.cpp has a code block that sets the path for the linux [dynamic] loader... which then fixes this issue of absurdly long search paths

setting options for cmake

I looked at all the cmake options with ccmake ... no way to set the linux loader path. Source has to be patched. I am not familiar with creating or modifying CMakeList.txt's ... so I don't know how I can modify the CMakeListtxt to have an option to set the dynamic loader path as a cmake option.

[andzai1995]

this specific code has conditional code that selects GCC pathes for Libc *crt[1in].o.

@andzai1995, thanks to your comment, I actually ran a grep command on the LLVM project source to see how the path was set. Sure enough, you were right: clang/lib/Driver/ToolChains/Linux.cpp has a code block that sets the path for the linux [dynamic] loader... which then fixes this issue of absurdly long search paths

setting options for cmake

I looked at all the cmake options with ccmake ... no way to set the linux loader path. Source has to be patched. I am not familiar with creating or modifying CMakeList.txt's ... so I don't know how I can modify the CMakeList.txt to have an option to set the dynamic loader path as a cmake option.

@dslm4515, Well, source need to be patched means source need to be patched.
It was big pleasure to help you.

What about chroot/grub?
I had sent you a variant to fix situation with bloating linker more then year ago, but you didn't implement it.
There are three tests for configure script that must be bypassed and some modifications for Makefiles
#23 (comment)

[dslm4515]

I haven't forgotten.

The change from LLVM-12 to LLVM-15.x.x was huge. I couldn't build llvmtools with LLVM 13/14. I decided to take a break. LLVM takes hours to compile on a i5 3750 with 4 CPU cores. It really pissed me off to wait for hours, only to realize the build failed or something broke. Then a month ago I returned to working on CMLFS. Decided to use LLVM 15.

Its only now I got llvmtools fixed with a known working stage1 clang. I am right now building the final system, so I will eventually get to GRUB and test out your build recipe!

[dslm4515]

I'm building on a new machine with a Ryzen 5 5600x. LLVM source now builds take ~ 20 minutes. So now the failures don't hurt as much as on the i5

[andzai1995]

I had successfully built CMLFS with LLVM 14, to do this was necessary to disable ORC subsystem.
And use good old way of build without variables.
My both PCs are far older than yours, and LLVM build always takes a bunch of hours.
On my very old machine it takes 20 hours, on newer it takes 10 hours.
But if some error happened, it always happens in the beginning of the build, in the beginning.
Llvmtools LLVM fails on libcxx due to lack of link to /llvmtools/include, or this directory didn't found by build system.

[dslm4515]

I had successfully built CMLFS with LLVM 14

Wow! Perhaps, share and I can add your work on the LLVM-14.x.x branch.

My both PCs are far older than yours, and LLVM build always takes a bunch of hours.

I ended up looking up the build targets so I can see which one fails and build it first. For example, I had LLVM fail at building LLD. So I can sometimes skip the LLVM runtimes and LLVM libraries to build LLD sooner with ninja -C build lld

But the patching for changing the dynamic loader fixed a lot of issues like building CMake required a compiler with C++11 support or when building the stage1 clang, compiler needed atomics. Hence, the extra tests I added to avoid using a broken compiler. Or forcing stage1 to find the libc++ headers in /llvmtools/include/c++/v1

My next hurdle at the moment is building LLVM for the final system.