Milvus Build Failing on Build Container. #36541

rohitreddy1698 · 2024-09-26T05:49:59Z

rohitreddy1698
Sep 26, 2024

I have started the containerised build environment using :

./scripts/devcontainer.sh up

➜  milvus git:(kafka-tls-2) ✗ docker compose -f docker-compose-devcontainer.yml ps          
NAME                COMMAND                  SERVICE             STATUS              PORTS
milvus-azurite-1    "docker-entrypoint.s…"   azurite             running             10000-10002/tcp
milvus-builder-1    "/entrypoint.sh tail…"   builder             running             
milvus-etcd-1       "/opt/bitnami/script…"   etcd                running (healthy)   2379-2380/tcp
milvus-jaeger-1     "/go/bin/all-in-one-…"   jaeger              running             4317-4318/tcp, 5775/udp, 5778/tcp, 9411/tcp, 14250/tcp, 14268/tcp, 16686/tcp, 6831-6832/udp
milvus-minio-1      "/usr/bin/docker-ent…"   minio               running (healthy)   9000/tcp
milvus-pulsar-1     "bin/pulsar standalo…"   pulsar              running             
➜  milvus git:(kafka-tls-2) ✗

Logged onto the container :
docker exec -ti milvus-builder-1 bash

Installed the dependencies using :
./scripts/install_deps.sh

Trying to build milvus using :

milvus

Fails with following error :

/usr/include/stdlib.h:565:13: note: '__libc_free' target declared here
  565 | extern void free (void *__ptr) __THROW;
      |             ^~~~
/usr/bin/ar: `u' modifier ignored since `D' is the default (see `U')
/usr/bin/ar: `u' modifier ignored since `D' is the default (see `U')
make[4]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build/3rdparty_download/jemalloc-subbuild/src/jemalloc_ep'
[ 85%] Performing install step for 'jemalloc_ep'
make[4]: Entering directory '/go/src/github.com/milvus-io/milvus/cmake_build/3rdparty_download/jemalloc-subbuild/src/jemalloc_ep'
/usr/bin/install -c -m 755 bin/jemalloc-config /go/src/github.com/milvus-io/milvus/internal/core/output/bin
/usr/bin/install -c -m 644 include/jemalloc/jemalloc.h /go/src/github.com/milvus-io/milvus/internal/core/output/include/jemalloc
/usr/bin/install -c -m 755 lib/libjemalloc.a /go/src/github.com/milvus-io/milvus/internal/core/output/lib
/usr/bin/install -c -m 644 jemalloc.pc /go/src/github.com/milvus-io/milvus/internal/core/output/lib/pkgconfig
make[4]: *** write jobserver: Bad file descriptor.  Stop.
make[4]: *** Waiting for unfinished jobs....
/usr/bin/install -c -m 755 bin/jemalloc.sh /go/src/github.com/milvus-io/milvus/internal/core/output/bin
/usr/bin/install -c -m 644 doc/jemalloc.html /go/src/github.com/milvus-io/milvus/internal/core/output/share/doc/jemalloc
/usr/bin/install -c -m 644 doc/jemalloc.3 /go/src/github.com/milvus-io/milvus/internal/core/output/share/man/man3
make[4]: *** write jobserver: Bad file descriptor.  Stop.
make[3]: *** [thirdparty/jemalloc/CMakeFiles/jemalloc_ep.dir/build.make:106: 3rdparty_download/jemalloc-subbuild/src/jemalloc_ep-stamp/jemalloc_ep-install] Error 2
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[2]: *** [CMakeFiles/Makefile2:1968: thirdparty/jemalloc/CMakeFiles/jemalloc_ep.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
/usr/bin/install -c -m 755 bin/jeprof /go/src/github.com/milvus-io/milvus/internal/core/output/bin
/usr/bin/install -c -m 755 lib/libjemalloc_pic.a /go/src/github.com/milvus-io/milvus/internal/core/output/lib
Note: Writing jemalloc.3
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[ 85%] Built target milvus_index
make[2]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[1]: *** [Makefile:146: all] Error 2
make[1]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make: *** [Makefile:235: build-cpp] Error 1
root@0085dad36552:/go/src/github.com/milvus-io/milvus#

make -j1

fails with the following error :

make[4]: Entering directory '/go/src/github.com/milvus-io/milvus/cmake_build/3rdparty_download/jemalloc-subbuild/src/jemalloc_ep'
[  7%] Built target milvus_log
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
/usr/bin/install -c -m 755 bin/jemalloc-config /go/src/github.com/milvus-io/milvus/internal/core/output/bin
/usr/bin/install -c -m 644 include/jemalloc/jemalloc.h /go/src/github.com/milvus-io/milvus/internal/core/output/include/jemalloc
Missing xsltproc.  doc/jemalloc.3 not (re)built.
Missing xsltproc.  doc/jemalloc.html not (re)built.
make[4]: *** write jobserver: Bad file descriptor.  Stop.
make[4]: *** Waiting for unfinished jobs....
make[4]: *** write jobserver: Bad file descriptor.  Stop.
make[3]: *** [thirdparty/jemalloc/CMakeFiles/jemalloc_ep.dir/build.make:106: 3rdparty_download/jemalloc-subbuild/src/jemalloc_ep-stamp/jemalloc_ep-install] Error 2
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[2]: *** [CMakeFiles/Makefile2:1968: thirdparty/jemalloc/CMakeFiles/jemalloc_ep.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[3]: Entering directory '/go/src/github.com/milvus-io/milvus/cmake_build'
/usr/bin/install -c -m 755 bin/jemalloc.sh /go/src/github.com/milvus-io/milvus/internal/core/output/bin
/usr/bin/install -c -m 755 bin/jeprof /go/src/github.com/milvus-io/milvus/internal/core/output/bin
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[  9%] Built target milvus_bitset
make[3]: Entering directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[ 10%] Built target milvus_monitor
make[3]: Entering directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[3]: Warning: File 'thirdparty/knowhere/knowhere-build/CMakeFiles/knowhere_utils.dir/compiler_depend.make' has modification time 0.016 s in the future
[ 10%] Compiling tantivy binding
make[3]: warning:  Clock skew detected.  Your build may be incomplete.
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[3]: Entering directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[ 11%] Built target knowhere_utils
[ 11%] Building CXX object src/pb/CMakeFiles/milvus_proto.dir/cgo_msg.pb.cc.o
[ 12%] Building CXX object src/pb/CMakeFiles/milvus_proto.dir/clustering.pb.cc.o
[ 12%] Building CXX object src/pb/CMakeFiles/milvus_proto.dir/common.pb.cc.o
[ 12%] Building CXX object src/pb/CMakeFiles/milvus_proto.dir/index_cgo_msg.pb.cc.o
[ 12%] Building CXX object src/pb/CMakeFiles/milvus_proto.dir/plan.pb.cc.o
[ 12%] Building CXX object src/pb/CMakeFiles/milvus_proto.dir/schema.pb.cc.o
[ 13%] Building CXX object src/pb/CMakeFiles/milvus_proto.dir/segcore.pb.cc.o
[ 13%] Linking CXX static library ../../lib/libmilvus_proto.a
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[ 16%] Built target diskann
warning: variants `Keyword`, `I64`, `F64`, and `Bool` are never constructed
 --> src/data_type.rs:4:5
  |
2 | pub enum TantivyDataType {
  |          --------------- variants in this enum
3 |     // Text,
4 |     Keyword,
  |     ^^^^^^^
5 |     // U64,
6 |     I64,
  |     ^^^
7 |     F64,
  |     ^^^
8 |     Bool,
  |     ^^^^
  |
  = note: `#[warn(dead_code)]` on by default

warning: `tantivy-binding` (lib) generated 1 warning
    Finished release [optimized] target(s) in 0.64s
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[ 16%] Built target tantivy_binding_target
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[ 26%] Built target milvus-storage
make[3]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
[ 26%] Built target milvus_proto
make[2]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make[1]: *** [Makefile:146: all] Error 2
make[1]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make: *** [Makefile:235: build-cpp] Error 1
root@0085dad36552:/go/src/github.com/milvus-io/milvus#

I have already tried installing xsltproc :

apt-get install libxslt-dev xsltproc docbook-xsl

root@0085dad36552:/go/src/github.com/milvus-io/milvus# which xsltproc
/usr/bin/xsltproc
root@0085dad36552:/go/src/github.com/milvus-io/milvus#

Still this is failing with xsltproc not found.

yhmo · 2024-09-27T02:05:51Z

yhmo
Sep 27, 2024
Collaborator

We didn't see this error before, you can search on google for more info.

8 replies

rohitreddy1698 Sep 27, 2024
Author

I have sent you a message on Discord, please do let me know if its possible to connect there.

Thank you !

rohitreddy1698 Sep 27, 2024
Author

Now the latest I am getting is :

c-ares/1.33.0: WARN: Build folder is dirty, removing it: /Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214
c-ares/1.33.0: Copying sources to build folder
c-ares/1.33.0: Building your package in /Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214
c-ares/1.33.0: Generator txt created conanbuildinfo.txt
c-ares/1.33.0: Calling generate()
c-ares/1.33.0: WARN: Using the new toolchains and generators without specifying a build profile (e.g: -pr:b=default) is discouraged and might cause failures and unexpected behavior
c-ares/1.33.0: Preset 'release' added to CMakePresets.json. Invoke it manually using 'cmake --preset release'
c-ares/1.33.0: If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release/generators/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
c-ares/1.33.0: Aggregating env generators
c-ares/1.33.0: Calling build()
c-ares/1.33.0: Apply patch (portability): fix compilation error for TCP_FASTOPEN_CONNECT
c-ares/1.33.0: CMake command: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/package/65d0c6387c314520223dd15a4a7aadd3107d2214" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/src"
-- Using Conan toolchain: /Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release/generators/conan_toolchain.cmake
-- Conan toolchain: Setting CMAKE_POSITION_INDEPENDENT_CODE=ON (options.fPIC)
-- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF
-- The C compiler identification is Clang 15.0.7
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/homebrew/bin/ccache - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test HAVE__Wall
-- Performing Test HAVE__Wall - Success

.
.
.
-- Looking for pthread_init
-- Looking for pthread_init - not found
-- Configuring done (20.3s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release
c-ares/1.33.0: CMake command: cmake --build "/Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release" '--' '-j14'
[  1%] Building C object src/lib/CMakeFiles/c-ares.dir/ares__addrinfo2hostent.c.o
[  2%] Building C object src/lib/CMakeFiles/c-ares.dir/ares__addrinfo_localhost.c.o
[  3%] Building C object src/lib/CMakeFiles/c-ares.dir/ares__hosts_file.c.o
[  4%] Building C object src/lib/CMakeFiles/c-ares.dir/ares__parse_into_addrinfo.c
.
.
.
[ 85%] Building C object src/lib/CMakeFiles/c-ares.dir/str/ares_strsplit.c.o
[ 87%] Building C object src/lib/CMakeFiles/c-ares.dir/util/ares__iface_ips.c.o
[ 87%] Building C object src/lib/CMakeFiles/c-ares.dir/util/ares__threads.c.o
[ 88%] Building C object src/lib/CMakeFiles/c-ares.dir/util/ares__timeval.c.o
[ 89%] Building C object src/lib/CMakeFiles/c-ares.dir/util/ares_math.c.o
[ 90%] Building C object src/lib/CMakeFiles/c-ares.dir/util/ares_rand.c.o
In file included from /Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/src/src/lib/ares_sysconfig_mac.c:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCNetworkConfiguration.h:31:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SystemConfiguration.h:126:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCPreferences.h:196:6: error: unknown type name 'AuthorizationRef'
                                        AuthorizationRef        __nullable      authorization
                                        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCPreferences.h:196:23: error: nullability specifier '_Nullable' cannot be applied to non-pointer type 'int'
                                        AuthorizationRef        __nullable      authorization
                                                                ^
<built-in>:337:20: note: expanded from here
#define __nullable _Nullable
                   ^
2 errors generated.
make[3]: *** [src/lib/CMakeFiles/c-ares.dir/ares_sysconfig_mac.c.o] Error 1
make[2]: *** [src/lib/CMakeFiles/c-ares.dir/all] Error 2
make[1]: *** [all] Error 2
c-ares/1.33.0: 
c-ares/1.33.0: ERROR: Package '65d0c6387c314520223dd15a4a7aadd3107d2214' build failed
c-ares/1.33.0: WARN: Build folder /Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release
ERROR: c-ares/1.33.0: Error in build() method, line 63
	cmake.build()
	ConanException: Error 2 while executing cmake --build "/Users/rohitmothe/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release" '--' '-j14'
conan install failed
make: *** [build-3rdparty] Error 1
➜  milvus git:(master) ✗

yhmo Sep 29, 2024
Collaborator

Sorry, I have no idea why you get the error. It is not a milvus product error, we didn't get any trouble building the official milvus docker image with the same script, and we didn't get the error like yours. There were users get problems compiling milvus source code before, we just provided some guidance based on our experience, not all the problems can be solved since the software environment is different.

rohitreddy1698 Oct 4, 2024
Author

Hi this was an issue with the Mac Os internal headers or the compatibility issue between Mac OS and llvm 15 , I was able to get over the issue once I upgraded to llvm 16.
The build went forward and build is successful. And I was able to create a docker image using the build/build_image.sh script.

But when trying to deploy it the components are failing with the following error :

➜  milvus git:(main) ✗ kubectl logs deployment.apps/milvus-querycoord
Found 2 pods, using pod/milvus-querycoord-74f8b48ddc-5lsbr
Defaulted container "querycoord" out of: querycoord, config (init)
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
/milvus/tools/run.sh: line 25: exec: milvus: not found

But I can see that I have the milvus executable in the pod :

➜  milvus git:(master) ✗ docker run --rm -it docker.apple.com/rohit-mothe/milvus_tls:1.2 /bin/bash

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
root@1132b3eaf74d:/milvus# 

root@1132b3eaf74d:/milvus# ls
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
bin  configs  lib
root@1132b3eaf74d:/milvus# 
root@1132b3eaf74d:/milvus# ls bin/
ERROR: ld.so: object '/milvus/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
milvus  protoc-gen-go  protoc-gen-go-grpc

And as the error specifies I cannot see the /milvus/lib/libjemalloc.so in the lib folder. And on checking I was unable to find the libjemalloc.so , after the build under /internal/core/output/lib as well.

On further inspection in the conanfile.py at /internal/core :

milvus/internal/core/conanfile.py

Line 89 in 3936d12

self.options["arrow"].with_jemalloc = False

def configure(self):
        if self.settings.arch not in ("x86_64", "x86"):
            del self.options["folly"].use_sse4_2
        if self.settings.os == "Macos":
            # By default abseil use static link but can not be compatible with macos X86
            self.options["abseil"].shared = True
            self.options["arrow"].with_jemalloc = False

Installing jemalloc is set to false if building on a Mac OS.
Is this expected/intended and if yes why is the startup of the Milvus components failing with the same library missing

Regards,
Rohit

rohitreddy1698 Oct 7, 2024
Author

Was an issue with the Mac Os internal headers or the compatibility issue between Mac OS and llvm 15 , I was able to get over the issue once I upgraded to llvm 16.
The build went forward and build is successful.

vepicurean · 2025-01-02T20:54:03Z

vepicurean
Jan 2, 2025

I get the same error when trying to start the container for Milvus standalone on a Podman Linux VM.

The same - ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.

Occurs when starting the container on Linux VM with lots of RAM ( over 8GB) on MacOS using Podman.

podman run -d name --milvus ( container build starts and then exits right away)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Milvus Build Failing on Build Container. #36541

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Milvus Build Failing on Build Container. #36541

rohitreddy1698 Sep 26, 2024

Replies: 2 comments · 8 replies

yhmo Sep 27, 2024 Collaborator

rohitreddy1698 Sep 27, 2024 Author

rohitreddy1698 Sep 27, 2024 Author

yhmo Sep 29, 2024 Collaborator

rohitreddy1698 Oct 4, 2024 Author

rohitreddy1698 Oct 7, 2024 Author

vepicurean Jan 2, 2025

rohitreddy1698
Sep 26, 2024

Replies: 2 comments 8 replies

yhmo
Sep 27, 2024
Collaborator

rohitreddy1698 Sep 27, 2024
Author

rohitreddy1698 Sep 27, 2024
Author

yhmo Sep 29, 2024
Collaborator

rohitreddy1698 Oct 4, 2024
Author

rohitreddy1698 Oct 7, 2024
Author

vepicurean
Jan 2, 2025