Milvus Abort on Startup

[danielsmaxwell]

We want to run Milvus on our university’s (University of Florida) supercomputer. Unfortunately, the Research Computing department here does not use Docker but Apptainer. We therefore pulled the latest Milvus image from Docker Hub into Apptainer and have been successful starting the milvus-etcd and milvus-minio processes. However, every time we try to start milvus-standalone, the process aborts immediately after it executes the line for milvus-standalone. No error messages are thrown. We have no idea what’s going on and nothing is listed in the core.milvus file. Our process is running with 10g of memory and 1 CPU. We can increase those if needed and ask for more memory, CPUs, and GPUs in our SLURM request.

If anyone has any ideas, we’d appreciate hearing them as we’re not making much progress on this one.

[yhmo]

Does the CPU instruction meet this requirement?
https://milvus.io/docs/prerequisite-docker.md

Required:

• SSE4.2
• AVX
• AVX2
• AVX-512

[danielsmaxwell]

Yes - we meet these CPU requirements. Thanks!

[codingjaguar]

Thanks for raising the issue. This is a duplicate of #33964.

I've asked around the Milvus maintainers and unfortunately there isn't a support of apptainer a now. Milvus only supports docker container and k8s. We will see if the community has any expert on HPC and Apptainer to help build that feature. Meanwhile I'd suggest to try docker or k8s deployment env. If performance isn't a concern (say the data set is less than a million vectors), you can try https://milvus.io/docs/milvus_lite.md which only requires python env.

[danielsmaxwell]

I just discovered that we're running PodMan on our HPC. PodMan solves the sudo / security issues with Docker in HPC environments. We're going to test that as a possible work around here.

[vepicurean]

This same behaviour, where the Milvus container exits right away on startup occurs on Podman on MacOS ( linux VM using Podman meets requirements). Can't find a way to get around this issue with the Milvus standalone setup.

[xiaofan-luan]

@zwd1208
is working on some test of apptainer.

[danielsmaxwell]

Excellent! Thank you!

[Linkwei]

Hi there and thank you for your patience.

I'm not very familiar with Apptainer. After studying it carefully, I tried to troubleshoot the issue you encountered in the following environment, but I couldn't reproduce the problem. Below are the operating environment and steps. Could you please provide the specific logs, environment version, and operation process on your side?

Version of Apptainer

apptainer version 1.3.4

OS Version
cat /proc/version

Linux version 6.8.0-1016-aws (buildd@lcy02-amd64-115) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #17-Ubuntu

Steps of Operation
1.Create a milvus configuration file

    cat << EOF > embedEtcd.yaml
listen-client-urls: http://0.0.0.0:2379
advertise-client-urls: http://0.0.0.0:2379
quota-backend-bytes: 4294967296
auto-compaction-mode: revision
auto-compaction-retention: '1000'
EOF

    cat << EOF > user.yaml
# Extra config to override default milvus.yaml
EOF

2.Build images
apptainer build --sandbox milvus.v2.4.13.sif docker://milvusdb/milvus:v2.4.13
3.Run container
sudo apptainer exec --writable-tmpfs --bind ./embedEtcd.yaml:/milvus/configs/embedEtcd.yaml --bind ./user.yaml:/milvus/configs/user.yaml milvus.v2.4.13.sif bash -c "export ETCD_USE_EMBED=true && export ETCD_DATA_DIR=/var/lib/milvus/etcd&&export ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml && export COMMON_STORAGETYPE=local && cd /milvus && ./bin/milvus run standalone"

PS.Need to execute the startup command in the relative path

[xiaofan-luan]

Hi there and thank you for your patience.

I'm not very familiar with Apptainer. After studying it carefully, I tried to troubleshoot the issue you encountered in the following environment, but I couldn't reproduce the problem. Below are the operating environment and steps. Could you please provide the specific logs, environment version, and operation process on your side?

Version of Apptainer

apptainer version 1.3.4

OS Version cat /proc/version

Linux version 6.8.0-1016-aws (buildd@lcy02-amd64-115) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #17-Ubuntu

Steps of Operation 1.Create a milvus configuration file

    cat << EOF > embedEtcd.yaml
listen-client-urls: http://0.0.0.0:2379
advertise-client-urls: http://0.0.0.0:2379
quota-backend-bytes: 4294967296
auto-compaction-mode: revision
auto-compaction-retention: '1000'
EOF

    cat << EOF > user.yaml
# Extra config to override default milvus.yaml
EOF

2.Build images apptainer build --sandbox milvus.v2.4.13.sif docker://milvusdb/milvus:v2.4.13 3.Run container sudo apptainer exec --writable-tmpfs --bind ./embedEtcd.yaml:/milvus/configs/embedEtcd.yaml --bind ./user.yaml:/milvus/configs/user.yaml milvus.v2.4.13.sif bash -c "export ETCD_USE_EMBED=true && export ETCD_DATA_DIR=/var/lib/milvus/etcd&&export ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml && export COMMON_STORAGETYPE=local && cd /milvus && ./bin/milvus run standalone"

PS.Need to execute the startup command in the relative path

Thanks for helping on this