[Bug]: Python SDK failed to connect local Milvus while the deployment is fine

[lbbc1117]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.5.1
- Deployment mode(standalone or cluster): standalone (docker)
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): [email protected]
- OS(Ubuntu or CentOS): MacOS
- CPU/Memory: 4
- GPU: 
- Others:

Current Behavior

Run code:

from pymilvus import MilvusClient

milvus_client = MilvusClient(uri="http://127.0.0.1:19530")

if __name__ == "__main__":
    print(milvus_client.list_collections())

Failed to connect to server:

While the telnet is succeeded:

And the Attu is working too:

Expected Behavior

Connected to local Milvus server

Steps To Reproduce

No response

Milvus Log

milvus-standalone | [2024/12/31 04:26:01.200 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:26:01.200 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:26:01.201 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685723779] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:26:30.052 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:27:00.052 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:27:01.200 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685724080] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:27:30.051 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:28:00.051 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:28:01.198 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:28:01.198 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:28:01.198 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685724381] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:28:30.049 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:29:00.047 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:29:01.196 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685724682] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:29:30.048 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:30:00.047 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:30:01.196 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:30:01.196 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:30:01.198 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685724983] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:30:30.046 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:31:00.044 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:31:01.194 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685725284] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:31:30.044 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:32:00.038 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:32:01.185 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:32:01.185 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:32:01.186 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685725585] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:32:30.028 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:33:00.026 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:33:01.175 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685725886] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:33:30.025 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:34:00.024 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:34:00.210 +00:00] [INFO] [server/rocksmq_impl.go:261] ["Rocksmq stats"] [module=rocksmq] [cache=1040] ["rockskv memtable "=2048] ["rockskv table readers"=384] ["rockskv pinned"=96] ["store memtable "=2048] ["store table readers"=192] ["store pinned"=96] ["store l0 file num"=1] ["store l1 file num"=0] ["store l2 file num"=0] ["store l3 file num"=0] ["store l4 file num"=0]
milvus-standalone | [2024/12/31 04:34:01.172 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:34:01.172 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:34:01.174 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685726187] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:34:01.591 +00:00] [INFO] [importv2/util.go:273] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:34:01.591 +00:00] [INFO] [importv2/util.go:273] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
milvus-standalone | [2024/12/31 04:34:30.023 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:35:00.024 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]
milvus-standalone | [2024/12/31 04:35:01.172 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=454982117685726488] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2024/12/31 04:35:30.021 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=26]

Anything else?

No response

[yanliang567]

@lbbc1117 please attach the milvus logs for investigation. For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs.

/assign @lbbc1117
/unassign

[lbbc1117]

milvus.log

Here the log file, @yanliang567

[yanliang567]

it seems that the milvus pod is restarting from time to time.
/assign @xiaocai2333 could you please help to take a look

there is an etcd error says:
milvus-standalone | [2024/12/31 04:23:59.543 +00:00] [WARN] [sessionutil/session_util.go:933] ["Watch service found compacted error"] [error="etcdserver: mvcc: required revision has been compacted"]

[lbbc1117]

it seems that the milvus pod is restarting from time to time. /assign @xiaocai2333 could you please help to take a look

there is an etcd error says: milvus-standalone | [2024/12/31 04:23:59.543 +00:00] [WARN] [sessionutil/session_util.go:933] ["Watch service found compacted error"] [error="etcdserver: mvcc: required revision has been compacted"]

I restarted the docker compose several times. Any chance its the reason of pod restarting in the log file ?

And all the 3 pods seem working well after the last start:

[Silversoul-07]

@lbbc1117 did you figure out the problem?

[lbbc1117]

@lbbc1117 did you figure out the problem?

not yet

[lbbc1117]

@xiaocai2333 Could you provide any help, thanks a lot

[Silversoul-07]

@lbbc1117 I was also having the same issue but its fixed for me now. I tried a few solutions check if any of it helps

1. ensure pymilvus version and container version as suggested in release page.
2. if you dont have any important data try deleting the container completely start fresh
3. i merged milvus docker compose with my own compose after adding network it worked for me

[xiaocai2333]

@xiaocai2333 Could you provide any help, thanks a lot

Sorry for the late response. I found in the logs that:

milvus-standalone  | [2024/12/31 04:23:57.310 +00:00] [WARN] [roles/roles.go:324] ["Get signal to exit"] [signal=terminated]

Could you check your deployment method? It seems that someone actively triggered the shutdown of Milvus.

[lbbc1117]

thx guys @Silversoul-07 @xiaocai2333
solved