Skip to content

[BUG] etcd scale-out pod role is null error: cluster ID mismatch #9698

@JashBook

Description

@JashBook

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: mongodb-zedvuh
  namespace: default
spec:
  clusterDef: mongodb
  topology: replicaset
  terminationPolicy: Delete
  componentSpecs:
    - name: mongodb
      serviceVersion: 4.4.29
      replicas: 3
      resources:
        limits:
          cpu: 100m
          memory: 0.5Gi
        requests:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
kubectl get cluster 
NAME          CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
etcd-eewrxk                        Delete               Running   51m

kbcli cluster list-instances etcd-eewrxk 
NAME                 NAMESPACE   CLUSTER       COMPONENT   STATUS    ROLE       ACCESSMODE   AZ               CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                        CREATED-TIME                 
etcd-eewrxk-etcd-0   default     etcd-eewrxk   etcd        Running   leader                  cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:21Gi   192.168.0.37/192.168.0.37   Sep 03,2025 16:14 UTC+0800   
etcd-eewrxk-etcd-1   default     etcd-eewrxk   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:21Gi   192.168.0.37/192.168.0.37   Sep 03,2025 16:14 UTC+0800   
etcd-eewrxk-etcd-2   default     etcd-eewrxk   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:21Gi   192.168.0.37/192.168.0.37   Sep 03,2025 16:14 UTC+0800  
  1. scale-out
kbcli cluster scale-out etcd-eewrxk --components etcd --replicas 1 --auto-approve
OpsRequest etcd-eewrxk-horizontalscaling-ccpmp created successfully, you can view the progress:
	kbcli cluster describe-ops etcd-eewrxk-horizontalscaling-ccpmp -n default
  1. See error
kubectl get cluster etcd-eewrxk 
NAME          CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
etcd-eewrxk                        Delete               Updating   92m

➜  ~ kubectl get pod  
NAME                 READY   STATUS    RESTARTS   AGE
etcd-eewrxk-etcd-0   2/2     Running   0          93m
etcd-eewrxk-etcd-1   2/2     Running   0          92m
etcd-eewrxk-etcd-2   2/2     Running   0          92m
etcd-eewrxk-etcd-3   2/2     Running   0          40m

➜  ~ kubectl get ops
NAME                                  TYPE                CLUSTER       STATUS    PROGRESS   AGE
etcd-eewrxk-horizontalscaling-ccpmp   HorizontalScaling   etcd-eewrxk   Running   0/1        41m
etcd-eewrxk-volumeexpansion-pjcdk     VolumeExpansion     etcd-eewrxk   Succeed   3/3        90m

kbcli cluster list-instances etcd-eewrxk
NAME                 NAMESPACE   CLUSTER       COMPONENT   STATUS    ROLE       ACCESSMODE   AZ               CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                        CREATED-TIME                 
etcd-eewrxk-etcd-0   default     etcd-eewrxk   etcd        Running   leader                  cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:21Gi   192.168.0.37/192.168.0.37   Sep 03,2025 16:14 UTC+0800   
etcd-eewrxk-etcd-1   default     etcd-eewrxk   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:21Gi   192.168.0.37/192.168.0.37   Sep 03,2025 16:14 UTC+0800   
etcd-eewrxk-etcd-2   default     etcd-eewrxk   etcd        Running   follower                cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:21Gi   192.168.0.37/192.168.0.37   Sep 03,2025 16:14 UTC+0800   
etcd-eewrxk-etcd-3   default     etcd-eewrxk   etcd        Running   <none>                  cn-guangzhou-b   100m / 100m          512Mi / 512Mi           data:21Gi   192.168.0.37/192.168.0.37   Sep 03,2025 17:06 UTC+0800   

logs pod

kubectl logs etcd-eewrxk-etcd-3 --tail 20
Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init)
{"level":"warn","ts":"2025-09-03T09:48:48.530510Z","caller":"rafthttp/stream.go:652","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"95d9be39c2616a56","remote-peer-cluster-id":"47c17342e3a1b4ec","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","error":"cluster ID mismatch"}
{"level":"warn","ts":"2025-09-03T09:48:48.613422Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"95d9be39c2616a56","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.613432Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"9a561d9e8f530bbc","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.613422Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"9a561d9e8f530bbc","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.613422Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"95d9be39c2616a56","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.613873Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"d0113665f403934f","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.613924Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"d0113665f403934f","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.619639Z","caller":"rafthttp/stream.go:652","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"d0113665f403934f","remote-peer-cluster-id":"47c17342e3a1b4ec","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","error":"cluster ID mismatch"}
{"level":"warn","ts":"2025-09-03T09:48:48.619647Z","caller":"rafthttp/stream.go:652","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"d0113665f403934f","remote-peer-cluster-id":"47c17342e3a1b4ec","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","error":"cluster ID mismatch"}
{"level":"warn","ts":"2025-09-03T09:48:48.619651Z","caller":"rafthttp/stream.go:652","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"9a561d9e8f530bbc","remote-peer-cluster-id":"47c17342e3a1b4ec","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","error":"cluster ID mismatch"}
{"level":"warn","ts":"2025-09-03T09:48:48.630019Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"9a561d9e8f530bbc","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.630559Z","caller":"rafthttp/stream.go:652","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"9a561d9e8f530bbc","remote-peer-cluster-id":"47c17342e3a1b4ec","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","error":"cluster ID mismatch"}
{"level":"warn","ts":"2025-09-03T09:48:48.630601Z","caller":"rafthttp/stream.go:652","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"95d9be39c2616a56","remote-peer-cluster-id":"47c17342e3a1b4ec","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","error":"cluster ID mismatch"}
{"level":"warn","ts":"2025-09-03T09:48:48.630601Z","caller":"rafthttp/stream.go:652","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"95d9be39c2616a56","remote-peer-cluster-id":"47c17342e3a1b4ec","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","error":"cluster ID mismatch"}
{"level":"warn","ts":"2025-09-03T09:48:48.713480Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"9a561d9e8f530bbc","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.713511Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"9a561d9e8f530bbc","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.713480Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"95d9be39c2616a56","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.713499Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"95d9be39c2616a56","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.713835Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"d0113665f403934f","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
{"level":"warn","ts":"2025-09-03T09:48:48.713899Z","caller":"rafthttp/http.go:502","msg":"request cluster ID mismatch","local-member-id":"fca7465b681861c4","local-member-cluster-id":"966fded6707566d3","local-member-server-version":"3.6.1","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"d0113665f403934f","remote-peer-server-version":"3.6.1","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"47c17342e3a1b4ec"}
kubectl logs etcd-eewrxk-etcd-3 --tail 20 kbagent 
2025-09-03T09:06:20Z	INFO	create service Action	{"actions": "switchover,memberJoin,memberLeave,dataDump,dataLoad,roleProbe"}
2025-09-03T09:06:20Z	INFO	create service Probe	{"probes": "roleProbe"}
2025-09-03T09:06:20Z	INFO	create service Streaming	{"actions": "dataDump,dataLoad"}
2025-09-03T09:06:20Z	INFO	service Action started...
2025-09-03T09:06:20Z	INFO	service Probe started...
2025-09-03T09:06:20Z	INFO	service Streaming started...
2025-09-03T09:06:20Z	INFO	starting the HTTP server
2025-09-03T09:06:20Z	INFO	register service to server	{"service": "Action", "method": "POST", "uri": "/v1.0/action"}
2025-09-03T09:06:20Z	INFO	register service to server	{"service": "Probe", "method": "POST", "uri": "/v1.0/probe"}
2025-09-03T09:06:20Z	INFO	register service to server	{"service": "Streaming", "method": "POST", "uri": "/v1.0/streaming"}
2025-09-03T09:06:20Z	INFO	probe started	{"probe": "roleProbe", "config": {"instance":"etcd-eewrxk-etcd","action":"roleProbe"}}
2025-09-03T09:06:20Z	INFO	starting the streaming server
2025-09-03T09:06:20Z	INFO	send probe event	{"probe": "roleProbe", "probe": "roleProbe", "code": 0, "output": "", "message": ""}
2025-09-03T09:06:20Z	INFO	Action Executed	{"action": "memberJoin", "result": ""}
2025-09-03T09:06:20Z	INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 13}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions