Multus: [hostpath-provisioner/hostpath-provisioner-operator-...]: error getting pod: pods "hostpath-provisioner-operator-..." is forbidden: User "system:serviceaccount:kube-system:multus" cannot get resource "pods" in API group "" in the namespace "hostpath-provisioner"

[jbpratt]

On reboot, my node was failing to bring up the hostpath-provisioner and kube-cni-linux-bridge pods due to a change in the Multus clusterrolebinding. I'm not sure why it is happening but others have ran into this before k8snetworkplumbingwg/multus-cni#667

❯ oc get pods -A
NAMESPACE                NAME                                                  READY   STATUS              RESTARTS   AGE
cdi                      cdi-apiserver-78bcbcc8ff-768lf                        1/1     Running             2          4d5h
cdi                      cdi-deployment-6ccdf4fb64-qj6m4                       1/1     Running             2          4d5h
cdi                      cdi-operator-54d5bbbdd9-mhzcj                         0/1     Completed           1          4d5h
cdi                      cdi-uploadproxy-649757bfb5-kjdbh                      1/1     Running             2          4d5h
cert-manager             cert-manager-57d89b9548-f4w6n                         1/1     Running             2          4d5h
cert-manager             cert-manager-cainjector-5bcf77b697-q9g8d              0/1     Completed           1          4d5h
cert-manager             cert-manager-webhook-9cb88bd6d-ks6qf                  1/1     Running             2          4d5h
cluster-network-addons   bridge-marker-c9vcb                                   1/1     Running             2          4d5h
cluster-network-addons   cluster-network-addons-operator-549b8f8966-rbxmf      0/1     Completed           1          4d5h
cluster-network-addons   kube-cni-linux-bridge-plugin-4jhc4                    0/1     Error               1          4d5h
cluster-network-addons   kubemacpool-cert-manager-68f745946c-jjx8h             0/1     Completed           1          4d5h
cluster-network-addons   kubemacpool-mac-controller-manager-868f5c6946-jrj9s   1/1     Running             2          4d5h
cluster-network-addons   macvtap-cni-wwc95                                     1/1     Running             2          4d5h
cluster-network-addons   multus-pgfff                                          1/1     Running             2          4d5h
cluster-network-addons   nmstate-cert-manager-748d47479f-7thlt                 0/1     Completed           1          4d5h
cluster-network-addons   nmstate-handler-kt5zf                                 1/1     Running             2          4d5h
cluster-network-addons   nmstate-webhook-7c56958777-4k6wf                      1/1     Running             2          4d5h
cluster-network-addons   nmstate-webhook-7c56958777-bhssb                      1/1     Running             2          4d5h
cluster-network-addons   ovs-cni-amd64-xhrhv                                   1/1     Running             2          4d5h
hostpath-provisioner     hostpath-provisioner-j6qw6                            0/1     Error               1          4d5h
hostpath-provisioner     hostpath-provisioner-operator-b8bf65759-rjmhf         0/1     Completed           1          4d5h
kube-system              calico-kube-controllers-8575b76f66-pvvmm              1/1     Running             2          4d5h
kube-system              calico-node-9xhpj                                     1/1     Running             2          4d5h
kube-system              coredns-8474476ff8-s8tw7                              1/1     Running             2          4d5h
kube-system              kube-apiserver-node1                                  1/1     Running             2          4d5h
kube-system              kube-controller-manager-node1                         1/1     Running             2          4d5h
kube-system              kube-multus-ds-amd64-8pl5l                            1/1     Running             2          4d5h
kube-system              kube-multus-ds-dwrs5                                  1/1     Running             2          4d5h
kube-system              kube-proxy-8595j                                      1/1     Running             2          4d5h
kube-system              kube-scheduler-node1                                  1/1     Running             2          4d5h
kube-system              nodelocaldns-t2twd                                    1/1     Running             2          4d5h
kubevirt                 virt-api-794854d7f4-4zk98                             1/1     Running             2          4d5h
kubevirt                 virt-api-794854d7f4-shsh6                             1/1     Running             2          4d5h
kubevirt                 virt-controller-974f9b54d-24kbl                       1/1     Running             2          4d5h
kubevirt                 virt-controller-974f9b54d-vwg99                       1/1     Running             2          4d5h
kubevirt                 virt-handler-v4sxv                                    1/1     Running             2          4d5h
kubevirt                 virt-operator-5c69b784bc-4bcnr                        1/1     Running             2          4d5h
kubevirt                 virt-operator-5c69b784bc-fsbc9                        1/1     Running             2          4d5h

with the event of

20m         Warning   FailedCreatePodSandBox   pod/hostpath-provisioner-operator-bd4966b44-d6cm4    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_hostpath-provisioner-operator-bd4966b44-d6cm4_hostpath-provisioner_63b16bb7-626f-426b-8beb-d90f7c7b29d0_0(ef47ddf1e279a20bf3c914129f1b52cf5eddc1f88c5fb882570db79b33046cd2): Multus: [hostpath-provisioner/hostpath-provisioner-operator-bd4966b44-d6cm4]: error getting pod: pods "hostpath-provisioner-operator-bd4966b44-d6cm4" is forbidden: User "system:serviceaccount:kube-system:multus" cannot get resource "pods" in API group "" in the namespace "hostpath-provisioner"

Following along with the ticket, it does seem that the namespace for the multus SA has been changed to cluster-network-addons

~
❯ kubectl get clusterrolebinding multus -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"multus"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"multus"},"subjects":[{"kind":"ServiceAccount","name":"multus","namespace":"kube-system"}]}
  creationTimestamp: "2021-11-06T14:44:55Z"
  labels:
    app.kubernetes.io/component: network
    app.kubernetes.io/managed-by: Helm
    networkaddonsoperator.network.kubevirt.io/version: 0.58.2
    prometheus.cnao.io: ""
  name: multus
  ownerReferences:
  - apiVersion: networkaddonsoperator.network.kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: NetworkAddonsConfig
    name: cluster
    uid: f9bd9f09-4c28-48eb-8bd7-0172b9d8c0ef
  resourceVersion: "2132"
  uid: 2c536bd8-810d-41f0-b810-3c24bf434eb2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus
subjects:
- kind: ServiceAccount
  name: multus
  namespace: cluster-network-addons

editing the value from cluster-network-addons -> kube-system allows the pods to be created

~
❯ oc get events -n hostpath-provisioner --sort-by=.metadata.creationTimestamp
LAST SEEN   TYPE      REASON                   OBJECT                                               MESSAGE
9m17s       Normal    SandboxChanged           pod/hostpath-provisioner-j6qw6                       Pod sandbox changed, it will be killed and re-created.
9m18s       Normal    SandboxChanged           pod/hostpath-provisioner-operator-b8bf65759-rjmhf    Pod sandbox changed, it will be killed and re-created.
27m         Normal    ScalingReplicaSet        deployment/hostpath-provisioner-operator             Scaled up replica set hostpath-provisioner-operator-bd4966b44 to 1
27m         Normal    SuccessfulCreate         replicaset/hostpath-provisioner-operator-bd4966b44   Created pod: hostpath-provisioner-operator-bd4966b44-d6cm4
27m         Normal    Scheduled                pod/hostpath-provisioner-operator-bd4966b44-d6cm4    Successfully assigned hostpath-provisioner/hostpath-provisioner-operator-bd4966b44-d6cm4 to node1
...
26m         Warning   FailedCreatePodSandBox   pod/hostpath-provisioner-operator-bd4966b44-d6cm4    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_hostpath-provisioner-operator-bd4966b44-d6cm4_hostpath-provisioner_63b16bb7-626f-426b-8beb-d90f7c7b29d0_0(0112e752706fd4205779caea7dac919f26926e70a9d77e8ed9d78192c163745c): Multus: [hostpath-provisioner/hostpath-provisioner-operator-bd4966b44-d6cm4]: error getting pod: pods "hostpath-provisioner-operator-bd4966b44-d6cm4" is forbidden: User "system:serviceaccount:kube-system:multus" cannot get resource "pods" in API group "" in the namespace "hostpath-provisioner"
...
7m5s        Warning   FailedCreatePodSandBox   pod/hostpath-provisioner-operator-bd4966b44-d6cm4    (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_hostpath-provisioner-operator-bd4966b44-d6cm4_hostpath-provisioner_63b16bb7-626f-426b-8beb-d90f7c7b29d0_0(3d22dd7351133fe957b342b7a504bdac85c068169411c8ced07742c990b331a0): Multus: [hostpath-provisioner/hostpath-provisioner-operator-bd4966b44-d6cm4]: error getting pod: pods "hostpath-provisioner-operator-bd4966b44-d6cm4" is forbidden: User "system:serviceaccount:kube-system:multus" cannot get resource "pods" in API group "" in the namespace "hostpath-provisioner"
6m26s       Normal    AddedInterface           pod/hostpath-provisioner-operator-b8bf65759-rjmhf    Add eth0 [10.233.90.98/32] from cni0
6m23s       Normal    AddedInterface           pod/hostpath-provisioner-j6qw6                       Add eth0 [10.233.90.101/32] from cni0
6m22s       Normal    SuccessfulDelete         replicaset/hostpath-provisioner-operator-b8bf65759   Deleted pod: hostpath-provisioner-operator-b8bf65759-rjmhf
6m22s       Normal    AddedInterface           pod/hostpath-provisioner-operator-bd4966b44-d6cm4    Add eth0 [10.233.90.102/32] from cni0
6m22s       Normal    ScalingReplicaSet        deployment/hostpath-provisioner-operator             Scaled down replica set hostpath-provisioner-operator-b8bf65759 to 0

~
❯ oc get pods -n hostpath-provisioner
NAME                                            READY   STATUS    RESTARTS   AGE
hostpath-provisioner-j6qw6                      1/1     Running   1          4d5h
hostpath-provisioner-operator-bd4966b44-d6cm4   1/1     Running   0          28m

Based on the response from the issue, this may be due to how we are installing things? Just wanted to report this with a patch in case others run into this. 🐱

[jbpratt]

Nope, not a fix, never mind. While that did allow those pods to come up if restarted fast enough, this gets written over by whatever is managing it.

[usrbinkat]

Right now I'm trying to lift several things up into a more sustainable code breakout model.

This is the source of truth as all this gets stabilized out for the 100DaysOfHomelab vlog series.

Multus has been a sticking point, the workaround isnt pretty yet but it is falling inline with this deployment method.

This Kargo repo will be pared down based on how that dust settles and multus likely will be cut out of this repo.