[Bug]: Dnsmasq can still become non-responsive while pod remains Running

[kachunkachunk]

Holorouter Version

9.0.2

VCF Version

VCF 9.0.1.0

Target Type

vCenter Cluster

Describe the bug

In Holorouter, dnsmasq might eventually become unresponsive. For example, doing a lookup of ops-a.site-a.vcf.lab may time out, even though dnsmasq-dns pods remain Running and service endpoints still exist.

Lookups to the holorouter front-end IP and the dns-service pod/cluster IPs (on the holorouter) all time out. This usually breaks VCF component name resolution, until manual intervention.

I actually have a zone delegation and a static route rule set up on my router, so my machines can reach vcf.lab without depending on the webtop or anything fancy.

When this issue occurs, from the LAN, I see connection refused against this delegated DNS server, upon trying to do lookups. Internal on the holorouter, I just see timeouts.

This has the appearance of prior AAAA/upstream issues. However, filter-AAAA is already present in current configs.

Reproduction steps

I don't yet have a deterministic repro sequence, but I suspect it could be a result of a performance-related pressure, or infrequent upstream DNS failures. I speculate that, because there have been logged issues which pointed at "unreliable upstream DNS."

With that said, I think dnsmasq in the Holorouter's architecture could be a lot more resilient, anyway. In practice, the issue appears occasionally during normal operation, long after a successful (and possibly intensive) extended period of deploying.

Observed checks during failure:
kubectl get pods -n default -l app=dnsmasq-dns shows pods Running
kubectl get endpoints -n default dns-service shows valid pod IPs
nslookup ops-a.site-a.vcf.lab <management IP> times out
nslookup ops-a.site-a.vcf.lab <dns-service-IP or clusterIP> times out

Workaround that sometimes works:
Reapply/delete/reapply dnsmasq manifests, or restart/delete dnsmasq pods. This usually brings it all back. My configmap is never empty/blank, nor do I have issues with AAAA filtering being lost.

Expected behavior

If dnsmasq becomes unhealthy or stops answering queries, Kubernetes should detect this with an appropriate liveliness/health check, and self-heal. Perhaps remove bad endpoints and restart pods.

DNS should remain available for delegated/internal records without requiring manual reapply/delete/apply cycles.

Relevant logs

Unfortunately, the issue is opaque and runtime/deployment/pod logs have proven to be pretty unhelpful so far.

Additional context

Small note, and probably nothing, but I deployed 9.0.1 to reproduce an issue I have been troubleshooting in VCF Ops. I have since upgraded to 9.0.2 via VCF Ops. I've encountered this bug once during part of an NSX precheck, and again long after while configuring log integration stuff. It isn't easily relatable to load of the underlying host, or anything like that.

[knikhilm]

How many dnsmasq pods do you see on the Holorouter? Please share the output of - netstat -tulpn | grep 53

[kachunkachunk]

Hey There - here you go, plus a couple more kubectl outputs, if they may help.

root@holorouter [ ~ ]# netstat -tulpn | grep 53
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           711/systemd-resolve
udp6       0      0 :::5353                 :::*                                711/systemd-resolve
udp6       0      0 :::55384                :::*                                960/rpc.mountd

root@holorouter [ ~ ]# kubectl get pods -A
NAMESPACE      NAME                                      READY   STATUS    RESTARTS       AGE
default        dnsmasq-dhcp-deployment-c9bcb98b6-n5nvw   1/1     Running   0              4h16m
default        dnsmasq-dns-deployment-66d9fbfb9b-6cctv   1/1     Running   0              3h41m
default        dnsmasq-dns-deployment-66d9fbfb9b-b4kgw   1/1     Running   0              3h41m
default        dnsmasq-dns-deployment-66d9fbfb9b-w6bnw   1/1     Running   0              3h41m
default        frr-k8s-jdc6t                             7/7     Running   14 (27h ago)   2d4h
default        frr-k8s-webhook-server-5c6bd54d4-sp2cj    1/1     Running   2 (27h ago)    2d4h
default        webtop-78cd7b4bb4-448g9                   1/1     Running   2 (27h ago)    2d5h
kube-flannel   kube-flannel-ds-vz8lk                     1/1     Running   2 (27h ago)    2d5h
kube-system    coredns-5db6bdb8c7-fmzss                  1/1     Running   2 (27h ago)    2d4h
kube-system    coredns-5db6bdb8c7-jz294                  1/1     Running   2 (27h ago)    2d4h
kube-system    etcd-holorouter                           1/1     Running   2 (27h ago)    2d5h
kube-system    kube-apiserver-holorouter                 1/1     Running   2 (27h ago)    2d5h
kube-system    kube-controller-manager-holorouter        1/1     Running   21 (27h ago)   2d5h
kube-system    kube-proxy-2gbhv                          1/1     Running   0              3h41m
kube-system    kube-scheduler-holorouter                 1/1     Running   21 (27h ago)   2d5h

root@holorouter [ ~ ]# kubectl get deployments -A
NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
default       dnsmasq-dhcp-deployment   1/1     1            1           4h17m
default       dnsmasq-dns-deployment    3/3     3            3           4h17m
default       frr-k8s-webhook-server    1/1     1            1           2d4h
default       webtop                    1/1     1            1           2d5h
kube-system   coredns                   2/2     2            2           2d5h

[knikhilm]

Have you configured dnsmasq as the DNS forwarder for multiple JHs or machines outside Holodeck?

Also share the output of the following commands -

kubectl get svc

for pod in $(kubectl get pods -l app=dnsmasq-dns -o name); do
    echo "Executing in $pod..."
    kubectl exec -it $pod -- netstat -tulpn | grep 53
done

[kachunkachunk]

If I understood your question correctly, I have not configured dnsmasq in the holorouter for other zones or... jump hosts? I think you're asking that, anyway.

root@holorouter [ ~ ]# kubectl get svc
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP                      PORT(S)         AGE
dns-service               ClusterIP   10.106.56.13     192.168.224.1,                   53/TCP,53/UDP   3d20h
                                                       192.168.225.1,
                                                       192.168.226.1,192.168.226.129,
                                                       192.168.227.1,192.168.227.129,
                                                       192.168.228.1,192.168.228.129,
                                                       192.168.229.1,192.168.229.129,
                                                       192.168.230.1,192.168.230.129,
                                                       192.168.231.1,192.168.231.129,
                                                       192.168.232.1,192.168.232.129,
                                                       192.168.233.1,192.168.234.129,
                                                       172.16.3.1
frr-k8s-webhook-service   ClusterIP   10.102.120.175   <none>                           443/TCP         3d20h
kubernetes                ClusterIP   10.96.0.1        <none>                           443/TCP         3d20h
webtop                    NodePort    10.108.40.237    <none>

(line-wraps included for readability)

I figure it would be substantially more helpful to get both of these outputs during a recurrence. I'll be sure to come back with it, if something happens again.

root@holorouter [ ~ ]# for pod in $(kubectl get pods -l app=dnsmasq-dns -o name); do
    echo "Executing in $pod..."
    kubectl exec -it $pod -- netstat -tulpn | grep 53
done
Executing in pod/dnsmasq-dns-deployment-66d9fbfb9b-6cctv...
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      8/dnsmasq
tcp        0      0 :::53                   :::*                    LISTEN      8/dnsmasq
udp        0      0 0.0.0.0:53              0.0.0.0:*                           8/dnsmasq
udp        0      0 :::53                   :::*                                8/dnsmasq
Executing in pod/dnsmasq-dns-deployment-66d9fbfb9b-b4kgw...
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      6/dnsmasq
tcp        0      0 :::53                   :::*                    LISTEN      6/dnsmasq
udp        0      0 0.0.0.0:53              0.0.0.0:*                           6/dnsmasq
udp        0      0 :::53                   :::*                                6/dnsmasq
Executing in pod/dnsmasq-dns-deployment-66d9fbfb9b-w6bnw...
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      8/dnsmasq
tcp        0      0 :::53                   :::*                    LISTEN      8/dnsmasq
udp        0      0 0.0.0.0:53              0.0.0.0:*                           8/dnsmasq
udp        0      0 :::53                   :::*                                8/dnsmasq