🐛 BUG: Random failure and "handshake timeout" spam

[SleepyLeslie]

What version of nebula are you using? (nebula -version)

1.10.0

What operating system are you using?

Flatcar Container Linux

Describe the Bug

This has happened quite a few times recently and I believe it is a Nebula bug.

I have a Kubernetes cluster running on top of a Nebula network. Nodes would randomly fall out the network and become partitioned until I manually restart the Nebula process, when it resumes normal operation. This renders my cluster extremely unstable and is very frustrating.

Might be related: I have used Nebula for less critical tasks before, and observed the same issue, especially on my laptop running Arch Linux. When I put the laptop to S3 suspend and resume later, the chance of Nebula failing is very high. Even when I do not suspend my laptop, it randomly fails every now and then. My servers running Ubuntu 24.04 have the same issue, so I believe it is distro agnostic.

When this happens, the logs show Nebula stuck indefinitely trying to re-establish handshake with a specific node. When I restart Nebula manually, it works again instantaneously so I am sure there is no underlying network issue. The node it fails to handshake with (in this case 10.18.1.3) is also perfectly reachable after the restart.

Logs from affected hosts

The logs are taken from the most recent failure, on the node where the failure occurred. You can see a massive chunk of "Handshake timed out" messages towards the end of the logs, before I restarted the service with systemctl.

Note: 10.0.2.207 is a Cilium address and 10.10.12.105 is the node's public-facing IP address (the node is behind NAT and that is the main reason why Nebula is used).

nebula.log

Config files from affected hosts

firewall:
  inbound:
    - host: any
      port: any
      proto: icmp
    - group: sysadmin
      port: any
      proto: any
    - group: k8s_node
      port: any
      proto: any
  inbound_action: reject
  outbound:
    - host: any
      port: any
      proto: any
  outbound_action: reject
handshakes:
  retries: 30
  try_interval: 1500
lighthouse:
  am_lighthouse: false
  hosts:
    - 10.18.0.1
  interval: 10
listen:
  host: 0.0.0.0
  port: 0
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  disconnect_invalid: true
  key: /etc/nebula/host.key
punchy:
  punch: true
  respond: true
relay:
  am_relay: false
  relays:
    - 10.18.0.1
static_host_map:
  10.18.0.1: [REDACTED]:4242
tun:
  dev: neb0
  mtu: 1300
  unsafe_routes:
    - install: true
      mtu: 1300
      route: 192.168.64.43/32
      via:
        - gateway: 10.18.1.1
        - gateway: 10.18.1.2
        - gateway: 10.18.1.3

[jekatigr]

I have the same issue. Nebula + cilium + k3s. After 4-5 days nebula connect breaks and reconnects only after restart on both sides.

[JackDoan]

Is there any chance (for either of y'all) that this is a NAT issue? listen.port: 0 + "a restart instantly fixes things" would make me want to investigate that, just in case.

Presumably the connection break you see occurs after some amount of time with the connection idle, not under load, right?

[jekatigr]

Is there any chance (for either of y'all) that this is a NAT issue?

Not sure how to check that. In my case nebula connects 2 servers: a vps (has a public ip) and on my home server (LAN).

Presumably the connection break you see occurs after some amount of time with the connection idle, not under load, right?

I don't think so. k3s constantly sends healthprobes to the worker node, so nebula overlay is utilized all the time.

Extracted boot-lose connection-reboot logs: logs.txt

My lighthouse config

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  key: /etc/nebula/host.key
static_host_map:
lighthouse:
  am_lighthouse: true
  interval: 60
listen:
  host: "0.0.0.0"
  port: 18767
punchy:
  punch: true
relay:
  am_relay: false
  use_relays: true
tun:
  disabled: false
  dev: nebula1
  drop_local_broadcast: false
  drop_multicast: false
  tx_queue: 500
  mtu: 1300
  routes:
  unsafe_routes:
logging:
  level: info
  format: text
firewall:
  outbound_action: drop
  inbound_action: drop
  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m
  outbound:
    - port: any
      proto: any
      host: any
  inbound:
    - port: any
      proto: icmp
      host: any
    - port: 10250
      proto: tcp
      host: any
    - port: 8472
      proto: udp
      host: any
    - port: 4240
      proto: udp
      host: any

[jekatigr]

Has happened again after 5 days of standby, just keep sending unsuccessful handshakes:

May 26 12:31:47 eagle nebula[247189]: time="2026-05-26T12:31:47Z" level=info msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=158939836 localIndex=158939836 remoteIndex=0 udpAddrs="[remote_lighthouse_ip:port]" vpnAddrs="[10.100.100.5]"
May 26 12:31:54 eagle nebula[247189]: time="2026-05-26T12:31:54Z" level=info msg="Handshake timed out" durationNs=6397639393 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=158939836 localIndex=158939836 remoteIndex=0 udpAddrs="[remote_lighthouse_ip:port]" vpnAddrs="[10.100.100.5]"

Just after restart (on the node itself, not on lighthouse) it was able to connect to remote lighthouse again.

I added another node in the same LAN and it lost connection just after nebula restart on the first one (not lighthouse, same LAN). It's very weird. Nebula restart also resolved it.