High CPU Spike in NATS Server when Leafnode Clients reach CPU Saturation [v2.12.2]

[yoonseo-han]

Observed behavior

We are running a NATS cluster where multiple Leafnodes subscribe to all subjects, leading to high fan-out traffic. We observed that when the client-side server (hosting the Leafnodes) reaches CPU saturation (100%+ utilization), the connected NATS Core server's CPU usage remains stable for a while and then suddenly spikes to ~99%.

Despite the CPU spike, the NATS server continues to function without immediate performance degradation or message loss. Profiling shows a massive increase in runtime.memclrNoHeapPointers and klauspost/compress/s2 related functions during this spike.

Expected behavior

We expected that the CPU saturation on the client-side would lead to backpressure but it should not cause the NATS Core server's CPU to spike to near 100% capacity.

Server and client version

NATS Server: v2.12.2

Host environment

NATS Core: 3-node cluster on AWS instances.
Client Host: Single m6i.large instance running multiple Leafnode instances.

Steps to reproduce

1. Deploy a NATS Core cluster.
2. Launch multiple Leafnode instances on a single client host, all subscribing to a high-throughput stream (simulating a full market data feed).
3. Induce CPU saturation on the client host by increasing the number of Leafnodes or adding artificial CPU load.
4. Observe the NATS Core server CPU utilization after the client host hits 100% CPU.

[yoonseo-han]

Profiling result during NATS server's CPU ~99% utilization

[sysadmin@ip-10-0-1-101 after]$ go tool pprof ./cpu-20260204-082002-nats1
File: nats-server
Build ID: 7847d75fa607456e4492f797c23844fa428eba2b
Type: cpu
Time: 2026-02-04 08:19:46 UTC
Duration: 15.16s, Total samples = 24.09s (158.94%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 15720ms, 65.26% of 24090ms total
Dropped 364 nodes (cum <= 120.45ms)
Showing top 10 nodes out of 142
      flat  flat%   sum%        cum   cum%
    7800ms 32.38% 32.38%     7800ms 32.38%  runtime.memclrNoHeapPointers
    2680ms 11.12% 43.50%     2680ms 11.12%  internal/runtime/syscall.Syscall6
    1450ms  6.02% 49.52%     3690ms 15.32%  runtime.scanobject
     930ms  3.86% 53.38%     9440ms 39.19%  github.com/klauspost/compress/s2.encodeBlockBest
     860ms  3.57% 56.95%     1020ms  4.23%  runtime.findObject
     710ms  2.95% 59.90%    15220ms 63.18%  github.com/nats-io/nats-server/v2/server.(*client).flushOutbound
     400ms  1.66% 61.56%      400ms  1.66%  runtime.memmove
     320ms  1.33% 62.89%      320ms  1.33%  internal/sync.(*Mutex).Lock (inline)
     310ms  1.29% 64.18%      310ms  1.29%  runtime.(*mspan).heapBitsSmallForAddr
     260ms  1.08% 65.26%      550ms  2.28%  internal/sync.(*HashTrieMap[go.shape.interface {},go.shape.interface {}]).Load
(pprof) top -cum
Showing nodes accounting for 0.82s, 3.40% of 24.09s total
Dropped 364 nodes (cum <= 0.12s)
Showing top 10 nodes out of 142
      flat  flat%   sum%        cum   cum%
         0     0%     0%     19.56s 81.20%  github.com/nats-io/nats-server/v2/server.(*Server).startGoRoutine.func1
     0.02s 0.083% 0.083%     15.55s 64.55%  github.com/nats-io/nats-server/v2/server.(*client).writeLoop
     0.71s  2.95%  3.03%     15.22s 63.18%  github.com/nats-io/nats-server/v2/server.(*client).flushOutbound
         0     0%  3.03%     14.06s 58.36%  github.com/nats-io/nats-server/v2/server.(*Server).createLeafNode.func3
         0     0%  3.03%     10.30s 42.76%  github.com/klauspost/compress/s2.(*Writer).Close (inline)
         0     0%  3.03%     10.30s 42.76%  github.com/klauspost/compress/s2.(*Writer).closeIndex
         0     0%  3.03%     10.27s 42.63%  github.com/klauspost/compress/s2.(*Writer).Flush
         0     0%  3.03%     10.23s 42.47%  github.com/klauspost/compress/s2.(*Writer).AsyncFlush
     0.03s  0.12%  3.15%     10.21s 42.38%  github.com/klauspost/compress/s2.(*Writer).writeSync
     0.06s  0.25%  3.40%      9.91s 41.14%  runtime.mallocgc

[wallyqs]

do you have leafnode compression enabled or have tried disabling it?

[wallyqs]

There is a fix for allocation issues that were happening when compression was enabled, this should ameliorate things a bit: #7901 To be released in v2.12.5

[yoonseo-han]

Thank you for the followup.

We’ll retest with this fix once v2.12.5 is available and compare the behavior against our current results.