Unbound starts to drop queries when caches are nearly full

[rygl]

Describe the bug
Unbound starts to drop some queries and increase requestlist.exceeded counters when caches are full or nearly full.

To reproduce
Steps to reproduce the behavior:

1. Unbound 1.24.2-1, running in a VM with 12 cores for couple of days. 12 threads configured. Validating resolver, with rather small RPZ. Acting as a backend in a pool behind a loadbalancer. A standard load of about 1k qps in ISP environment. Cache hitrate about 20% due to cache on the balancer. No excessive load peaks, no silly traffic, no bad clients. No susspicious queries in request list.

2. When Unbound is restarted and everything is fresh it is fine for couple of days. I can see dropped queries just in case there is a real issue with upstream authoritative NS and SERVFAIL is not cached yest. Ater some days I can see random drops of a queries in rate of about 0.5% and increasing even for domains that are clearly without any issues (google.com, apple.com or any other frequently used). I can also see that counters requestlist.exceeded are going up for some threads. The number of events here seems to be corresponding with amount of dropped queries.
   Drops are confirmed by a packet capture directly on the VM.

When analyzing this I have noticed that the moment when Unbound starts to drops roughly corresponds with the moment when configured caches are full or nearly full (Grafana in use). Until this moment everything seems to be fine.

3. The cache size does not matter here. Neither amount of cache slabs. I can reproduce this with default slab values and these caches:

        num-threads: 12
        num-queries-per-thread: 4096

        msg-cache-size:    128M
        rrset-cache-size:  2G
        key-cache-size:    256M
        neg-cache-size:    64M

        infra-cache-numhosts: 1000000
        infra-cache-max-rtt:  4000
        max-query-restarts:   7

and also with following settings:

        msg-cache-size:    8G
        rrset-cache-size:  16G
        key-cache-size:    2G
        neg-cache-size:    512M

        msg-cache-slabs:   64
        rrset-cache-slabs: 64
        infra-cache-slabs: 64
        key-cache-slabs:   64

and even with this settings:

        msg-cache-size:    8G
        rrset-cache-size:  16G
        key-cache-size:    2G
        neg-cache-size:    512M

        msg-cache-slabs:   32
        rrset-cache-slabs: 64
        infra-cache-slabs: 16
        key-cache-slabs:   16

4. This issue is fully reproducible on all the Unbound servers in the pool - 11 in total . Normal "drop rate" is up to 0.2 qps, when this starts to appear it goes over 1-1.5 qps.

Expected behavior
No drops

System:

• Unbound version: Debian package 1.24.2-1
• OS: Debian Trixie

# unbound -V
Version 1.24.2

Configure line: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --with-pythonmodule --with-pyunbound --enable-subnet --enable-dnstap --enable-systemd --enable-cachedb --with-libhiredis --with-libnghttp2 --with-chroot-dir= --with-dnstap-socket-path=/run/dnstap.sock --disable-rpath --with-pidfile=/run/unbound.pid --with-libevent --enable-tfo-client --with-rootkey-file=/usr/share/dns/root.key --enable-tfo-server
Linked libs: libevent 2.1.12-stable (it uses epoll), OpenSSL 3.5.4 30 Sep 2025
Linked modules: dns64 python cachedb subnetcache respip validator iterator
TCP Fastopen feature available

Additional information
I am runnig also Knot resolver in the pool of servers where I do not see this issue. The more cache is configured the more time it takes until this issue appears. I can provide a graphical stats of drops in time, cache usage as well as the full config. I could see this also in previous versions of Unbound, so I have upgraded to lastest and it still persists.

[gthess]

Hi @rygl,

Some initial questions to get them out of the way:

• Is maybe swap involved?
• Since it is behind a load balancer, and I assume dnsdist, is so-reuseport: enabled (yes by default)? In that case check the option sockets in https://www.dnsdist.org/reference/config.html#newServer. You may need to configure that to get even distribution among threads in Unbound.
• How do you measure the drops? Packets that got no answer in the capture? Is that per DNS ID or query name?
• Could you share a configuration that is the fastest to reach the dropping condition?

Some things to try:

• udp-connect: no could help if there are a lot of outgoing udp sockets (num-threads: 12 * num-queries-per-thread: 4096).
• Since there is a single client talking to Unbound and is secure , you can configure wait-limit: 0 and discard-timeout: 0 to disable some counter measures for wild clients.

[rygl]

Hi @rygl,

Some initial questions to get them out of the way:

* Is maybe swap involved?

No, I am running without a swap. There is 70GB RAM

* Since it is behind a load balancer, and I assume dnsdist, is [`so-reuseport:`](https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound.conf.html#unbound-conf-so-reuseport) enabled (yes by default)? In that case check the option `sockets` in https://www.dnsdist.org/reference/config.html#newServer. You may need to configure that to get even distribution among threads in Unbound.

You are right. All backends are sitting behind dnsdist. Each server in it's config uses 24 sockets. so-reuseport is set to true in Unbound.

* How do you measure the drops? Packets that got no answer in the capture? Is that per DNS ID or query name?

Dnsdist is reporting drops per backend. I am ploting them in Grafana - I have found out this way. And confirmed by a capture directly on the backend using display filter "dns.response_missing". I think wireshark uses DNS ID to detect this.

* Could you share a configuration that is the fastest to reach the dropping condition?

Yes, sure, I just need some time to remove sensitive data...

Some things to try:

* [`udp-connect: no`](https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound.conf.html#unbound-conf-udp-connect) could help if there are a lot of outgoing udp sockets (num-threads: 12 * num-queries-per-thread: 4096).

The number of queries per thread is rather oversized. Just to be able to handle potential peaks. I will try.

* Since there is a single client talking to Unbound and is secure , you can configure [`wait-limit: 0`](https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound.conf.html#unbound-conf-wait-limit) and `[discard-timeout: 0`](https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound.conf.html#unbound-conf-discard-timeout) to disable some counter measures for wild clients.

In fact, there are multiple dnsdist load balancers in our configuration that share a pool of Unbound servers. All af them have nearly identical config unsing sockets in server definition, etc.

[rygl]

Hi,

take a look at my configuration below. I would say there is nothing special about it. All backends are almost identical. The cache size is reduced in this case.

Ales
unbound.txt

[gthess]

In fact, there are multiple dnsdist load balancers in our configuration that share a pool of Unbound servers. All af them have nearly identical config unsing sockets in server definition, etc.

Still, there are a handful of secure clients that query Unbound, I would still advise to use wait-limit: 0 and discard-timeout: 0.

wait-limit can drop incoming queries and is reported as num.queries_wait_limit. Note: It used to be recorded as requestlist.exceeded from 1.20.0 up to and including 1.22.0.

discard-timeout can drop replies and is reported as num.queries_discard_timeout. Note: It used to be recorded as requestlist.exceeded from 1.20.0 up to and including 1.22.0.

We do have some extra comments but nothing that ties the cache full condition with the drop behavior.

• use-caps-for-id: yes, if engaged, leads to delays (because of retries) and probably resolution issues at the end. Still no relation to a full cache.

• outgoing-range: 4096, you can raise that value (double) to see if it helps? If the request list is full (from incoming queries), there may not be enough outgoing sockets to serve the accumulated queries (+Unbound generated ones). There could be a backlog of queries waiting for an available socket to proceed resolution. Still no relation to a full cache.

From what we know until now is that drops are happening because there is no space for incoming queries (i.e., requestlist.exceeded). Unbound already has logic to deal with this that you can see with requestlist.overwritten where it tries to replace "slow" queries with the new incoming ones. What is the value of the requestlist.overwritten statistic?

Other questions:

• Do you use any unbound-control reload commands or similar while Unbound is running? Or any unbound-control commands?
• When the caches are full, I assume that there is still plenty of memory on the system for Unbound to handle RPZ and auth-zone updates. Do you see maybe XFRs happening at the same time?
• I see you have a health check configured local-data. Is dnsdist reporting Unbound as down (I am not expecting that)?
• Do the numbers in threadX.num.queries seem somewhat evenly distributed?

[rygl]

Still, there are a handful of secure clients that query Unbound, I would still advise to use wait-limit: 0 and discard-timeout: 0.

No problem, I will try this on one of backends

wait-limit can drop incoming queries and is reported as num.queries_wait_limit. Note: It used to be recorded as requestlist.exceeded from 1.20.0 up to and including 1.22.0.

I have zeros here:

~# unbound-control stats_noreset | grep wait_limit
thread0.num.queries_wait_limit=0
thread1.num.queries_wait_limit=0
thread2.num.queries_wait_limit=0
thread3.num.queries_wait_limit=0
thread4.num.queries_wait_limit=0
thread5.num.queries_wait_limit=0
thread6.num.queries_wait_limit=0
thread7.num.queries_wait_limit=0
thread8.num.queries_wait_limit=0
thread9.num.queries_wait_limit=0
thread10.num.queries_wait_limit=0
thread11.num.queries_wait_limit=0
total.num.queries_wait_limit=0

discard-timeout can drop replies and is reported as num.queries_discard_timeout. Note: It used to be recorded as requestlist.exceeded from 1.20.0 up to and including 1.22.0.

discard-timeouts are non-zero here and are slowly increasing:

date; unbound-control stats_noreset | grep queries_discard_timeout
Wed Mar 11 12:33:26 PM CET 2026
thread0.num.queries_discard_timeout=72377
thread1.num.queries_discard_timeout=41242
thread2.num.queries_discard_timeout=55516
thread3.num.queries_discard_timeout=66286
thread4.num.queries_discard_timeout=33426
thread5.num.queries_discard_timeout=91248
thread6.num.queries_discard_timeout=51924
thread7.num.queries_discard_timeout=75285
thread8.num.queries_discard_timeout=29898
thread9.num.queries_discard_timeout=69946
thread10.num.queries_discard_timeout=64111
thread11.num.queries_discard_timeout=52126
total.num.queries_discard_timeout=703385
~# date; unbound-control stats_noreset | grep queries_discard_timeout
Wed Mar 11 12:33:30 PM CET 2026
thread0.num.queries_discard_timeout=72377
thread1.num.queries_discard_timeout=41242
thread2.num.queries_discard_timeout=55516
thread3.num.queries_discard_timeout=66286
thread4.num.queries_discard_timeout=33426
thread5.num.queries_discard_timeout=91249
thread6.num.queries_discard_timeout=51924
thread7.num.queries_discard_timeout=75285
thread8.num.queries_discard_timeout=29898
thread9.num.queries_discard_timeout=69946
thread10.num.queries_discard_timeout=64111
thread11.num.queries_discard_timeout=52127
total.num.queries_discard_timeout=703387
~#

We do have some extra comments but nothing that ties the cache full condition with the drop behavior.

* `use-caps-for-id: yes`, if engaged, leads to delays (because of retries) and probably resolution issues at the end. Still no relation to a full cache.

* `outgoing-range: 4096`, you can raise that value (double) to see if it helps? If the request list is full (from incoming queries), there may not be enough outgoing sockets to serve the accumulated queries (+Unbound generated ones). There could be a backlog of queries waiting for an available socket to proceed resolution. Still no relation to a full cache.

It could be just a coincidence of two things. I could not have find anything else related to the moment when drops appear. Unfortunately I have to wait two or three days until this issue comes in. If is is caused by a client queries I would see it nearly immediately after restart. But the server runs for three days or more.

I woud say that the server load is really low in terms of open sockets:

~# ss -s
Total: 440
TCP:   414 (estab 144, closed 216, orphaned 21, timewait 216)

Transport Total     IP        IPv6
RAW       0         0         0
UDP       104       80        24
TCP       198       169       29
INET      302       249       53
FRAG      0         0         0

~# cat /proc/net/sockstat
sockets: used 439
TCP: inuse 169 orphan 20 tw 92 alloc 199 mem 0
UDP: inuse 77 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
~

From what we know until now is that drops are happening because there is no space for incoming queries (i.e., requestlist.exceeded). Unbound already has logic to deal with this that you can see with requestlist.overwritten where it tries to replace "slow" queries with the new incoming ones. What is the value of the requestlist.overwritten statistic?

Zeros. All threads. This is confusing for me. See the graph attached.

~# date; unbound-control stats_noreset | grep requestlist.overwritten
Wed Mar 11 12:59:09 PM CET 2026
thread0.requestlist.overwritten=0
thread1.requestlist.overwritten=0
thread2.requestlist.overwritten=0
thread3.requestlist.overwritten=0
thread4.requestlist.overwritten=0
thread5.requestlist.overwritten=0
thread6.requestlist.overwritten=0
thread7.requestlist.overwritten=0
thread8.requestlist.overwritten=0
thread9.requestlist.overwritten=0
thread10.requestlist.overwritten=0
thread11.requestlist.overwritten=0
total.requestlist.overwritten=0

Other questions:

* Do you use any `unbound-control reload` commands or similar while Unbound is running? Or any `unbound-control` commands?

No, there is just a telegraf service getting counters:

[[inputs.unbound]]
   server = "127.0.0.1:8953"
   use_sudo = true
   thread_as_tag = true

* When the caches are full, I assume that there is still plenty of memory on the system for Unbound to handle RPZ and auth-zone updates. Do you see maybe XFRs happening at the same time?

Memory is not an issue. The VM have ~70GiB. The RPZ or local zone updates are not frequent, maybe once a day.

* I see you have a health check configured local-data. Is dnsdist reporting Unbound as down (I am not expecting that)?

No, not at all. It just reports increased number of drops for regular queries.

* Do the numbers in `threadX.num.queries` seem somewhat evenly distributed?

Yes, queries are rather evenly distributed:

~# unbound-control stats_noreset | egrep num.queries=
thread0.num.queries=38347053
thread1.num.queries=30831251
thread2.num.queries=37416400
thread3.num.queries=44464431
thread4.num.queries=29033109
thread5.num.queries=47861800
thread6.num.queries=31294358
thread7.num.queries=43378725
thread8.num.queries=18480802
thread9.num.queries=35355680
thread10.num.queries=36865260
thread11.num.queries=30344570
total.num.queries=423673439

In order to ilustrate that there maybe a relation between exceeded requestlist events and number of drops reported by dnsdist see the picture below. Drops are scaled by 200.000 so that the thedn is more visible. Note, there are always some drops. But the trend goes up here...

Is there anything what I can grab for you when I have a backed which starts to drop? I plan to restart the others with your recommended options.

Ales

[rygl]

An example of an unanswered query which was checked manually couple of seconds later:

~# dig a  @127.0.0.1 api.tapas.io

; <<>> DiG 9.20.18-1~deb13u1-Debian <<>> a @127.0.0.1 api.tapas.io
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25111
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;api.tapas.io.                  IN      A

;; ANSWER SECTION:
api.tapas.io.           60      IN      A       35.166.0.191
api.tapas.io.           60      IN      A       54.213.175.44

;; Query time: 16 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Wed Mar 11 13:44:51 CET 2026
;; MSG SIZE  rcvd: 73

[gthess]

Still pocking at things here but can you turn prefetch off? Since you have a frontend to Unbound, this shouldn't provide anything more. Probably not related but I was going through your config once more.

High 'exceeded' number without 'overwritten' means that the request list is full with recent queries, and seems to point that Unbound is being more busy than usual while resolving the current request list.

Another thought, does anything stand out wrt to the following statistics during the incident?

• num.answer.bogus
• num.rrset.bogus
• num.valops

Maybe there is a connection with Unbound retrying validation.

In the last plot, I am assuming that if you plotted the cache size as well, it would start to plateau?

As for the dropped queries, maybe some of them could have something to do with traffic but generally, when the request list is full, you will get drops regardless of the query itself.

Also in general Unbound will try hard to find an answer so the drops you are seeing casually (if we ignore this issue for a moment) is Unbound not having replied to a specific DNS ID in the client's waiting window. Unbound would reply to a client only if there is a definite answer to give, not with early error codes.
An exception to this is serve-expired (but you are not using that).

I will try to see if I can reproduce locally the behavior you are seeing.

[edmonds]

1.24.2 does not have this fix: 0a15118. You would want to set discard-timeout: 0 to work around it.

[rygl]

Still pocking at things here but can you turn prefetch off? Since you have a frontend to Unbound, this shouldn't provide anything more. Probably not related but I was going through your config once more.

I will try.

High 'exceeded' number without 'overwritten' means that the request list is full with recent queries, and seems to point that Unbound is being more busy than usual while resolving the current request list.

Is there a way how to dump request list for all threads?

Another thought, does anything stand out wrt to the following statistics during the incident?

* `num.answer.bogus`

* `num.rrset.bogus`

* `num.valops`

Does not look like that. See the plot below. I have a new occurence on a backend that was restarted 5.3. From 10.3. I see a constant increase of exeeded counters (green line), the statistics you wanted seems to be increasing nearly linearly

In the last plot, I am assuming that if you plotted the cache size as well, it would start to plateau?
Yes, it would. It seems that my theory pointing to a full cache was really just a coincidence:

As for the dropped queries, maybe some of them could have something to do with traffic but generally, when the request list is full, you will get drops regardless of the query itself.

Yes, sure. Anyway if it is related to the traffic, it would come back nearly immediately after a restart. But is is not like that. If I restart all the backedns I have overall amount of drops goes down.

Also in general Unbound will try hard to find an answer so the drops you are seeing casually (if we ignore this issue for a moment) is Unbound not having replied to a specific DNS ID in the client's waiting window. Unbound would reply to a client only if there is a definite answer to give, not with early error codes. An exception to this is serve-expired (but you are not using that).

I will try to see if I can reproduce locally the behavior you are seeing.

Thanks. I will go on with disabled prefetch and wait-limit: 0 and discard-timeout: 0. This has not been done on the backend ploted above yet.

[gthess]

Is there a way how to dump request list for all threads?

No, it is only a peek for the first thread.

Still pocking at things here but can you turn prefetch off? Since you have a frontend to Unbound, this shouldn't provide anything more. Probably not related but I was going through your config once more.

I will try.

Interesting, in the new graphs I see that prefetch is exercised. Maybe the frontend is evicting records, caps the TTL to a lower value, or does prefetch itself. Keeping it on would help at the end for cache hits.

In the last plot, I am assuming that if you plotted the cache size as well, it would start to plateau?

Yes, it would. It seems that my theory pointing to a full cache was really just a coincidence:

Just to make sure, the memory plateaus because of usage patterns or because it reached its maximum? I can't really tell from that graph.

Thanks. I will go on with disabled prefetch and wait-limit: 0 and discard-timeout: 0. This has not been done on the backend ploted above yet.

Ok, I will wait to see if something changes.

One thing you could also plot is all the total.requestlist.* values, to see if the exceeded value is correct or the bug that @edmonds pointed at (it goes away with discard-timeout: 0).

[rygl]

In the last plot, I am assuming that if you plotted the cache size as well, it would start to plateau?

Yes, it would. It seems that my theory pointing to a full cache was really just a coincidence:

Just to make sure, the memory plateaus because of usage patterns or because it reached its maximum? I can't really tell from that graph.

It is because of its maximum. In order to simulate the behavior I have reduced the caches dramatically:

msg-cache-size: 64M rrset-cache-size: 256M key-cache-size: 256M neg-cache-size: 64M

One thing you could also plot is all the total.requestlist.* values, to see if the exceeded value is correct or the bug that @edmonds pointed at (it goes away with discard-timeout: 0).

The numbers are low. I have scaled num. of queries.