Potential bug in GRO code for Windows? #2041
Description
We are seeing an issue on a Windows machine where quinn-udp gives us packets larger than the configured MTU (2540 and 3810). Here is how we split the packets: https://github.com/firezone/firezone/blob/5437c3e2df001c8c491749c7bff0c063ccdf39e1/rust/socket-factory/src/lib.rs#L210-L255
This code has been working on Linux for several months and we have also not had reports that it doesn't work on Windows. Recently however a customer filed a bug report and looking through the logs, I discovered that we receive packets (i.e. DatagramIn in the above snippet) that are much larger than expected.
Going through quinn-udp's source code, I developed two theories and would be interested in hearing your thoughts on it:
strideis being read incorrectly (endianess-issue?) and instead of being smaller thanlen, it is bigger thanlen, meaning we never end up splitting the packets despite it being coalesced.quinn/quinn-udp/src/windows.rs
Line 241 in d23e4e4
- The buffer for control messages is too small, thus Windows fails to fit the
UDP_COALESCED_INFOmessage into it and can't tell us that the buffer is coalesced even though it is.
Unfortunately, I've not been able to reproduce this on my own hardware :(
Activity
stormshield-damiend commentedon Nov 14, 2024
Does the windows run on intel CPU or on ARM CPU ?
mxinden commentedon Nov 14, 2024
We see a similar issue on Windows on ARM, see summary in last comment on this thread.
While I myself don't have access to a Windows on ARM machine, someone at Mozilla does and is able to reproduce the issue. We have not been able to reproduce this on a Windows on ARM VM.
Interesting. I will try reproducing the issue with a larger buffer. For what it is worth, here is MSQUIC's buffer length.
thomaseizinger commentedon Nov 14, 2024
We see this issue on an x64_86 CPU.
thomaseizinger commentedon Nov 14, 2024
I thought about this a bit more and I am no longer sure that is a viable possibility unless we receive a lot of other control messages that we don't match on.
stormshield-damiend commentedon Nov 14, 2024
Could the receiver interface be a dual stack (IPv4 and IPv6) one ? I had no chance to test that case when writing the code, maybe we receive ipv4 + ipv6 control messages ?
thomaseizinger commentedon Nov 14, 2024
Interesting, according to https://bugzilla.mozilla.org/show_bug.cgi?id=1916558#c43, the
UDP_COALESCED_INFOnever gets received. We only getIP_PKTINFOandIP_ECN.thomaseizinger commentedon Nov 14, 2024
In https://learn.microsoft.com/en-us/windows-hardware/drivers/network/udp-rsc-offload, it is mentioned that this only works on Windows 11, version 24H2 and later. I'll have to investigate what version our customer is using.
thomaseizinger commentedon Nov 14, 2024
I can see in our logs that this customer did have dual-stack but I don't think it matters here because the way we do dual-stack is by having two sockets, one for IPv4 and one for IPv6.
stormshield-damiend commentedon Nov 14, 2024
This code work fine on our side on Windows 10 and 11.
stormshield-damiend commentedon Nov 14, 2024
I do not see any special case related to this in msquic C code https://github.com/microsoft/msquic/blob/main/src/platform/datapath_winuser.c#L3216
thomaseizinger commentedon Nov 14, 2024
@mxinden Do you know by any chance if the affected user is using a dock? We have reports that this only happens when the user is connected via a Dell dock WD19S. It works fine over WiFi and via a separate Ethernet dongle.
stormshield-damiend commentedon Nov 14, 2024
@mxinden could your colleague print the stride value (at L210) before the loop ? it is init with the length of the received packet to handle the case were received packet is not coalesced ?
saschanaz commentedon Nov 14, 2024
The colleague here 👋
I'm not using any dock but only the builtin WiFi, and the stride value is 1280.
mxinden commentedon Nov 14, 2024
Just to avoid any confusion, the stride value is
1280AND the datagram length value is1280. This is with the unit test I provided in this commit, see also the logs here. That said, I have the suspicion that the unit test simply doesn't force datagram coalescing in the kernel, see this bugzilla comment.Looking at pcaps with fosstodon.org that @saschanaz provided, we see >1_500k UDP datagrams over an internet path. >1_500k over an internet path is very unlikely. My suspicion is, that the Windows kernel correctly coalesces the UDP datagrams (< 1_500k) into one >1_500k datagram, but then either does not provide the stride length, or we fail to read the stride length.
stormshield-damiend commentedon Nov 14, 2024
I think you wanted to link this commit mxinden@7ce09c3
54 remaining items
thomaseizinger commentedon Jul 11, 2025
So on this hardware, the unit tests of
quinn-udppass just fine? Because should should use GRO if I am not mistaken.The reason I am asking is because if we can create self-contained test, then we could ship such a function with
quinn-udpand perform a self-test when a new socket is created. Like:If yes, we know the hardware is safe to use with GRO, if no, we can disable it.
That would be a much more reliable way of conditionally enabling it because we don't need to "wait" for a traffic pattern that triggers the detection as we can just generate the traffic ourselves and therefore know, how big the packets should be that we are reading.
ehoogeveen-medweb commentedon Jul 11, 2025
FWIW, in the linked Firefox issue I found that receive segment coalescing needed to be specifically available and enabled in the relevant network drivers:
Recv Segment Coalescingneeds to be present and enabled1RSCv4and/orRSCv6needs to be present and enabled2 (presumably they're for IPv4 and IPv6 respectively)And of course URO needs to be available in Windows - I only started having issues after updating to Windows 11 24H2.
If you run
netsh int udp show globalit should show the URO state asReceive Offload State(though this is only the Windows feature, the drivers still have to agree).Footnotes
https://bugzilla.mozilla.org/show_bug.cgi?id=1916558#c59 ↩
https://bugzilla.mozilla.org/show_bug.cgi?id=1916558#c62 ↩
mxinden commentedon Jul 11, 2025
Yes, back then we were not able to reproduce the bug on the buggy devices using the
quinn-udpunit tests. The unit tests do use URO. I assume the bugs didn't surface, as they are using localhost networking, i.e. don't go through the (presumably buggy) network card (drivers).BarabasGitHub commentedon Aug 18, 2025
Hello, I'm having a peculiar issue and this is the only place I've found anything that looks similar.
We have a situation where only some end users end up receiving concatenated udp datagrams from a recv call. For them this happens consistently, though far from all the time. I've been scratching my head a a lot and went through our code, adding debug logging etc. I've never seen anything like this happen before. I know udp can duplicate, reorder and drop packets, but it should never give me two from a recv call.
This coalescing option is the only potential candidate I've been able to identify, however we don't enable or disable it explicitly. When I use getsockopt with UDP_RECV_MAX_COALESCED_SIZE it tells me it's 0. Any chance it's still possible to trigger this bug?
This is on windows x64.
mxinden commentedon Aug 18, 2025
@BarabasGitHub please open a new issue. I don't think this is related. Please include the
quinn-udpversion you are using.BarabasGitHub commentedon Aug 18, 2025
@mxinden sorry, I'm not using quinn, but this is the only similar problem I found that resembles what goes wrong in our application (using plain udp). So I was hoping someone here knows more about this kind of bug where you receive multiple udp datagrams in one recv.
feat(16-02): WSAEMSGSIZE Windows tracing suppression + crossterm osc5…
feat(connlib): enable URO on Windows until proven broken (firezone#14392
perf: batch QUIC UDP I/O
perf: batch QUIC UDP I/O
perf: batch QUIC UDP I/O
perf: batch QUIC UDP I/O (#8069)
perf: batch QUIC UDP I/O (oferchen#8069)