Skip to content

test_connect_uds can fail on a short read #967

Description

@ShanireZ

Describe the bug

connectors::tests::test_connect_uds reads the mock server's 9-byte response with
AsyncReadExt::read and discards the returned length:

let mut buf = [0; 9];
let _ = stream.read(&mut buf).await.unwrap();
assert_eq!(&buf, b"it works!");

read completes as soon as any bytes are available, and a stream socket is free to
deliver write_all(b"it works!") in more than one segment. When that happens, buf
holds a partial message padded with zeros and the assertion fails. Because the length
is dropped, nothing in the output says "we only read N bytes" — the failure looks like
the server sent the wrong thing.

This is a latent flake rather than a guaranteed failure: on a loopback UDS the response
usually arrives in one piece.

Note this test lives under #[cfg(feature = "any_tls")], and both pingora-core and
pingora default to no TLS feature, so the default cargo test --workspace used by CI
does not compile it. It affects anyone running the test suite with a TLS backend
enabled (--features rustls, openssl, boringssl, ...).

Pingora info

Pingora version: main @ 0046038
Rust version: cargo 1.97.1
Operating system version: Debian 13 (trixie), x86_64

Steps to reproduce

The flake can be made deterministic by having the mock server split its write, which is
legal stream-socket behaviour. In spawn_mock_uds_server, replace

let _ = stream.write_all(response).await;

with

let (head, tail) = response.split_at(1);
let _ = stream.write_all(head).await;
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
let _ = stream.write_all(tail).await;

then run:

cargo test -p pingora-core --lib --features rustls connectors::tests::test_connect_uds

Expected results

The test reads the full 9-byte response and passes.

Observed results

With the perturbation above, the test fails every time (3/3 runs):

assertion `left == right` failed
  left: [105, 0, 0, 0, 0, 0, 0, 0, 0]
 right: [105, 116, 32, 119, 111, 114, 107, 115, 33]

105 is i — only the first byte had arrived. That [105, 0, 0, ...] shape is also
what the intermittent failure looks like when it happens on its own.

Additional context

stream.read_exact(&mut buf).await.unwrap() fixes it; with the fix the test passes both
with and without the perturbation (5/5 each). I have a patch ready and will open a PR
referencing this issue.

For what it's worth, pingora-core/src/listeners/mod.rs has the same
let _ = stream.read(&mut buf) shape in test_listen_tls, but that one only drains the
request and never asserts on the contents, so it is not affected. I have left it alone.

Metadata

Metadata

Assignees

Labels

AcceptedThis change is accepted by us and merged to our internal repobugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions