A slow RTR client can stall future payload processing in StayRTR

[wangxin1115]

I found a StayRTR behavior where a single valid RTR client that stops draining a normal full response can block the processing of later payload generations.

I reproduced this locally using valid RTR protocol traffic only.

Tested revision：v0.6.4

Summary

The behavior can be triggered by a single valid RTR client:

1. StayRTR has an initial payload generation G0.
2. One RTR client connects and sends a valid Reset Query.
3. The client reads only the first 8 bytes of the response, then stops draining the connection while keeping it open.
4. StayRTR later processes and commits a new payload generation G1.
5. A subsequent generation G2 becomes available through the normal payload refresh path.
6. While the slow client remains connected, G2 is not processed as the current generation.
7. Normal observer clients can still connect and receive a complete RTR response through End Of Data, but they continue receiving G1.
8. Once the slow client disconnects, payload processing resumes and G2 becomes current.

The observed sequence is therefore:

valid Reset Query
        |
        v
client reads 8 bytes and stops draining
        |
        v
G1 becomes current
        |
        v
G2 becomes available
        |
        v
G2 is not processed
        |
        v
normal observers still receive G1
        |
        v
slow client disconnects
        |
        v
G2 processing resumes

Source-level explanation

At the source level, each client has a buffered transmit channel:

transmits chan PDU

with capacity 256.

A full Reset Query response is queued through this per-client transmit path, while a separate send loop writes queued PDUs to the network connection.

If the client stops draining its connection, the socket write path can stop making sufficient progress, allowing the per-client transmits queue to fill.

The important point is that the same queue is also used by Notify().

The relevant source-level sequence is:

Reset Query
    |
    v
full response queued through client.transmits
    |
    v
client stops draining
    |
    v
socket writer stops making sufficient progress
    |
    v
client.transmits can fill

Later, when a new payload generation is processed:

AddData(G1)
    |
    v
G1 becomes current
    |
    v
NotifyClientsLatest()
    |
    v
synchronous iteration over clients
    |
    v
c.Notify()
    |
    v
Serial Notify queued through the same client.transmits channel

If the slow client's transmit queue is already full, c.Notify() can block while trying to enqueue the Serial Notify.

Because NotifyClientsLatest() performs client notification synchronously and sequentially, it cannot return while blocked on that client.

The resulting path is:

G1 committed
    |
    v
NotifyClientsLatest()
    |
    v
c.Notify() reaches slow client
    |
    v
blocking send to full transmits queue
    |
    v
NotifyClientsLatest() does not return
    |
    v
current payload-update call does not complete
    |
    v
periodic update loop cannot continue to process G2

This allows the output backpressure of one client to propagate into the global payload-update path.

Runtime observations

I tested three relevant cases.

Normal control

Without the slow client, the normal update sequence completed successfully:

G0 -> G1 -> G2

Shallow-slow control

A client that was slow but did not reach the deeply blocked transmit state also allowed the update sequence to complete successfully:

G0 -> G1 -> G2

Target case

With one client that:

• sent one valid Reset Query;
• read exactly 8 response bytes;
• stopped draining the response;
• kept the TCP connection open;

the observed behavior was:

G0
  |
  v
G1 becomes current
  |
  v
G2 becomes available
  |
  v
G2 does not become current
  |
  v
normal observer receives G1 through End Of Data

After the slow client was intentionally disconnected:

G2 processing resumed

This gives a clean behavioral recovery oracle: removing the blocked client allows payload processing to continue.

15-minute duration corroboration

I also ran a separate bounded duration test using:

• the same StayRTR process;
• the same slow client connection;
• one valid Reset Query;
• exactly 8 response bytes read before the client stopped draining.

The same slow connection remained open for 900 seconds.

At all of the following checkpoints:

• T0
• T+60s
• T+300s
• T+600s
• T+900s

the following remained true:

G2 available: yes
G2 current:   no
observer:     G1 through End Of Data

No automatic StayRTR disconnect of the slow client was observed during the 15-minute interval.

After the slow client was intentionally disconnected, G2 processing resumed.

Availability boundary

This was not a whole-server availability failure.

During the stalled update interval, unrelated clients could still:

• establish new RTR connections;
• send valid Reset Queries;
• receive the currently committed G1 payload;
• receive End Of Data successfully.

The specific observed effect is that future payload-generation processing is stalled while the blocking client remains connected.

In other words, StayRTR continues serving the previously committed generation, but does not process a later available generation until the slow client is released.

Expected behavior

I would expect one client's output backpressure to be isolated from the global payload-update path.

A slow or non-draining client should not be able to prevent later payload generations from being processed for all clients.

Possible approaches might include isolating notification delivery from the global update loop, making per-client notification non-blocking, disconnecting clients whose output path stops making progress, or otherwise bounding the time spent waiting on one client's transmit path.

Actual behavior

A single valid RTR client can:

1. send one legitimate Reset Query;
2. stop draining the normal response;
3. cause a later Notify() operation to block on its per-client transmit queue;
4. prevent NotifyClientsLatest() from returning;
5. prevent the current update call from completing;
6. consequently prevent a later available payload generation from being processed.

Normal clients can still connect and retrieve the previously committed generation, but they continue receiving that generation until the blocking client is released.

The confirmed behavior is specifically:

A single valid RTR client that stops draining a full response can block StayRTR's synchronous notification path after a generation has been committed, preventing later available payload generations from being processed until the slow client is released.

I have additional reproduction details, source references, logs, control results, and causal timelines available if useful.