Skip to content

Reverse tunnel handshake blocks with no deadline when the agent does not answer #6777

Description

@gustavosbarreto

Dialer.DialTo takes a context.Context and hands it to Manager.Dial, which uses it to open the stream. Once the stream is open the context stops being consulted: Target.prepare talks to the agent over the raw net.Conn with no deadline and no cancellation, so a stream the agent accepts but never answers blocks the calling goroutine indefinitely.

server/ssh/pkg/dialer/dialer.go:82-97:

conn, version, err := t.Manager.Dial(ctx, NewKey(tenant, uid))  // ctx ends here
...
return target.prepare(conn, version)                            // no ctx, no deadline

Where each target blocks (server/ssh/pkg/dialer/target.go):

  • HTTPProxyTarget, V1 — http.ReadResponse waiting for the agent's 200 to the CONNECT prelude
  • HTTPProxyTarget, V2 — the json.Decode of the negotiation reply
  • SSHOpenTarget / SSHCloseTarget, V2 — the read inside multistream.SelectProtoOrFail

Effect

The request does not fail; it hangs. Whatever sits in front decides when it ends: nginx closes at proxy_read_timeout and answers 504, and the API logs nothing until then, because the access log line is written when the handler returns. A client that gives up and disconnects does not release the server side either, since nothing observes the request context. Until the front-end timeout fires, the hijacked client connection and the tunnel stream both stay open.

An agent that is reachable and holding a healthy tunnel is enough to reach this: the device answers the reverse dial, so the stream opens, and only the handshake stalls.

How it showed up

A web endpoint pointing at a port on the device whose listener had wedged — a single-threaded HTTP server holding one keep-alive connection, refusing to serve anything else. The device was online, the tunnel was healthy, the stream opened, and the handshake never completed. Every request to that endpoint hung until the edge timed out.

The same shape covers a wedged or partially-upgraded agent, which is the more interesting case: the tunnel looks healthy from the server's side while every dial through it parks a goroutine.

Direction

Give prepare the context and bound the handshake with a deadline on the connection, cleared before the connection is handed back for streaming — the handshake is a short exchange with a known shape, unlike the streaming phase that follows it, which is legitimately long-lived. Both call sites already have a request context to derive from.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions