Skip to content

Vector appears to leak TCP connections that are not politely closed #25054

Open
Bug
#25062
Open
Vector appears to leak TCP connections that are not politely closed#25054
Bug
#25062
@erhudy

Description

@erhudy
erhudy
opened on Mar 26, 2026

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Vector appears to be prone to leaking TCP connections with the only resolution being to fully restart Vector, since there is not a global configuration to enforce TCP keepalives on sockets or close idle connections; some individual sources that are TCP-oriented (e.g. raw TCP or Logstash) may have keepalive configurations, but there is no global mechanism of controlling it. We discovered this problem when we noticed that our Vector receivers were leaking connections established by Vector leaf instances on ephemeral VMs in certain CI/CD OS image build pipelines, at the end of which the VM is abruptly terminated without a clean shutdown. ss -natp on the Vector receiver host showed TCP connections in ESTABLISHED state even when the host on the other side no longer existed, and they could not be cleared except by restarting that Vector instance.

My testing was originally focused on Linux, but I also tested my reproducer for the bug with Vector on Windows and the sender in WSL2 on the same machine and it also exists there, so I would suggest that the bug is OS-independent.

To replicate:

  1. Set up Vector somewhere and start it with a basic config file like the one I have provided in this ticket - I tested with the Vector receiver on both Windows and Linux, did not try to reproduce on macOS but I would expect also possible there
  2. On a source with a separate TCP stack (I tested with separate Linux hosts and also with WSL2 -> Windows on the same Windows machine), use the attached dirty_conn.py script, which will open a connection to Vector and then intentionally sabotage it by inserting an iptables rule to prevent sending FIN or RST to Vector, and then sleep for 60 seconds to absorb retransmits.
  3. Observe Vector behavior on the target with ss -natp | grep PORT on Linux or netstat -a -n -i -p TCP | sls -Pattern ":PORT" on Windows PowerShell. Long after the script exits, the output will still show connections in ESTABLISHED state on the target for which the corresponding connection has long since disappeared from the source.

Even when shutting down Vector, the orphaned connections will remain present until the process completely shuts down:

 edmun   ~    netstat -a -n -i -p TCP | sls -Pattern ":5046"                                 in pwsh at 19:19:08

TCP    0.0.0.0:5046           0.0.0.0:0              LISTENING          0
TCP    192.168.116.138:5046   192.168.116.138:58028  ESTABLISHED        955104
TCP    192.168.116.138:5046   192.168.116.138:58056  ESTABLISHED        1015415

# started Vector shutdown here

 edmun   ~    netstat -a -n -i -p TCP | sls -Pattern "5046"                                  in pwsh at 19:25:16

TCP    192.168.116.138:5046   192.168.116.138:58028  ESTABLISHED        1047753
TCP    192.168.116.138:5046   192.168.116.138:58056  ESTABLISHED        1108065

# Vector shutdown completed

 edmun   ~    netstat -a -n -i -p TCP | sls -Pattern "5046"                                  in pwsh at 19:26:48
 edmun   ~                                                                                   in pwsh at 19:27:58

dirty_conn.py

Configuration

sources:
  http:
    type: http_server
    address: 0.0.0.0:5046

sinks:
  print:
    type: "console"
    inputs: ["http"]
    encoding:
      codec: "json"
      json:
        pretty: true

Version

0.54.0

Debug Output

2026-03-26T23:28:25.243657Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2026-03-26T23:28:25.243772Z  INFO vector::app: Log level is enabled. level="trace"
2026-03-26T23:28:25.243829Z DEBUG vector::app: Building runtime. worker_threads=16
2026-03-26T23:28:25.245037Z  INFO vector::app: Loading configs. paths=["..\\config\\vector.yaml"]
2026-03-26T23:28:25.250917Z DEBUG vector::config::loading::secret: No secret placeholder found, skipping secret resolution.
2026-03-26T23:28:25.251588Z DEBUG vector::topology::builder: Building new source. component_id=http
2026-03-26T23:28:25.251987Z DEBUG vector::topology::builder: Building new sink. component_id=print
2026-03-26T23:28:25.452173Z  INFO vector::topology::running: Running healthchecks.
2026-03-26T23:28:25.452287Z DEBUG vector::topology::running: Connecting changed/added component(s).
2026-03-26T23:28:25.452294Z  INFO vector::topology::builder: Healthcheck passed.
2026-03-26T23:28:25.452410Z DEBUG vector::topology::running: Configuring outputs for source. component_id=http
2026-03-26T23:28:25.452578Z DEBUG vector::topology::running: Configuring output for component. component_id=http output_id=None
2026-03-26T23:28:25.452642Z DEBUG vector::topology::running: Connecting inputs for sink. component_id=print
2026-03-26T23:28:25.452862Z DEBUG vector::topology::running: Adding component input to fanout. component_id=print fanout_id=http
2026-03-26T23:28:25.452970Z DEBUG vector::topology::running: Spawning new source. component_id=http
2026-03-26T23:28:25.453084Z TRACE vector::topology::running: Spawning new sink. component_id=print
2026-03-26T23:28:25.453129Z DEBUG source{component_kind="source" component_id=http component_type=http_server}: vector::topology::builder: Source pump supervisor starting.
2026-03-26T23:28:25.453104Z DEBUG source{component_kind="source" component_id=http component_type=http_server}: vector::topology::builder: Source starting.
2026-03-26T23:28:25.453162Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::topology::builder: Sink starting.
2026-03-26T23:28:25.453193Z  INFO vector: Vector has started. debug="false" version="0.54.0" arch="x86_64" revision="2b8b875 2026-03-10 15:47:37.284215410"
2026-03-26T23:28:25.453211Z DEBUG source{component_kind="source" component_id=http component_type=http_server}: vector::topology::builder: Source pump starting.
2026-03-26T23:28:25.453264Z  INFO source{component_kind="source" component_id=http component_type=http_server}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:5046
2026-03-26T23:28:25.453359Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2026-03-26T23:28:25.456604Z TRACE vector: Beep.
2026-03-26T23:28:26.458542Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:28:35.461197Z TRACE vector: Internal log [Beep.] has been suppressed 9 times.
2026-03-26T23:28:35.461352Z TRACE vector: Beep.
2026-03-26T23:28:36.005426Z TRACE hyper::proto::h1::conn: Conn::read_head
2026-03-26T23:28:36.005582Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Busy }
2026-03-26T23:28:36.465403Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:28:46.455968Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:28:46.456096Z TRACE vector: Beep.
2026-03-26T23:28:47.464757Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:28:56.464934Z TRACE vector: Internal log [Beep.] has been suppressed 9 times.
2026-03-26T23:28:56.465070Z TRACE vector: Beep.
2026-03-26T23:28:57.466231Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:29:07.461241Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:29:07.461329Z TRACE vector: Beep.
2026-03-26T23:29:08.464914Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:29:18.466684Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:29:18.466875Z TRACE vector: Beep.
2026-03-26T23:29:19.464240Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:29:29.466535Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:29:29.466643Z TRACE vector: Beep.
2026-03-26T23:29:30.467356Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:29:40.468153Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:29:40.468268Z TRACE vector: Beep.
2026-03-26T23:29:41.467574Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:29:46.398064Z  INFO vector: Vector has stopped.
2026-03-26T23:29:46.398190Z DEBUG source{component_kind="source" component_id=http component_type=http_server}: hyper::server::shutdown: signal received, starting graceful shutdown
2026-03-26T23:29:46.398212Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::utilization: Couldn't send utilization start wait message. component_id=ComponentKey { id: "print" } error="Closed(..)"
2026-03-26T23:29:46.398300Z TRACE hyper::proto::h1::conn: disable_keep_alive; in-progress connection
2026-03-26T23:29:46.398323Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::utilization: Internal log [Couldn't send utilization start wait message.] is being suppressed to avoid flooding.
2026-03-26T23:29:46.398371Z TRACE hyper::proto::h1::conn: Conn::read_head
2026-03-26T23:29:46.398464Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Disabled }
2026-03-26T23:29:46.412420Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="print, http" time_remaining="59 seconds left"
2026-03-26T23:29:51.407813Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] is being suppressed to avoid flooding.
2026-03-26T23:29:51.469294Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:29:51.469385Z TRACE vector: Beep.
2026-03-26T23:29:52.456083Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:30:01.402593Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] has been suppressed 2 times.
2026-03-26T23:30:01.402740Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="print, http" time_remaining="44 seconds left"
2026-03-26T23:30:02.464354Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:30:02.464434Z TRACE vector: Beep.
2026-03-26T23:30:03.461529Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:30:06.414430Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] is being suppressed to avoid flooding.
2026-03-26T23:30:13.465461Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:30:13.465552Z TRACE vector: Beep.
2026-03-26T23:30:14.467577Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:30:16.410455Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] has been suppressed 2 times.
2026-03-26T23:30:16.410621Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="print, http" time_remaining="29 seconds left"
2026-03-26T23:30:21.406240Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] is being suppressed to avoid flooding.
2026-03-26T23:30:24.464421Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:30:24.464543Z TRACE vector: Beep.
2026-03-26T23:30:25.462983Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:30:31.401242Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] has been suppressed 2 times.
2026-03-26T23:30:31.401358Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="print, http" time_remaining="14 seconds left"
2026-03-26T23:30:35.457739Z TRACE vector: Internal log [Beep.] has been suppressed 10 times.
2026-03-26T23:30:35.457877Z TRACE vector: Beep.
2026-03-26T23:30:36.404254Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] is being suppressed to avoid flooding.
2026-03-26T23:30:36.466727Z TRACE vector: Internal log [Beep.] is being suppressed to avoid flooding.
2026-03-26T23:30:41.408300Z  INFO vector::topology::running: Internal log [Shutting down... Waiting on running components.] has been suppressed 1 times.
2026-03-26T23:30:41.408393Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="print, http" time_remaining="4 seconds left"
2026-03-26T23:30:45.465937Z TRACE vector: Internal log [Beep.] has been suppressed 9 times.
2026-03-26T23:30:45.466071Z TRACE vector: Beep.
2026-03-26T23:30:46.401188Z ERROR vector::topology::running: components="print, http" Failed to gracefully shut down in time. Killing components. internal_log_rate_limit=false

Example Data

No response

Additional Context

A corollary; I have also noticed that Vector can be prone to getting DoS'd by network issues or network scan tools, shown on a couple of graphs I have attached here where the value of vector_open_connections monotonically increases in a perilous way before somebody notices and kicks Vector, or the instigating issue resolves itself. In general I would believe it to be good if Vector were able to handle such issues in a defensive way.

Image Image

References

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions