How to receive data in deffirent pods on the same node?

[caibirdme]

If I install vector as daemonset(not sidecar) in k8s cluster, how could vector receive log data in pods on the same node? I don't find any documentation about this. Or if I try to send logs to vector aggregator, what's the protocol should be, what's the endpoint? I cannot find document about that either? Can anyone help?

[wtchangdm]

Container logs are located in worker node's /var/log/containers/... (which actually are symbolic links to /var/log/pods/...).

Usually, when a log shipper (vector, fluentd or fluent-bit, etc) is deployed as DaemonSet, it mounts that hostPath and collects files from there (while observing things like file rotation).

You can check vector chart .

flowchart LR
  subgraph Node
    subgraph User Pod
      container
    end
    container -. "stdout/stderr" .-> container_runtime["CRI Container Runtime"]
    container_runtime -. write .-> files
    files["/var/log/pods/..."]
    subgraph Log Shipper Pod
      log_shipper_container["Log Shipper Container"]
    end
    files -. "Volume mounted as 'hostPath'" .-> log_shipper_container
  end

log_shipper_container -. send .-> storage([Cool Storage])

Loading

I can't guarantee this is 100% correct since I might miss some details. But that's the idea.

You can also check Logging Architecture.

As for the aggregator pattern, I think it is only a matter of where you put it.

[spencergilbert]

Thanks @wtchangdm - this is correct and how the node logging agent pattern should work.

The second question you had @caibirdme, sending directly from your app to Vector (exposing directly from the application) depends more on your app or the logging libraries you have access to. You'd configure Vector with a network accessible source, http or syslog for example, and configure your application to write logs to that endpoint.