Issues when clustering with TLS

[tscolari]

Hello Andy,

I'm seeing some issues, maybe it's some misconfiguration on my side.
But it looks like that when I try Piko with replicas > 1 there are some TLS errors - in particular when the proxy connects to one node and the upstream to the other:

{"level":"warn","ts":"2025-09-30T08:03:13.613Z","subsystem":"proxy","msg":"httputil: ReverseProxy read error during body copy: read tcp 10.0.28.94:44020->10.0.13.236:8000: read: connection reset by peer"}

In this case the proxy-side-connection is connected to10.0.28.94 and the upstream-side-connection to 10.0.13.236.
I've TLS configured in the proxy server. It looks like it's terminating the TLS and then forwarding to the node with the upstream.
Have you seen this before/have any recommendation here?

(I'm using Kubernetes/the helm charts)

Many thanks!

[andydunstall]

Hi @tscolari, yeah thats a bug, I hadn't considered this case (TLS support was added after the initial design). I'll think about a fix and get back to you

[andydunstall]

The issue is Piko was initially designed to be hosted behind a TLS terminating load balancer or proxy, so never supported TLS between Piko nodes (forwarding proxy traffic and gossip)

Piko nodes can forward proxy connections to other nodes via the proxy port with raw TCP (no TLS), so if one node enables TLS on the proxy port, the other nodes can't connect to it.

Therefore I think the solution is to configure the proxy with both server and client TLS configuration, such as:

proxy:
  tls:
    cert: ...
    key: ...
    client_cas: ... # Optional

    client:  # Optional
      cert: ... # Optional
      key: ... # Optional
      root_cas: ... # Optional
      insecure_skip_verify: ... # Optional

Then if TLS is enabled, nodes attempt to connect to one another using TLS. This will also let us configure client certificates, root CAs and so on if required. The only change is adding proxy.tls.client so this will be backwards compatible.

If that sounds ok, I'll open a PR (unless you want to)

[tscolari]

Hey Andy, I think that's a good plan.

I don't think I've a lot of time because I'm about to change jobs, but let me give it a try and see if I can at least get something closer to the finishing line!

[tscolari]

@andydunstall just looking around on how to implement this.
Shouldn't we only need something like this:

proxy:
  tls:
    cert: ...
    key: ...
    client_cas: ... # Optional

    client:  # Optional
      root_cas: ... # Optional
      insecure_skip_verify: ... # Optional

I think I'm missing where we would use the cert/key when creating the connection to the other proxy 🤔

Other thoughts I'm having are around root_cas, should we require a list of CAs and append it to the system's cert poll instead?
It might get hard to setup if we require customers to set CAs for public certs included 🤔

[tscolari]

Early draft here: #284

[tscolari]

I think I'm missing where we would use the cert/key when creating the connection to the other proxy 🤔

Actually as soon as I went back to code I remembered Piko supports mTLS - because I don't use I didn't think of it!

[andydunstall]

I don't think I've a lot of time because I'm about to change jobs, but let me give it a try and see if I can at least get something closer to the finishing line!

Ok it's up to you, I have some free time over the next 2 weeks so can take it. Let me know what you prefer

I remembered Piko supports mTLS

Yeah, see the Piko agent for an example: https://github.com/andydunstall/piko/blob/main/agent/config/config.go#L154

[tscolari]

@andydunstall I gave it a try - #284 - I haven't properly tested it yet though.

I wasn't sure on how to introduce the nested client to the TLSConfig without affecting all other components that use it, so I ended up creating a side property (I tried using composition but json/yaml parsing didn't behave like I wish they would). Please let me know what you think of this direction 🙇

[andydunstall]

I gave it a try - #284

Thanks - as a draft that looks good

I wasn't sure on how to introduce the nested client to the TLSConfig without affecting all other components that use it

I don't understand the issue here. I'd prefer to nest it as having tls and client_tls at the root seems confusing to me

[andydunstall]

Closed by #287