Nginx proxy protocol to Haraka on port 465 w/ proxy protocol enabled = SSL handshake fail

[DoobleD]

Is proxy protocol supposed to work also on port 465 (implicit SSL/TLS)?

I've set up an Nginx stream proxy in front of Haraka, that works fine. But when I enable the proxy protocol (proxy_protocol directive on Nginx and haproxy_hosts config file on Haraka), the connection breaks before Haraka logs even a single thing, with the following error from Nginx:

peer closed connection in SSL handshake while SSL handshaking to upstream

I tried the same thing without SSL (targeting port 587 STARTTLS instead), and it worked fine. Hence I'm wondering if there could be some issue when using proxy protocol with SSL/TLS?

Interestingly Dovecot works fine with the same setup (proxy protocol to implicit SSL/TLS IMAP and POP ports).

[darkpixel]

I just noticed some strangeness around the proxy protocol too.

Using swaks on port 25 with the --tls flag:

<-  220 relay-96596467c-t4w47 ESMTP Test Server (96E2E2DC-AD48-4F27-821B-E9DC11A96ED9)
 -> EHLO iridal
<** 220 relay-96596467c-t4w47 ESMTP Test Server (96E2E2DC-AD48-4F27-821B-E9DC11A96ED9)
 -> HELO iridal
<-  250-relay-96596467c-t4w47 Hello -redacted-.com [re.da.ct.ed]Haraka is at your service.
<-  250-PIPELINING
<-  250-8BITMIME
<-  250-SMTPUTF8
<-  250-SIZE 0
<-  250 STARTTLS
*** Host did not advertise STARTTLS
 -> QUIT

I mean...it shows the server sending 250 STARTTLS from the client side, so I'm not sure why it's not picking it up.

I'm assuming you're using Kubernetes.
I'm not sure how you're defining your ingress-nginx, but I noticed that it's possible to misconfigure it with Digital Ocean.

If you set:
controller.service.annotations."service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol"="true"
and:
tcp.25="relay/relay:25:25:PROXY"
STARTTLS will fail.

If you set:
tcp.25="relay/relay:25:25:"
things should work properly...

[DoobleD]

Thanks for the input @darkpixel. We haven't used Kubernetes in our attempt, we simply ran both Nginx and Haraka on the same machine. The simplified Nginx stream config we used looks like that (assuming we set Haraka smtps to run on port 466):

stream {
  ...

  server {
    listen 465 ssl;
    listen [::]:465 ssl;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_certificate /etc/ssl/server.crt;
    ssl_certificate_key /etc/ssl/server.key;
    ssl_dhparam /etc/ssl/dh4096.pem;

    proxy_ssl on;
    proxy_protocol on;
    proxy_pass 127.0.0.1:466;
  }
}

A similar config works for Dovecot with proxy protocol enabled. But it fails with Haraka, though connecting via openssl s_client to port 466 and manually sending proxy protocol commands does work. The certs served by Nginx and Haraka are both valid.

Possibly Nginx is sending proxy protocol commands before Haraka is ready to accept them? Just a guess.

[darkpixel]

Could be. I've been doing a bunch of crazy/stupid stuff in my test environment today, and for the life of me, I can't bring up 465. 25 and 587 work just fine. I'll do some testing tomorrow after I handle an unrelated issue with external-dns and multiple load balancers.

[DoobleD]

Just wondering if anybody else has encountered this and found a solution?

[DoobleD]

I can't find the root cause, but here is the difference near the end of the connection when testing with proxy_protocol on vs proxy_protocol off, in case someone can figure it out.

With proxy_protocol on:

Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: A76CA50A28C563074570BA6B7635030F8376ECE01647D49EB311AE9167A26DC5
    Session-ID-ctx: 
    Resumption PSK: AE5F539F1CC8A238542F898E6776EA4DECE5EB3921C55491D4273902C5355349E300139291E979264B5279D1EF2BC533
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - c6 92 47 e2 3a 9b a9 0e-38 f8 ff dc a9 c2 45 b1   ..G.:...8.....E.
    0010 - a7 20 1e fa 3e 08 dd ad-48 24 8a 26 69 b0 cd 27   . ..>...H$.&i..'
    0020 - 04 c3 63 60 59 38 e7 4a-af c5 03 1c 05 0c a0 3a   ..c`Y8.J.......:
    0030 - c0 9b d0 33 e0 81 c7 f4-e6 e2 59 31 96 07 6f c5   ...3......Y1..o.
    0040 - 0c 2f 57 87 77 9d 91 ba-0f f6 09 e3 a5 92 ab 3a   ./W.w..........:
    0050 - 95 77 78 22 51 ee 38 90-a7 2c b9 4a 6c 81 77 8a   .wx"Q.8..,.Jl.w.
    0060 - 94 41 f0 e9 c7 55 af ef-16 b4 fe e6 cf 8c 9c 54   .A...U.........T
    0070 - 52 7b 40 ab 8a 4d 21 ec-13 28 e1 cb 47 30 d8 a3   R{@..M!..(..G0..
    0080 - a4 67 4e f9 15 3c d2 3b-ce 49 f8 cb ef ee ca b8   .gN..<.;.I......
    0090 - b8 7a 89 ae 56 17 33 51-c8 86 91 39 64 77 16 45   .z..V.3Q...9dw.E
    00a0 - 4b a3 dd 5e 95 23 75 e1-7b be fc 88 dc aa 07 e2   K..^.#u.{.......
    00b0 - 34 d7 78 f2 a3 7d 16 48-32 a5 8c b1 c2 59 b2 94   4.x..}.H2....Y..
    00c0 - aa 75 ac 75 42 02 2f 9b-b2 c4 c6 78 3d ec 5c a6   .u.uB./....x=.\.
    00d0 - d8 bd 43 37 6a 2e 67 11-77 8f 91 2a 6e 64 e1 92   ..C7j.g.w..*nd..
    00e0 - c7 21 a8 bb 27 f6 9c 48-1c bb 0d d0 62 26 f7 08   .!..'..H....b&..

    Start Time: 1732398063
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
read from 0x55585f8cdc60 [0x55585f8d4b63] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 13                                    .....
read from 0x55585f8cdc60 [0x55585f8d4b68] (19 bytes => 19 (0x13))
0000 - 45 f0 c9 36 08 0c e5 3c-84 9e d3 cf 83 e9 8d 5f   E..6...<......._
0010 - 94 c6 d8                                          ...
closed
write to 0x55585f8cdc60 [0x55585f8d8cb3] (24 bytes => 24 (0x18))
0000 - 17 03 03 00 13 fa 23 21-64 a1 ba 0a 73 c6 b7 c3   ......#!d...s...
0010 - 5c 85 78 24 44 b7 04 12-                          \.x$D...
read from 0x55585f8cdc60 [0x55585f825670] (8192 bytes => 0)

With proxy_protocol off:

Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 9955619784B38DBE4A2C8D40CA7845ED20492F22070EF8AEF09CD9766FDADBC3
    Session-ID-ctx: 
    Resumption PSK: 4E08924F4A898EB6313C20000331D0A6099B0E27EB6C29C5F9E62D77E85A71581932006908503AB7EE6AC367D5163417
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - c1 b2 a0 3e a2 da e3 16-74 e4 7c 00 4c b6 85 f8   ...>....t.|.L...
    0010 - db c3 39 4a f0 13 a9 fe-0f 66 05 ef 81 94 8a 02   ..9J.....f......
    0020 - fb d5 9e c8 04 4d 11 86-31 ba a4 e7 60 a2 dc 44   .....M..1...`..D
    0030 - 03 73 8c fa 27 a2 b9 a4-16 be 70 6d 22 61 3b 48   .s..'.....pm"a;H
    0040 - 2f 3e 67 70 fa 3b ca 69-24 ef 0b b3 e5 27 59 0f   />gp.;.i$....'Y.
    0050 - de 33 bb f1 01 6c 77 f0-0c b0 ac 9f 54 fe c4 0e   .3...lw.....T...
    0060 - d4 f4 6a 91 67 d4 ae 4e-41 a1 1b ec 1b 82 36 13   ..j.g..NA.....6.
    0070 - 8b 8d 99 e2 4d 5a d0 0a-e1 99 53 ce 29 ee a5 38   ....MZ....S.)..8
    0080 - d2 3c 3a f5 ea be d3 79-b5 91 e0 6c 14 4e d5 97   .<:....y...l.N..
    0090 - fb e0 49 ae ec af 16 71-b7 62 db e9 6a 9b 0b 22   ..I....q.b..j.."
    00a0 - db 19 10 6f 70 bd d0 11-8f e0 5a 79 40 11 ff bd   ...op.....Zy@...
    00b0 - e2 c1 f3 a7 6f e0 54 97-8b 1a 18 fa db 4e 7e 3f   ....o.T......N~?
    00c0 - 46 bc 70 6d 9c 4e a7 19-b7 51 33 99 76 b8 da 49   F.pm.N...Q3.v..I
    00d0 - a3 f6 26 96 f8 7f 4f 7b-d7 02 18 b8 f6 d3 d1 ed   ..&...O{........
    00e0 - db 83 c6 eb 19 8c f3 00-f1 e9 58 8e d9 80 58 be   ..........X...X.

    Start Time: 1732397226
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
read from 0x5645dd24fc60 [0x5645dd256b63] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 3f                                    ....?
read from 0x5645dd24fc60 [0x5645dd256b68] (63 bytes => 63 (0x3F))
0000 - 24 86 ff a4 db 9d c3 e1-71 4c c4 94 3c 5a ac 16   $.......qL..<Z..
0010 - 8b e7 75 c2 63 bf a0 67-9e d7 d8 71 97 21 03 f9   ..u.c..g...q.!..
0020 - ca b6 6d b5 22 08 34 21-7c 9a 83 e6 75 ab e0 a7   ..m.".4!|...u...
0030 - 06 24 e5 63 86 97 d5 b6-b3 ac ab d9 4a c5 d9      .$.c........J..
220 localhost ESMTP Haraka/3.0.5 ready

The latter is what happens if the proxy IP is not in the haproxy_hosts file. If it is, the log waits after read R BLOCK (for the proxy commands I presume). This behavior is expected.

But with proxy_protocol on, the log is always the same and the connection is always closed right away with Nginx error peer closed connection in SSL handshake while SSL handshaking to upstream, no matter wether the proxy IP is in the haproxy_hosts file.

As mentioned in the first message, this happens only on the smtps port (e.g. 465). On non SSL ports (e.g. 587 and 25), Haraka's haproxy support works fine with Nginx's proxy_protocol on.

[DoobleD]

I think I can confirm that it breaks because the PROXY header is sent first (before the SSL handshake), as per the proxy protocol specs.

Adding this log after creating the smtps TLS server:

server.on('tlsClientError', (err, socket) => {
    console.error('TLS error:', err.message);
});

We get:

TLS error: 4018D13D317F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:

Suggesting that Haraka is attempting to read a malformed SSL version header, which in reality is the PROXY header.

I then tried fiddling with Haraka's code, to create a TCP server instead of a TLS one, that reads the PROXY command and then upgrades to a TLS server. I could have the server read the PROXY command successfully, but after that so far I can't make it work.