Bad HTTP 503 On Proxied Hook Request

[hiddenalpha]

How To Reproduce

Start gateleen instance (I used paisa/eagle @ 03.00.22.00, but also reproduces with gateleen-playground).

Setup two terminals with nodejs available.

In 1st, run node reproduce.js --zaphod (see attachments further down). This will start a http server at 7080 which sets a hook on the gateleen at localhost:7012 to itself.

(optional, but recommended) Start tcpdump (or wireshark) with port 7012 to see what is going on.

In 2nd, run node reproduce.js --slarti. This emulates a client performing two HTTP requests to that same gateleen, which then should get routed to the --zaphod mock server.

Expected Outcome: Both slarti requests receive a HTTP 200 OK with the small example response.

Actual Outcome: One of the requests (usually the 2nd one) fails with HTTP 502 Bad Gateway.

A Few Details

About the situation:

• There's a gateleen instance running.
• There's a service which places its service hook on that gateleen. (see my --zaphod mock).
• There's a client, doing multiple requests intended for that other (zaphod) server.
• gateleen now should proxy those requests to zaphod and then hand back the results to slarti.

Here what I've observed:

Client (slarti) gets 200 for its 1st request, but a 502 for its 2nd request. This is because gateleen tries to re-use the upstream connection which got closed in the previous request-response cycle. This connection re-use MUST NOT happen. Fiddling around with logging, I was able to get this message in the logs:

So obviously gateleen at some point realizes that the connection is closed. But this is already too late. Nevertheless gateleen MUST recognize this earlier, as upstream response clearly announces the closing via Connection: close header and therefore MUST NOT re-use that connection.

Also worth noting:

When creating a plain vertx app with similar configuration, it is impossible to reproduce the bug. Vertx always behaves exactly as it should. Looks like eagle/gateleen/friends do something ugly somewhere.

More detailed analysis is available in SDCISA-22791 (not publicly available unfortunately).

Attachments

reproduce.js.gz.b64

H4sIAAAAAAAAA81Y3XPbNhJ/lv+KDR8aKpYokbnWqXR+4NmK7TSx3Eip0zadDixBEh2KYAnIsuPqf79dgN+ic725Ppw9oyGB3cV
+7w8c2otNNFOBiOz248HBTERSwUTMPnMFx5DwPzZBwm0r4spqO2Z9mFKtlIorNGLOB7RotTMSfh+QmDgRMy6lQ6/ZllRzniSlTb
MwPOj1DMGSq0uxHcWz1TuJZKdMcScS2+FBpmUolrj+uE0CxQe5FWu5RENawcKGZ/js8GgurwO1sq1PkdWGNqhVIrYwShKR2BaTk
ifKrhA+/xQ9b7fJiFZVN0cfZUd8q7Wx244SF5PxRCVBtLTbcGhBF6xDIBWGB7sdqZrphWYGkTrnYUx+LgsWG5UKtqyDVguFAGqa
Pk3Hp+PfX51e+P7oVej7/7od+74P10T+jhdkxVO3+4XFKzEvVuhvolii0HIOm1iqhLM1oN13PIHtKpitIA7ZgwTD6TSLlSGKCKp
iTwUwMBs6DbhUVW5yQ9kJqbvhjoUdchPkocKVemxyT7YSrjZJRFw1eZKrX7TS50J8toHFsZbY65FMlWx4+trCXMmcfO2/vxxAwQ
bzQLKbkM+NvkRtztPPu4MWSmjdsQRiplaYcFbP+Kn3+wq5ZS/BAHJrmBKhG5CGqsBJPWITf2slpBpk9qOeDi20O7QVi6S6RQtma
80VnjQA6+rD1DLEqMRAq6JfV5xhZsoB/EpvLescpVqdvXMwNQfozsYzWtaJiBSPVHf6EHNktnA/DGaMPNy7lSKyCrqIa8cT1SwU
kmdbH7uj+xi7QNdfKJ7Qtntk9n6j310HRPSeyxgLl5OT0cV/OFitFqdYW2Zbu0sHPyfBorTfTMaXWChUZMHiwbjTMp6RFlquT2h
Zc2QOIq01rloUgkGv53pHTh//3Zr9aa5rN6B30pgala1ZbuiVEOEk+MJRotuB3guY8wXbhOr42z686BFxr2et2f01C9SPG77hKX
G3St11C2oVrDmmDBK97FeIXvbhLWUUhMFnjqmNWsyl4du1ySO7PO8LZ9qQSJP0uo7wBV3F1EaeYDeGZ8fg9ftmu1wD59PpVQ8dQ
uTkqJ8wh0gulm26WhKStTWd/4dWWiYtrIzWXFwof8mC6C32w8TeV7IIqY3DwChS0uM1C7DwQAmsm2UgUQZQUQ3oOH6P7TXrrcWx
Xz2yvknHYY+YGpfblXbRgaN+HyOA2aCFVDuLiJBSzC+imVijBqklusV0qMg7ud9rZyy+nIx8/4c332G3/nm29f0OuP1+Vc09GqM
p9Q9qisfwtRFjLHiHoxgkde1VoJtP4VUSUfhLB5Pj5KFGYTfWcE7XxF82be7h9Hm9fen7Jx9Ols2m1WmMaSSdarnB08kmKkwqen
ipSNNRlbbVGY4vxSd6zW4Ok3MT4FlpbOnIPWFOSNkW2U90hA6UUVElZUvuNzLwWMwWytialMKJO/qpziqTdXVHTPQsrTqCsiJKw
4a1cgz9oUk6bB1b/jzhsBSkAxZRxJFg9NE/mb79GdRWgJrFUDQzCVvEN7AIIt5dJlglfI5i4AWRqESEJAITbblEZxNWuNksHdN/
yDIz5x2Jyrlpiho8aD9q++pE3l8hcp1UPfsRnhiR8MR8BD1Ucsc0qvC3Si93tnzDzlvv4WEpStR4Ec/kQIKCdSPQ+VJ7Q2ZRQdJ
GB+vZOGeKWaVUxIAsMCWgToydgjnxRq5sJGgPoTkc/41Ir1kkGjEt5YdLML6C/owxuo/p+Uxw6WyE0GUPPKUM3TWe1nXL8ImSSy
Ph1mEmxYyldP6AHl6u435KDNZEsHmug1ud8F8DPlSaBXfREwfwmfO4y8LgjpcIPnap3QQzPkjt1WqXFdCPtTbcoHvWVPfCbVjIe
gzOt/0Z4n0X54LfP8PfWoetb5sUxNhQPLJAIOEW8ZgJCMUNLxKwZg83HBCAUL+gZiGBzW83Uq0R/qVQo9zv+Y9937/8/gxPOnqj
+/0/vJoydZJcmQWd7kXz/zE1vHJqeA2pAf+XueH9bbnhFbnhYW7cXwvfP399jf7++MtyLzfq23vjSzehepLgtdqBM7SLyhqvhmK
ts4TTQJixjeR0Sccr0lo2oYKzCebj2Yjy8eXCT1FBHRbUiQrEg9CASr6pqQ0LEm+fxMtIUuDrOiGPlpRLOCPhzz81W2UNG13vRU
Rg04BneODK4PL/aI+2urgYavSbaq6PvsUZbGdAyjOrXmm1FIJPUZf+wJ1M4f1ocjW+nIxAL6F3D40/DnMq7/L0CSqvRDXCQhsvI
LsVyJwy7dtU/s2Dhq5NiXiwm3OvvLuP20porSiiQBqYgy4/NndwDMXexoKF5iZY4TT4p4GztJFzUtgrcg3SrMDJDNxz5IGCIRWX
MZRgV5mhVjqjy/F08uEqcyp9zLLdBsesWWBgo/lMRS46hvHNLbYRRJ8s1DdYA0es/Hqqb54Gixz1XY/eMrsGxmSzZFQtLcl0QV/
xdEgHEG3CsJO9e7V3XVn5tTkvpGxlRz9f0oMfi+8TR/1XfcOhAXQhlBg0PKBKZcnyrvRNj15xZ0HXP9oOfNx0YUgP/9TEWX0OET
/hYt4VcI8qHil+DfzfsionfhLwzTeAYwJPmW9mvOcoupohbTu96RKeDaINH2K5B8tIJNTYAon9jpvrdJEOdAwOlW53xcPYSvlL3
+qGJQzXxJZ+N0gZq2muc/gpRhO3KmOW5VVGE9nyF0hrdHH5k//WTLWkfFsrZSW1qF1DlaDv9ougWfwJiyKhADGDHhtikcE9hu3G
WA7MfFeULJ0ZTlN9mG98dTWe/VU93tFnCDxdxDxhpsYCKRHENB9Gnam4X5lypG+xO5sug/8GSAqVP/IWAAA=