Looking for help on "API error: reconnecting" error when accessing externally

[reddwarf666]

I have Backrest running on Docker. Whenever I try to access this externally I get the error "API error: reconnecting".
But when I connect internally, i.e. directly from my laptop to the docker server, I do not see this error.
My setup is running Backrest running on docker with a reverse proxy + authentication in front of it.

I tried different reverse proxies (NPM, Godoxy) and the issue is the same.

I have Backrest running 2 different servers, one at home and one at Hetzner.
One is accessible via "backrest.sub.example1.com" (Hetzner) and another via "backrest.example2.com" (home)
Setup at Hetzner: NPM + Oauth2Proxy + Pocket-ID
Setup at home: Godoxy + Authentik

Accessing "backrest.sub.example1.com" I get these errors from OAuth2Proxy:

backrest.sub.example1.com POST / "/v1.BackrestSyncStateService/GetPeerSyncStatesStream" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:141.0) Gecko/20100101 Firefox/141.0" 502 2240 30.001
[error_page.go:91] Error proxying to upstream server: net/http: timeout awaiting response headers
backrest.sub.example1.com POST / "/v1.Backrest/GetOperationEvents" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:141.0) Gecko/20100101 Firefox/141.0" 502 2240 30.002
backrest.sub.example1.com POST / "/v1.Backrest/GetOperations" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:141.0) Gecko/20100101 Firefox/141.0" 200 1350 0.003

Accessing "backrest.example2.com" I get these errors from Godoxy:

DBG http proxy error error="net/http: timeout awaiting response headers" url=backrest.example2.com/v1.Backrest/GetOperationEvents
DBG http proxy error error="net/http: timeout awaiting response headers" url=backrest.example2.com/v1.BackrestSyncStateService/GetPeerSyncStatesStream

Does anybody know what could be wrong here?

[garethgeorge]

Backrest uses streaming capabilities of h2c over http1.1 when using a reverse proxy (or http2 natively when available with a supporting reverse proxy).

It looks like you have a reverse proxy sitting in front of backrest which isn't playing nicely with the streaming requests, it's waiting on response headers which aren't going to come since those are long requests that never terminate (until dropped by the client disconnecting or if the session times out).

Does godoxy have any docs on how it handles streaming requests? Or do you have any middleware enabled that might not be playing nicely?

[reddwarf666]

Hi, thanks for responding!

Godoxy or NPM exhibit the same behaviour it seems.
Here's the Godoxy project:
Github Godoxy
Wiki for Godoxy

With NPM I have set the option for "websockets support", if that helps. The SSL portion is set to enforce SSL and HTTP/2 support.
But I do have OAuth2Proxy in between NPM and Backrest, to support Pocket-ID.

With Godoxy I use the midlewares option to involve Authentik

Do you perhaps need some logs or settings from any of these services?

[reddwarf666]

Ok, with the help of the Godoxy author @yuzing I figured it out.

In the OAuth2Proxy config file I needed to set upstream_timeout to a higher value then the 30 seconds and for now I set it to 600 seconds.
Documentation for this setting

For Godoxy it helps to set the label response_header_timeout and I also set that to 600 seconds.
Documentation for this label

This really helped a lot and it seems to have fixed my problem as the API error did not appear anymore and the UI behaves like expected.