Oathkeeper returns a 401 "Access credentials are invalid" when exceeding Ory Network's rate limiting on /sessions/whoami endpoint

[wewelll]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

We're using Ory Network as our authentication provider, and Oathkeeper to protect our APIs.
Oathkeeper is configured with a cookie_session authenticator and uses the /sessions/whoami endpoint to check the cookie.

When a user makes a lot of calls to our API at the same time, Oathkeeper can sometimes return a 401 - Access credentials are invalid response, even when the cookie is valid.

I did a bit of digging with our OpenTelemetry instrumentation, and figured out that the /sessions/whoami endpoint is in fact returning a 429 status, because we're exceeding rate limiting.

I see two potential improvements:

• Change the response status

Instead of returning a misleading 401 status, Oathkeeper should return another status code, like 429, so it can help the end-user understand what is happening.

• Add cache to the check_session_url call

When a user performs a lot of calls in a small timeframe, it's not necessary to call the /sessions/whoami for all calls because the session cookie hasn't changed.
I suggest to add an optional caching mechanism to the cookie_session authenticator, it would solve this issue even with a low TTL (~1s)
This caching mechanism could be similar to the one found in the oauth2_client_credentials authenticator.

Reproducing the bug

• Run Oathkeeper with a cookie_session authenticator linked to an Ory Network project.
• Log-in and retrieve the session cookie
• Write and run a script that sends 100 requests in parallel to the running Oathkeeper, with the session cookie
• You should see a 401 at some point, even though the cookie is valid

Relevant log output

No response

Relevant configuration

authenticators:
  cookie_session:
    enabled: true
    config:
      check_session_url: https://my-project-slug.projects.oryapis.com/sessions/whoami
      preserve_path: true
      extra_from: '@this'
      subject_from: 'identity.id'

Version

latest

On which operating system are you observing this issue?

macOS

In which environment are you deploying?

Kubernetes

Additional Context

No response

[aeneasr]

That makes sense, we should definitely return 429 in this scenario instead of 401 which is masking the underlying reason for the error.

[wewelll]

This issue also shows that when we build an API on top of Oathkeeper & Ory Network, the rate-limiting of our API can't be better than Ory Network's rate-limiting on the whoami endpoint.

This is quite a big limitation ... I was suggesting to add a cache mechanism based on the session cookie, in order to avoid unnecessary requests to the whoami endpoint. If the TTL is low (~1s for instance), the security trade-off is acceptable.

Do you think that it would make sense to add this feature ? I see that it exists for other authenticators like oauth2_client_credentials.

In our case, we are trying to lift this limitation by creating our own whoami that has cache, but IMO it would be simpler to have it directly in oathkeeper

[sboily]

@wewelll let me know if you can test my PR to solve your issue and what do you think about it. On my side, it works perfectly.

[finsterwalder]

Is there any update on this?
This is really a pressing issue for us since sessions are reported as broken instead of the rate limit our users see auth errors. :-/

[wewelll]

@sboily I'm not a maintainer of oathkeeper unfortunately, I guess @aeneasr could help with the PR.

In my case, I fixed this issue by adding a small http server between Ory Network and Oathkeeper, that acts as proxy & cache for the /sessions/whoami endpoint.

I'm caching sessions with ~1s TTL, using the session cookie as cache key, and this has fixed my issues with 429 requests. Not an ideal fix, but it's a workaround

[finsterwalder]

@wewelll Thanks for this idea! I was also wondering, why Oathkeeper does not cache events for a short time, like a second or so. I think that would be a good tradeoff.
But I still believe we should fix the 429 handling independent of this.

[wewelll]

@finsterwalder I agree, 429 handling is independant

But concerning the cache, I think this should be a feature of Oathkeeper. This cache feature already exists for some oauth authenticators (see: https://www.ory.com/docs/oathkeeper/pipeline/authn#oauth2_introspection).

I need to find some time to push a PR for that

[wewelll]

Hey @aeneasr I have pushed a new PR: #1244 that both fixes the 429 error handling, and adds an optional cache for sessions.