Exceeded reuse of refresh token

[JohannesRabauer]

I received the following message in my quarkus application:

{
   "body":"Request https://some-keycloak.de/realms/some-realm/protocol/openid-connect/token has failed: status: 400, error message: {\"error\":\"invalid_grant\",\"error_description\":\"Maximum allowed refresh token reuse exceeded\"}",
   "severity":"ERROR",
   "resources":{
     ...
   },
   "instrumentation_scope":{
      "name":"io.quarkus.oidc.runtime.OidcProviderClientImpl"
   }
}

Now i got three questions that could help me here:

1. Can i somehow handle this invalid_grant-Error gracefully? It looks like the refresh token is not valid anymore (it is only valid 6 hours) and it might be, that this error is expected.
2. Does Quarkus check if the Refresh Token is still valid when using it to get a new Access Token? Maybe i have a very specific setup, where the refresh token is not valid for all eternity.
3. Could it be that the Keycloak-Server where i authenticate against, is requested multiple times almost at the same time and this is an concurrency issue? This should be handled automatically by quarkus right?

[sberyozkin]

Hi @JohannesRabauer

Can i somehow handle this invalid_grant-Error gracefully? It looks like the refresh token is not valid anymore (it is only valid 6 hours) and it might be, that this error is expected.

This is not a refresh token expiry issue, but a restriction on a number of times it can be used, as far as I recall, advanced client configuration in the Keycloak dashboard allows to configure it.

I'd expect a user re-authentication redirect in this case, is it what you are seeing ? There is also an option to redirect a user to a session expired page, with something like quarkus.oidc.authentication.session-expired-path=/tenant-refresh/session-expired-page, there is an example of such a page at https://quarkus.io/guides/security-oidc-code-flow-authentication#oidc-redirect-filters,

In general, you can also catch AuthenticationFailedException with a JAX-RS ExceptionMapper

Does Quarkus check if the Refresh Token is still valid when using it to get a new Access Token? Maybe i have a very specific setup, where the refresh token is not valid for all eternity.

If the refresh token is in a JWT format, it can check its expiry time, if it has expired, a re-authentication is initiated

Could it be that the Keycloak-Server where i authenticate against, is requested multiple times almost at the same time and this is an concurrency issue? This should be handled automatically by quarkus right?

There could be concurrent requests, Quarkus just forwards refresh requests to Keycloak, or what do you mean by This should be handled automatically by quarkus ?

[sberyozkin]

As far as the concurrency is concerned, do you mean a single user may have multiple tabs/etc that causes concurrent refresh requests for the same session ?

Right, we can tighten it

[sberyozkin]

@JohannesRabauer I doubt though it can make a difference unless the ID token lifetime is very small, say in a region of a few minutes.
Let's say the RT lifetime is 6 hours, as in your case. If you set ID token lifetime to 2 hours and RT reuse to say 10, you'd never see the problem even with a multi-tab same session token refresh

[JohannesRabauer]

Thanks a lot for your input @sberyozkin!

I'd expect a user re-authentication redirect in this case, is it what you are seeing ? There is also an option to redirect a user to a session expired page, with something like quarkus.oidc.authentication.session-expired-path=/tenant-refresh/session-expired-page, there is an example of such a page at https://quarkus.io/guides/security-oidc-code-flow-authentication#oidc-redirect-filters,

Unfortunately we don't know what the user experiences, but since we don't hear complaints, we suspect that the redirect is already in place.
The point that i don't understand is, why the error is thrown in the first place. "invalid_grant" is more of an expected error and with the current settings, no error should occur, as far as i can see.

In general, you can also catch AuthenticationFailedException with a JAX-RS ExceptionMapper

This would certainly be a way to suppress the error message, but would feel more like a workaround than a solution.

If the refresh token is in a JWT format, it can check its expiry time, if it has expired, a re-authentication is initiated

Does that mean Quarkus does check the expiry time or could it check the time? Maybe there is some kind of configuration that we did not set?

As far as the concurrency is concerned, do you mean a single user may have multiple tabs/etc that causes concurrent refresh requests for the same session ?
Right, we can tighten it

I think it could be very well possible, that the frontend, that uses this quarkus application, is calling multiple endpoints in the same session simultaneously.
So i thought Quarkus has something to prevent it from using expired refresh tokens even if they are called simultaneously.

For example: Keycloak is configured to only use refresh tokens once and then invalidate them. One call uses the refresh token and the refresh token is therefore invalidated. Another, simultaneous call uses the same (now invalidated) refresh token and gets "invalid_grant".
Could this happen?

@JohannesRabauer I doubt though it can make a difference unless the ID token lifetime is very small, say in a region of a few minutes.
Let's say the RT lifetime is 6 hours, as in your case. If you set ID token lifetime to 2 hours and RT reuse to say 10, you'd never see the problem even with a multi-tab same session token refresh

Yes, but i don't have control over the keycloak server (barely know its configuration). I suspect the RT reuse is 1 and this causes problems with quarkus.
Isn't the RT used to refresh the AccessToken? The ID token shouldn't be relevant here, right?

[sberyozkin]

Hi @JohannesRabauer

I'd expect a user re-authentication redirect in this case, is it what you are seeing ? There is also an option to redirect a user to a session expired page, with something like quarkus.oidc.authentication.session-expired-path=/tenant-refresh/session-expired-page, there is an example of such a page at https://quarkus.io/guides/security-oidc-code-flow-authentication#oidc-redirect-filters,
Unfortunately we don't know what the user experiences, but since we don't hear complaints, we suspect that the redirect is already in place.

Good news I guess and indeed, I had a look at the code to confirm. If the session refresh fails, a new redirect is initiated. 401 would be reported when for example a signature verification fails.

The point that i don't understand is, why the error is thrown in the first place. "invalid_grant" is more of an expected error and with the current settings, no error should occur, as far as i can see.

Keycloak throws it because its configured allowed number of the same RT reuse has been exceeded. I actually saw it in tests only, this is the first time I see such an issue reported upstream. I think just increasing that number a bit would fix it - but see below, we'll look at preventing concurrent same session refresh in any case.

@pedroigor Hey Pedro, where in Keycloak one can check how many times a RT can be reused ? I had a quick look and I could not find it.
@JohannesRabauer Do you recall how the Keycloak realm was setup, did you use Keycloak Admin API ? I recall that with the admin API, one can set this number...

If the refresh token is in a JWT format, it can check its expiry time, if it has expired, a re-authentication is initiated
Does that mean Quarkus does check the expiry time or could it check the time? Maybe there is some kind of configuration that we did not set?

There is no standard property that would indicate a RT expiry time in the authorization code flow response, so Quarkus does a best try and checks if the RT is in JWT format and if yes, it will check if it is expired or not. But the Keycloak error is not related to the RT expiry, but to how many times it can be reused.

I think it could be very well possible, that the frontend, that uses this quarkus application, is calling multiple endpoints in the same session simultaneously. So i thought Quarkus has something to prevent it from using expired refresh tokens even if they are called simultaneously.

This is not an expired RT case though.

For example: Keycloak is configured to only use refresh tokens once and then invalidate them. One call uses the refresh token and the refresh token is therefore invalidated. Another, simultaneous call uses the same (now invalidated) refresh token and gets "invalid_grant". Could this happen?

Yes, it might, and like I said we'll tighten it asap. But it may not be sufficient.
For example, If the allowed number is say 3, and if the ID token lifespan is 60 mins, then, in the course of the 6 hour RT lifespan, there would be 6 RT calls. We'll look at the concurrency situation in any case

Yes, but i don't have control over the keycloak server (barely know its configuration). I suspect the RT reuse is 1 and this causes problems with quarkus.

Sure, it is a Keycloak restriction that KC enforces per its configuration, this is not a Quarkus problem, it does not crush, it does what it should do when the session refresh can not be completed, initiates a re-authentication.
I do agree that in the concurrent same session refresh case, Quarkus must do better though and have a single RT request going through only.

Isn't the RT used to refresh the AccessToken?

In pure OAuth2 - yes

The ID token shouldn't be relevant here, right?

But how would an RP server like Quarkus deal with the expired ID token that is a representation of a user authentication and that is used to govern the Quarkus session lifetime ? Most OIDC servers would also return an ID token with the RT token grant.
If we don't attempt to refresh, then the only option is request a user re-authentication.

You can prevent Quarkus from using a refresh token altogether for sure (...refresh-expired=false) - and instead redirect a user to that session expired page, like most well known websites do, where an option to login again is offered to a user, instead of a user immediately redirected to Keycloak which is indeed not a great experience.

Let me know what you think, I can help with setting up the session expired page

So far, the action item for us is to memoise RT calls to avoid concurrent same session RT requests, I'll ping you once it is done

Thanks

[sberyozkin]

@JohannesRabauer

By the way, what exactly is the end goal, avoid a user re-authentication altogether ? The issue we are discussing here may cause a new reauthentication indeed, perhaps more frequently than expected, but overall, ultimately, the user will eventually have to re-authenticate in any case, for ex, when the RT actually does expire in 6 hours.

As suggested above, if the main concern is a user experience, for example, an authenitcated user is accessing some custom portal and now suddenly sees a Keycloak login screen due to a token refresh failing that can indeed be quite unexpected, then IMHO the best way to handle is to register a custom session expired page and it will keep it pleasant for users when dealing with such situations - this is what I see every day when going to my airline etc - I'm in a session and then I'm told the session has expired, and I have an option to relogin

[JohannesRabauer]

Good news I guess and indeed, I had a look at the code to confirm. If the session refresh fails, a new redirect is initiated. 401 would be reported when for example a signature verification fails.

This is great to hear, although unsurprising, because users would let me know, i think :D

Keycloak throws it because its configured allowed number of the same RT reuse has been exceeded. I actually saw it in tests only, this is the first time I see such an issue reported upstream. I think just increasing that number a bit would fix it - but see below, we'll look at preventing concurrent same session refresh in any case.

Unfortunately this server is not in my control and the people running it are very defensive with regards of information about it. But i will definitly try to reach out to them again.
But another thought: This behaviour could also be provoked by a backchannel-logout if the user is logged out of the auth-server (i.e. keycloak). Then all tokens are revoked and again an error message would occur, even though this is expected behaviour, right?

There is no standard property that would indicate a RT expiry time in the authorization code flow response, so Quarkus does a best try and checks if the RT is in JWT format and if yes, it will check if it is expired or not. But the Keycloak error is not related to the RT expiry, but to how many times it can be reused.

Ah, thanks! That's great info. So we can rule this out.

Yes, it might, and like I said we'll tighten it asap. But it may not be sufficient.
For example, If the allowed number is say 3, and if the ID token lifespan is 60 mins, then, in the course of the 6 hour RT lifespan, there would be 6 RT calls. We'll look at the concurrency situation in any case

Thanks again! That would definetly help.

But how would an RP server like Quarkus deal with the expired ID token that is a representation of a user authentication and that is used to govern the Quarkus session lifetime ? Most OIDC servers would also return an ID token with the RT token grant.
If we don't attempt to refresh, then the only option is request a user re-authentication.

Ah, i see what you are going for. I think you are right and there is no other way then re-authenticate, but again, this is expected and not an error at all. It might be logged in info or debug, but not error. Am i wrong?

You can prevent Quarkus from using a refresh token altogether for sure (...refresh-expired=false) - and instead redirect a user to that session expired page, like most well known websites do, where an option to login again is offered to a user, instead of a user immediately redirected to Keycloak which is indeed not a great experience.

Yes, i also think that this would be a bad user experience ^^

...what exactly is the end goal, avoid a user re-authentication altogether ?

The core goal would be to get rid of the error message, but only to be sure that the auth system (communication between keycloak and quarkus) is set up correctly.
So far i have the impression, that everything is working alright, but the error is still logged as error.