Rethink inactivity timeout under high load

[tillrohrmann]

At the moment, the inactivity timeout defines when a bi-directional streaming invocation w/o activity is suspended. The current default value is 60s. The idea is to keep the invocation running in case that a completion arrives that enables the invocation for make further progress. This works fine as long as the invoker has enough capacity to invoke other invocations. If, however, the concurrency limit is set and the system is under load it might also lead to a slow down of invocation processing. See #2758 (comment) for more details.

Under load there is the tension between keeping an invocation running and occupying the slot for making faster progress once a completion arrives vs. freeing its slot and letting waiting invocations make progress. One idea to avoid occupying slots w/o making progress is to suspend in-flight invocations whenever a new invocations arrives that does not get a free slot.

[muhamadazmy]

Just a thought: do we actually need an inactivity timeout at all? Perhaps we could keep the invocation alive indefinitely, as long as we haven't hit the global concurrency limit (or the service-level limit, if that's reached first). If we do hit a limit (or approach it, say at 95%) we could start evicting invocations based on the longest period of inactivity.

edit: I think this is probably what you mean by suspending invocations whenever a new invocations arrives that does get a free slot 🤔. If not then I am not sure what you mean by this suggestion

[tillrohrmann]

Yes exactly. This was what I meant with my last sentence.

There are, however, some problematic corner cases where invocations would repeatedly preempt invocations rendering the bidi streaming protocol to behave similar to the request response model. For example assume you have a single free slot and there is an invocation running. The invocation runs a couple of side effects. Now another invocation comes which causes the first one to be suspended. Now since the first invocation can still make progress it would suspend the currently running second invocation and so on. If the system would have completed the first invocation first and then second it might have finished faster because it could have made use of the streaming protocol and wouldn't have to pay the overhead of invoking invocations multiple times. This is something to think through and to evaluate.

[muhamadazmy]

I see your point, though I think this specific scenario would be quite rare when we factor in inactivity time. In practice, if a second invocation arrives, it would more likely cause a different (3rd) invocation to be suspended rather than the first one. Similarly, when the first invocation resumes to perform a side effect, it would likely suspend another invocation, not the second one that just started.

In other words, this kind of back-and-forth preemption seems most likely to occur only when the concurrency limit is extremely low, such as set to 1.

That said, I agree there’s room to refine the eviction strategy. A few ideas:

• Consider both the age of the invocation and the age of its last activity. More recent or more recently active invocations should be less likely to be suspended.

• Factor in the type of side-effect currently being performed. For example, lightweight operations like SetState or GetState should probably not trigger suspension, while longer or more expensive effects like Call or Sleep are more reasonable candidates for eviction.

[tillrohrmann]

Factor in the type of side-effect currently being performed. For example, lightweight operations like SetState or GetState should probably not trigger suspension, while longer or more expensive effects like Call or Sleep are more reasonable candidates for eviction.

At the moment it's unfortunately not easily possible to say what the service is currently awaiting from the server's perspective.

In other words, this kind of back-and-forth preemption seems most likely to occur only when the concurrency limit is extremely low, such as set to 1.

I think it's function of ready invocations (invocations that are awaiting a slot) and available slots. If the concurrency limit is 1000 and you have 1000000 ready invocations, then I would expect a similar behavior of repeated preemption.

From the server's perspective it would be really helpful to know whether an invocation is currently awaiting a result from the server or not to enable some form of cooperative scheduling. In the absence of this, I think preemption is probably the only feasible thing we could do right now.

@igalshilman, @slinkydeveloper would it theoretically be possible to tell the server whenever the invocation is blocked on awaiting a Restate future?

[muhamadazmy]

I have been checking the internal details of both the service invocation runner and the sdk-shared-core (only superficially) and I think the best approach would be to introduce a new control message that can be sent from the sdk.

• The propose-preemption or preempt-hint can be implemented as an extra optional flag on the SuspendV2 message. In that case, invocation will not actually be suspended, but will become a candidate for suspension if the invoker is hitting the concurrently limit.
• Since this new message is completely optional, it also can be implemented a completely new message. This does not require change in the protocol version (I think) and new cooperative SKDs can send it, older non cooperative versions of the SDKs will keep being handled normally (rely on inactivity-timeout)
• I also noticed we have a Custom message type that is not used and not implemented, we definitely can piggyback on this one although I would prefer introducing own message or better the optional new flag on the Suspend message.
• The new message/flag (Propose-Preemption or Yield) is only a suggestion from the SDK that this invocation can be suspended (if needed)
• The ProposePreemption message must include the possibly running effects/notifications (like Suspend), this will allow the runtime to avoid preemption if those effects has already been completed before this message was sent, for example

selector := restate.Select(call1, call2)
// do some work here 
selector.select() // -> this should send a propose-preemption even before polling for progress

Runtime should then should ignore the proposal if any of the awaited effects has been completed already.

[muhamadazmy]

A quick draft PR to show how this could work is open here

[AhmedSoliman]

I'd suggest first to take a step back and discuss the purpose of the concurrency limit. What resource does it protect, and is it the right control knob or not.