River leader in multi-service ecosystem

[omriktalon]

Hey folks,
I've the following use case and I would like to hear your opinion on a limitation I faced once starting to tackle it.
I'm dealing with microservices architecture where N services would like to schedule periodic jobs (independently).
The services are compute distributed with centralized single DB instance (multi schema).
Meaning - each service creates a river client and set its internal scheduling jobs (worth to mention, each service defines its scheduling job kind).
In this scenario with the current leader election process there is a race between the services on the leadership ownership.
Let's assume service N_1 assigned to be the leader, as a result services N_2....,k schedule jobs will be starved (and actually won't run).
Are you aware of this limitation?
There is an alternative to achieve the requirements described aforementioned?
Thanks in advanced.

[brandur]

I'm assuming the different microservice workers wouldn't know how to work each others jobs would they? Do you get around this by using separate queues?

@bgentry I suppose we could address this by exposing the river_leader.name field and making it a per-client configurable value so that different clients could be part of separate leadership races. This wouldn't be perfect though, because it would mean other maintenance services like the job cleaner would end up running in multiple places (that may be okay, but it might cause contention).

[bgentry]

Are each of your services running on their own independent Postgres schema/namespace? I had always envisioned there being just a single leader in a given namespace, so if you want to have isolated services w/ independent jobs they would live on separate schemas and thus be able to have their own leader and their own maintenance processes for their own respective schema.

Our multi schema support is still held back by sqlc not supporting schemas other than as a connection param, but once we get past that we should be in a better place.

[omriktalon]

Thanks both @brandur and @bgentry.
There is a chance to proceeded with @brandur solution as a midterm bypass to the obstacle?
I can understand why its interfere the singularity of the rescue/cleanup jobs, but still it will give a significant granularity to the program to choose if to use a "global" leader or a dedicated "specific" leader.