Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

We need to re-look at liveness / readiness Probs logic of our pods #2992

Open
t83714 opened this issue Oct 2, 2020 · 7 comments · May be fixed by #2997
Open

We need to re-look at liveness / readiness Probs logic of our pods #2992

t83714 opened this issue Oct 2, 2020 · 7 comments · May be fixed by #2997
Assignees
Labels

Comments

@t83714
Copy link
Contributor

t83714 commented Oct 2, 2020

We need to re-look at liveness/readiness Probs logic of our pods

We have done some work around liveness/readiness Probs for zero-downtime deployment here: #1471

We need to re-look at it to make sure that, during k8s rolling update (particularly for registry API), if DB is not accessible, will liveness/readiness Probs reports this correctly.

@t83714 t83714 added the bug label Oct 2, 2020
@t83714
Copy link
Contributor Author

t83714 commented Oct 2, 2020

Registry seems not checking anything with database at all and just simply reply OK (for both liveness & readiness):

path("ready") { complete(ReadyStatus(true)) }

@soyarsauce
Copy link
Contributor

:feature:

@soyarsauce
Copy link
Contributor

At a minimum we should resolve this for these ones that utilise DB, namely authorization-api, content-api, registry-api

@soyarsauce
Copy link
Contributor

@soyarsauce soyarsauce self-assigned this Oct 6, 2020
@soyarsauce soyarsauce linked a pull request Oct 6, 2020 that will close this issue
3 tasks
@soyarsauce
Copy link
Contributor

turns out authorization & content are OK
registry PR at #2997

@soyarsauce
Copy link
Contributor

@t83714
Copy link
Contributor Author

t83714 commented Nov 3, 2020

Add #3024 as the blocker as, currently, there is a performance bottleneck that only registry-full pod serve the /api/v0/registry endpoint and we can't scale it up.

Our UI always sends read requests to /api/v0/registry-read-only but we can't guarantee that third-party software will do the same --- especially, for metadata crawlers.

Querying DB in readiness probe may add extra burden to it when registry-full is already on full speed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants