Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(devservices): Add healthcheck wait condition and parallelize starting of containers #178

Merged
merged 20 commits into from
Dec 23, 2024

Conversation

hubertdeng123
Copy link
Member

This starts containers in parallel and makes sure devservices waits until containers are healthy before succeeding.

@hubertdeng123
Copy link
Member Author

I think relay sometimes doesn't pass the healthcheck, I may need to increase the retries there

Copy link

codecov bot commented Dec 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.85%. Comparing base (8aae9ef) to head (138b41f).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #178      +/-   ##
==========================================
+ Coverage   92.56%   92.85%   +0.29%     
==========================================
  Files          22       22              
  Lines        1412     1470      +58     
==========================================
+ Hits         1307     1365      +58     
  Misses        105      105              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@joshuarli joshuarli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imo --wait should be default

Copy link
Member

@joshuarli joshuarli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we also define a redis healthcheck in sentry's devservices config? otherwise we could run into Redis is loading the dataset in memory during migrations in sentry sync.py

@hubertdeng123
Copy link
Member Author

hubertdeng123 commented Dec 17, 2024

imo --wait should be default

I agree

can we also define a redis healthcheck in sentry's devservices config? otherwise we could run into Redis is loading the dataset in memory during migrations in sentry sync.py

It is already defined:

https://github.com/getsentry/sentry-shared-redis/blob/9c15e58b7b254e2fd4621fb47d790c22d1eeaf3e/devservices/config.yml#L14

@hubertdeng123
Copy link
Member Author

Decided to use in-house logic for healtchecks because it seems like --wait is not reliable for relay unfortunately. Healthcheck is failing when the container itself clearly is displaying as healthy. Spent a few hours trying to figure out the problem with docker compose, and decided this was better as in the future we can add logging to indicate specific services have passed these healthchecks.

@@ -30,13 +87,14 @@ def get_matching_containers(label: str) -> list[str]:
[
"docker",
"ps",
"-q",
"--format",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docker container names are unique so this will be safe. Decided to make this change so that we could print a better status message if containers fail healthchecks

@hubertdeng123 hubertdeng123 changed the base branch from main to hubertdeng123/use-namedtuple-docker-compose-command December 18, 2024 23:51
Base automatically changed from hubertdeng123/use-namedtuple-docker-compose-command to main December 19, 2024 17:44
@hubertdeng123 hubertdeng123 force-pushed the hubertdeng123/add-healthcheck-wait branch from d4f7b3d to 9814382 Compare December 20, 2024 17:12
Copy link
Member

@IanWoodard IanWoodard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Left a few comments but otherwise we should be good to go

devservices/utils/docker.py Show resolved Hide resolved
tests/utils/test_docker.py Outdated Show resolved Hide resolved
tests/commands/test_up.py Show resolved Hide resolved
tests/utils/test_docker.py Outdated Show resolved Hide resolved
tests/utils/test_docker.py Outdated Show resolved Hide resolved
devservices/utils/docker_compose.py Outdated Show resolved Hide resolved
Comment on lines +162 to +165
container_names = get_container_names_for_project(
cmd.project_name, cmd.config_path
)
containers_to_check.extend(container_names)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: since we don't use container_names anywhere else, why don't we combine these two lines?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually opted to do this here, since I think it's more readable than

containers_to_check.extend(get_container_names_for_project(cmd.project_name, cmd.config_path))

tests/utils/test_docker.py Outdated Show resolved Hide resolved
@hubertdeng123 hubertdeng123 merged commit cd9edb9 into main Dec 23, 2024
14 checks passed
@hubertdeng123 hubertdeng123 deleted the hubertdeng123/add-healthcheck-wait branch December 23, 2024 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants