Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: terminate zombie migration runners #741

Merged
merged 1 commit into from
Nov 17, 2023
Merged

fix: terminate zombie migration runners #741

merged 1 commit into from
Nov 17, 2023

Conversation

abc3
Copy link
Member

@abc3 abc3 commented Nov 17, 2023

While applying migrations to a tenant's database, I observed that Ecto.Migration.Runner does not stop when encountering errors. Upon brief inspection, I noticed that the Ecto.Migrator only stops the Agent if the migration succeeds. In the case of an error, the Migrator catches it and exits with a normal reason, which does not lead to the termination of the linked Runner process. As a result, this leads to memory leaks when Realtime continuously attempts to run migrations that encounter errors

As a quick solution from our side, we could stop all runners that are linked only to MigratorSupervisor after a failure. Normally, they also have a link to the Migrator that initiated them

Screen.Recording.2023-11-17.at.13.34.29.mov

Copy link

vercel bot commented Nov 17, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated (UTC)
realtime-demo ⬜️ Ignored (Inspect) Nov 17, 2023 0:36am

@abc3 abc3 requested review from chasers, filipecabaco, w3b6x9 and a team November 17, 2023 12:38
@abc3
Copy link
Member Author

abc3 commented Nov 17, 2023

@supabase/dashbit Could you take a look to see if this makes sense?

Copy link
Member

@filipecabaco filipecabaco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤩

Copy link

@philss philss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @abc3

This makes sense, yeah! And LGTM! We will investigate that on the Ecto-SQL side, because it could be a bug there :)

BTW, thanks for the supervision tree video and the details! :D

@abc3
Copy link
Member Author

abc3 commented Nov 17, 2023

@philss thanks for the feedback! Please let me know if you need any additional info to reproduce such behavior

@abc3 abc3 merged commit c818972 into main Nov 17, 2023
3 checks passed
@abc3 abc3 deleted the fix/ecto_runners branch November 17, 2023 16:17
@kiwicopple
Copy link
Member

🎉 This PR is included in version 2.25.34 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants