source-*: Dynamic memory limits for Go connectors#4065
Merged
willdonnelly merged 1 commit intomainfrom Mar 24, 2026
Merged
Conversation
Previously all Go connectors hard-coded an assumption that they're running with a 1GiB memory allocation and so we should attempt to keep Go heap size below 90% of available RAM to avoid OOMing. After this change we instead read `/sys/fs/cgroup/memory.max` at runtime to see how much memory we're working with and set the Go heap limit to 90% of that. In addition `source-postgres` has been updated to use half of that value as the maximum CDC event receive buffer size, in keeping with the previous hard-coded 512MiB.
willdonnelly
added
the
change:unplanned
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Previously all Go connectors hard-coded an assumption that they're running with a 1GiB memory allocation and so we should attempt to keep Go heap size below 90% of that 1GiB to avoid OOMing.
After this change we instead read
/sys/fs/cgroup/memory.maxat runtime to see how much memory we're working with and set the Go heap limit to 90% of that.In addition
source-postgreshas been updated to use half of that value as the maximum CDC event receive buffer size, in keeping with the previous hard-coded 512MiB.Workflow steps:
No user action required. No behavior change expected on data planes with the default 1GiB connector memory allocation. Data planes with >1GB of connector RAM available should be able to make better use of that extra memory where necessary.
Notes for reviewers:
This should obviously have no effect in data planes with the default 1GB limits because the dynamic value will still be based on that 1GB limit. In data planes with >1GB available I think that this should still not increase steady-state memory usage, and if it does that suggests that the current 1GiB assumption is probably also doing the same thing and we should think about how to encourage connectors to release memory more proactively when they can get by with less.