Allow multiprocessshared to spawn process and delete directly with obj #37112

AMOOOMA · 2025-12-15T21:48:04Z

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

AMOOOMA · 2025-12-15T21:48:22Z

R: @damccorm

gemini-code-assist · 2025-12-15T21:48:27Z

Summary of Changes

Hello @AMOOOMA, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the MultiProcessShared utility by introducing the capability to host shared objects within dedicated, spawned processes. This change provides more explicit control over the lifecycle of shared resources, allowing for direct termination and cleanup of the serving process, and includes mechanisms to prevent zombie processes and handle server startup failures gracefully.

Highlights

Process Spawning for Shared Objects: The MultiProcessShared utility now supports hosting shared objects within dedicated, spawned processes, controlled by a new spawn_process boolean parameter in its constructor.
Direct Object Deletion: A new unsafe_hard_delete method has been introduced, allowing for the explicit termination and cleanup of the shared object and its serving process, providing more granular control over resource lifecycle.
Robust Process Management: The newly spawned server processes include a 'Suicide Pact' monitor, ensuring they automatically terminate if their parent process dies. The server startup also incorporates robust error handling and logging.
Zombie Process Reaping: The acquire method now includes logic to sweep and reap any finished (zombie) child processes, improving overall resource management and preventing accumulation of defunct processes.
Serialization Support for Proxies: The _AutoProxyWrapper class now includes __setstate__ and __getstate__ methods, enhancing the serialization capabilities of proxy objects.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-12-15T21:49:47Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

damccorm · 2026-01-13T18:12:56Z

sdks/python/apache_beam/utils/multi_process_shared.py

    assert self._SingletonProxy_valid
    self._SingletonProxy_valid = False

+  def unsafe_hard_delete(self):


Could you help me understand why we need the unsafe_hard_delete changes? Its not really clear to me what behavior this enables which we can't already do

It's mainly because the way that models are passed around is directly a _SingletonProxy instead of _SingletonEntry so we would need a way to directly call delete with the _SingletonProxy

Ok - lets at least give it a name like singletonProxy_unsafe_hard_delete. Otherwise we will run into issues if someone has an object with a function or property called unsafe_hard_delete, which seems like it could happen.

damccorm · 2026-01-13T18:15:18Z

sdks/python/apache_beam/utils/multi_process_shared.py

+    self.__dict__.update(state)
+
+  def __getstate__(self):
+    return self.__dict__


I assume this is so that this is pickleable, but is it valid? Normally I'd expect this to not be pickleable since the proxy objects aren't necessarily valid in another context

Yeah this is exactly what was needed for the pickling stuff. It does seems to be valid in testing with the custom built beam version loaded on custom container.

I think it would only be valid if you unpickle onto the same machine (and maybe even in the same process). Could you remind me what unpickling issues you ran into?

damccorm · 2026-01-13T18:20:14Z

sdks/python/apache_beam/utils/multi_process_shared.py

+    """Checks if parent is alive every second."""
+    while True:
+      try:
+        os.kill(parent_pid, 0)


Why are we sending a kill signal to the parent process? Isn't this the opposite of what we want?

This is not actually a kill signal but uses that interface to send a check, it will fail with OSError if the parent_pid is dead and if alive nothing happens.

if alive nothing happens.

Could you help me understand why this happens? https://www.geeksforgeeks.org/python/python-os-kill-method/ seems to say this will actually send the kill signal. Does the parent just ignore it?

damccorm · 2026-01-13T18:25:29Z

sdks/python/apache_beam/utils/multi_process_shared.py

-    self._get_manager().unsafe_hard_delete_singleton(self._tag)
+    try:
+      self._get_manager().unsafe_hard_delete_singleton(self._tag)
+    except (EOFError, ConnectionResetError, BrokenPipeError):


I'd typically expect the caller to catch/handle this. As it is, there is no indication passed back that this call failed.

Sounds good! Updated.

damccorm · 2026-01-13T18:26:03Z

sdks/python/apache_beam/utils/multi_process_shared.py

+    # Trigger a sweep of zombie processes.
+    # calling active_children() has the side-effect of joining any finished
+    # processes, effectively reaping zombies from previous unsafe_hard_deletes.
+    if self._spawn_process: multiprocessing.active_children()


Suggested change

if self._spawn_process: multiprocessing.active_children()

if self._spawn_process:

multiprocessing.active_children()

style nit to be consistent with the rest of the repo.

damccorm · 2026-01-13T18:28:33Z

sdks/python/apache_beam/utils/multi_process_shared.py

+    t = threading.Thread(target=_monitor_parent, daemon=True)
+    t.start()


Might be better to start this after we've initialized our MPS object to avoid racy unsafe hard deletes

damccorm · 2026-01-13T18:42:34Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a significant enhancement to MultiProcessShared by allowing it to spawn a dedicated server process and providing a mechanism for forceful deletion. The implementation is robust, incorporating features like a "suicide pact" for server process lifecycle management and detailed error reporting from the child to the parent process. The accompanying tests are thorough, covering various edge cases. I have a few suggestions to further improve the code, mainly around removing a redundant line of code, enhancing logging in exception handlers, and fixing a minor bug in the test setup.

sdks/python/apache_beam/utils/multi_process_shared_test.py

sdks/python/apache_beam/utils/multi_process_shared.py

AMOOOMA added 2 commits December 15, 2025 21:45

Allow multiprocessshared to spawn process and delete directly with obj

a5ee30d

Remove oom protection

0e8456e

github-actions bot added the python label Dec 15, 2025

AMOOOMA mentioned this pull request Dec 15, 2025

Add model manager that automatically allocate models and prevents OOMs #37103

Draft

3 tasks

damccorm reviewed Jan 13, 2026

View reviewed changes

gemini-code-assist bot reviewed Jan 13, 2026

View reviewed changes

Resolve comments

3c4ef28

	if self._spawn_process: multiprocessing.active_children()
	if self._spawn_process:
	multiprocessing.active_children()

		t = threading.Thread(target=_monitor_parent, daemon=True)
		t.start()

Allow multiprocessshared to spawn process and delete directly with obj #37112

Are you sure you want to change the base?

Allow multiprocessshared to spawn process and delete directly with obj #37112

Uh oh!

Conversation

AMOOOMA commented Dec 15, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

AMOOOMA commented Dec 15, 2025

Uh oh!

gemini-code-assist bot commented Dec 15, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damccorm commented Jan 13, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants