Is your feature request related to a problem? Please describe.
When someone says import parsl, a lot of stuff gets imported: for example, all executors, including some executors which actively do stuff on import such as Globus Compute.
This is quite expensive. Although that is arguably acceptable at workflow startup time as a one-shot cost, this also happens in any Python process where parsl code is used.
So this full heavy import happens on every htex worker process, for example, or on every task with per-task workers (such as work queue without coprocess mode) and potentially inside every task if remote-side monitoring switches to spawn away from fork (see #3723)
When running Parsl from slow filesystems where file opens can sometimes take many seconds (or on occasion, minutes) this cost of imports is extremely damaging to performance.
Describe the solution you'd like
Stop import parsl and other intermediate packages from importing all their interesting sub-packages.
Users would now have to import the executor they wanted from that executor's package, for example, so that other executors would not be imported.
Describe alternatives you've considered
I have experimented in desc with replacing modules with dynamically-loading-modules that load a sub-import only when it is referenced by name. This also gives speedup, but was very ugly. It allows a user to import parsl and then refer to parsl.WhateverExecutor.
Additional context
Is your feature request related to a problem? Please describe.
When someone says
import parsl, a lot of stuff gets imported: for example, all executors, including some executors which actively do stuff on import such as Globus Compute.This is quite expensive. Although that is arguably acceptable at workflow startup time as a one-shot cost, this also happens in any Python process where parsl code is used.
So this full heavy import happens on every htex worker process, for example, or on every task with per-task workers (such as work queue without coprocess mode) and potentially inside every task if remote-side monitoring switches to spawn away from fork (see #3723)
When running Parsl from slow filesystems where file opens can sometimes take many seconds (or on occasion, minutes) this cost of imports is extremely damaging to performance.
Describe the solution you'd like
Stop
import parsland other intermediate packages from importing all their interesting sub-packages.Users would now have to import the executor they wanted from that executor's package, for example, so that other executors would not be imported.
Describe alternatives you've considered
I have experimented in desc with replacing modules with dynamically-loading-modules that load a sub-import only when it is referenced by name. This also gives speedup, but was very ugly. It allows a user to import parsl and then refer to parsl.WhateverExecutor.
Additional context