Skip to content

Separate Download Bool for images and workspace #193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ErinWeisbart opened this issue Feb 5, 2025 · 3 comments · May be fixed by #196
Open

Separate Download Bool for images and workspace #193

ErinWeisbart opened this issue Feb 5, 2025 · 3 comments · May be fixed by #196

Comments

@ErinWeisbart
Copy link
Member

ErinWeisbart commented Feb 5, 2025 •
edited
Loading

Might want to mount one bucket but download from the other (SOURCE_BUCKET vs WORKSPACE_BUCKET).
(e.g. reading images off of the public bucket Cell Painting Gallery, mount is good but workspace files are likely in another bucket)

We currently only support:
Same bucket - Download = True for both
Same bucket - Download = False for both

I think we want to support:
Different bucket - Download = True for both
Different bucket - Download = False for source, Download = True for workspace

I think we don't need to support:
Different bucket - Download = False for both (i.e. mounting two buckets)

I'm indifferent to supporting:
Different bucket - Download = True for source, Download = False for workspace
@ErinWeisbart
Copy link
Member Author

I'm leaning toward:

  1. we have mounting for just the SOURCE_BUCKET (where the images are) and default to download workspace files regardless of download bool. Pro, doesn't expand the config, easily adds on support for the two additional options I think we should support. Con, might be situations with large workspace files that we really would want to mount if we're getting more into DL stuff?? If so, could likely workaround by making custom docker as likely large workspace files would be consistent between batches.

Possible that instead:
2) we separate the bool for mount of different buckets. Pro, more configurable. Con, would mean we would need to support mounting two buckets as I really don't want a situation where someone could set something in the config that we don't support

@ErinWeisbart
Copy link
Member Author

@bethac07 do you have an opinion?

@bethac07
Copy link
Collaborator

bethac07 commented Apr 7, 2025

I think your preferred option (second bucket is always download, current boolean is only for the source_bucket) makes a lot of sense to me!

@ErinWeisbart ErinWeisbart linked a pull request Apr 8, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

force workspace download DistributedScience/Distributed-CellProfiler
2 participants
@bethac07 @ErinWeisbart