Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Improve integration with smart_open #97

Merged
merged 4 commits into from
Apr 17, 2024
Merged

Python: Improve integration with smart_open #97

merged 4 commits into from
Apr 17, 2024

Conversation

pspoerri
Copy link
Contributor

@pspoerri pspoerri commented Apr 15, 2024

The GEDS smart_open integration has been extended so that additional GEDS configuration options are supported:

  • GEDS_AVAILABLE_STORAGE: Configure the storage threshold for GEDS.
  • GEDS_AVAILABLE_MEMORY: Configure the memory threshold for GEDS.
  • GEDS_RELOCATE_WHEN_STOPPING: Relocate files in GEDS to a configured object store when GEDS ist stopped.

Changes to the smart open integration:

  • GEDS is now stopped when the application exists.
  • relocation is exposed as an API.
  • open(file, mode='wb') now always creates a new file in GEDS
  • open(file, mode='ab') does an explicit copy of the file if the file is not writable (e.g. on a remote node).
  • The Python API has now been improved so that GEDS read calls directly allocate a writable Python buffer.

@pspoerri pspoerri force-pushed the smart_open branch 2 times, most recently from b50d5f4 to b4d7c32 Compare April 17, 2024 17:23
@pspoerri pspoerri changed the title WIP: Python: Fix handling of files in smart_open. Python: Improve integration with smart_open Apr 17, 2024
@pspoerri
Copy link
Contributor Author

pspoerri commented Apr 17, 2024

UBI8: Build failed.

@pspoerri pspoerri merged commit 0e7132c into main Apr 17, 2024
1 of 3 checks passed
@pspoerri pspoerri deleted the smart_open branch April 17, 2024 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant