Skip to content

Improve file update performance #1

@gqgs

Description

@gqgs

S3 doesn't allow partial updates to objects.
That means if we want to update only a small section of the object the only choice currently available is to download the full object, update the desired section, and write the full object again, overwriting the previous version.

This is clearly wasteful and inefficient.
An option to improve this, for some use cases (e.g. music), would be to split the metadata and actual data into different file objects.
This would allow a fast and efficient update of the file metadata.
Small downsides:

  • Increase of complexity of managing files
  • Additional get needed to get metadata (metadata + data gets can likely be done concurrently)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions