Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OpenViSUS and idx2 integration] idx2 chunks being too small for cloud #8

Open
hoang-dt opened this issue Nov 28, 2022 · 0 comments
Open
Assignees

Comments

@hoang-dt
Copy link
Contributor

I have added an option to group multiple bit planes into a chunk (default is 4). To change it, provide, say --bit_planes_per_chunk 8, when encoding. With this change, the chunks are about 150KB on average.

The chunk size can also be changed with two options when encoding:

  • --bricks_per_chunk (default is 512). For example, --bricks_per_chunk 4096 should create chunks of size 1 MB ish.
  • --brick_size (default is 32 32 32). For example, --brick_size 64 64 64 should also create chunks of size 1 MB ish.

The first option (increasing the bricks-per-chunk) will result in more metadata to be stored, but is better in that the bricks are kept small so fewer samples need to be decoded when we ask for a small region of interest (e.g., a slice).

TODO

  • Can we plot the impact of the chunk size on data retrieval time?
  • Can we allow fetching parts of an object? (so that each chunk does not have to be stored as a file)
@hoang-dt hoang-dt changed the title [OpenViSUS and idx2 integration] idx2 chunk being too small for cloud [OpenViSUS and idx2 integration] idx2 chunks being too small for cloud Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants