Object Packing Support

[TimucinK]

Hello everyone!

I was wondering about object packing. Recently I have a need for an S3 compliant object storage. Most of my data is 2KB or less and with more than 50 Billion objects with 300000 new objects daily. I have been looking into different solutions and Noobaa for my purposes really stands out. I need some mirroring configuration. I will not get into the details right now.

I was thinking to use Noobaa on top of 2 or different S3 storages (Ceph for one) and mirror the data between them. However there are some real problems with it because of space amplification due lack of object packing support for almost any S3 storages out there (There is a great article about it from Software Heritage https://wiki.softwareheritage.org/wiki/A_practical_approach_to_efficiently_store_100_billions_small_objects_in_Ceph). I noticed that NooBaa stores objects in chunks but I think it is mainly for splitting large objects rather than packing small objects together.

I know that even implementing something like this would be a big undertaking but I wanted to hear your opinions about it and maybe some advice ?

Thank you everyone and have a nice day.