[FIX] Limit MinIO folder depth to 2

[chibenwa]

Avoids creating tons of intermediate folders thus reduces inode pressure while still on disk splitting objects in 4.000 slots

So the MinIO retex gains complexity.

Actually we are encountering more complex than it seems issues with it.

After the OOM as everyting was stored in single GB big folder ( !!! ) that is impossible to ls, thus breaking MinIO inner work (...) which needs spreading accross several folders, we did run into a inode shortage issue (isize 512, inpode space percentage 5%, XFS). Our study found that each object stored resulted in ~5 inodes being used (including versionning, meta info part info, object entry, but also `/`` into the part.

Email being small blob intensive, we are running out of into before running out of disk space.

One generation with 4 / and storing ~2 million objects consummes 2,250 M additional inodes with the folders prefix. Density of level 4 is terrible: 1 folder for one object. Density for level 3 is also not good: one level for ~8 underlying levels.

We thus believe that 2 levels (whish already splits objects in slots of 4096 items thus can scale to hundreds of millions items per generation) is thus a much more default.

Please also find hereby recommended XFS settings / MinIO settings for a 4 node cluster:

• isize = 256 (which is XFS default and not 512)
• imaxpct = 10
• Errasure coding settings: 8 stripes with 2 or 3 parity block which allows for a 75-63% density while tollerating one server failure (and potentially one additional disk failure). Note choosing a strip of 16 is important as it spreads workload better

[chibenwa]

+1 thanks @quantranhong1999