Should ObjectStore sizes be u64 instead of usize?

[ttencate]

I noticed that object_store::ObjectMeta::size is of type usize. On 32-bit platforms, this would limit the value to 4 GiB.

There are object stores out there that accept files larger than 4 GiB. For instance, Google Cloud Storage supports objects up to 5 TiB. Common local filesystems (with the exception of FAT32) also support larger files.

There might be more APIs where usize is used to represent file size, but I didn't check.

[alamb]

I don't think it was intentional to limit the size of objects to 4GB on 32-bit platforms (though maybe @tustvold or @carols10cents have some different memory)

[tustvold]

It wasn't intentional, however, it has lacked for a compelling use-case to justify making what would be a very disruptive change.

One suggestion in the past has been wasm32, but the support for this is currently fairly limited.

TLDR yes it should be u64, but there hasn't really been a good use-case nor someone willing to drive support for 32-bit platforms forward

[tustvold]

Closing this as duplicate of apache/arrow-rs#5351

[ttencate]

I understand, just thought I'd mention it (and I failed to find the duplicate). Thank you the impressively quick response!

[alamb]

Migrating from arrow-rs issue #6111