Which version control system?

[NathanCummings]

Your requirement in #3:

I think the set of data models allowed should be any which have the following properties:

version-controlled at rest, with a uniquely identifiable address/hash for each commit (i.e. Icechunk, git itself)

adds a level of complexity that not even GitHub has to handle (it supports git, and only git. It sunset its SVN support a year ago I believe).

What do we think about the need for a more unified data versioning approach? How far away are we from something that is essentially a combination of Icechunk, iceberg, and other technologies that can version-control unstructured data repositories, like lakeFS/DVC/(git LFS?)?

Or do you think that it would be easier to somehow support several of these technologies?

[TomNicholas]

This thing is more complicated than GitHub, inescapably. GitHub doesn't separate the storage layer, and has no need to support version control for non-text files. We definitely would need a separate storage layer (though it already exists, it's S3), and the ability to support more than one data type.

I don't know what would be easiest, but I think this is the right question to be asking. I do think you could cover 95% of use cases with only 3 or 4 data types.