more fine-grained API?

[tmbdev]

I'm considering porting over an open source library (webdataset) for reading and writing large scientific and machine leanring datasets (petabytes) to Julia. We currently have implementations in Python, Golang, and C++. This library requires processing files as a stream rather than processing the entire archive at once; the code for the reader is actually quite simple.

Tar.jl seems to have the necessary low-level functionality, but its documented API only provides whole-archive operations. Would it be possible to expose the tar operations in a way similar to Python's tarfile API?

[StefanKarpinski]

That could certainly be done. Let me take a look at the tarfile API and see what makes sense.