Too many files in .gd/indices -- use a database

[vrusinov]

Looks like drive creates a separate index file with some info for every file it pulls/pushes. All of them are being created in single directory .gd/indices.

If drive manages many small files, it quickly becomes not efficient & wasteful. A lot of tools won't work very well with lots of files in single directory. Also, this could be quite wasteful with metadata taking almost as much space as data itself.

I think ideally I'd like to see .gd/indices/ replaced with single (or a few) files. We could for example use sqlite or bdb or one of other embedded databases.

[odeke-em]

Yeah but the thing is how do we keep track of conflicts without taking note of the metadata? There was an issue in which the user was not happy because they lost data, even though drive stated it did a naive push and pull -- issue #57. Also the multiple files are there to increase the ability for parallel operations on file. Great suggestion on using a DB!

[nkabir]

Also consider LMDB/Bolt
https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database
https://github.com/boltdb/bolt

[odeke-em]

Hello @nkabir thank you for that awesome suggestion, and welcome to drive. I just read the Wikipedia reference you posted as well as just peeked at the project page. This looks a great match for this project. I'll take a better look at it when I get up. Thank you again.

[nkabir]

You're welcome! Your work is excellent and very much appreciated!

[odeke-em]

@nkabir with your awesome suggestion (thank you very much), I've been able to hackup something for saving indices and information to a db. Please see PR #268

[nkabir]

Fantastic. I'll give it a try!

[odeke-em]

Addressed by PR #268. Thank you @vrusinov and @nkabir. Please get the latest code.