Adding a library to support for reading from/writing to memory addresses

[robinlee09201]

Request

Currently, CLP/CLG only supports writing to/reading from files under a specific directory structure in a POSIX file system. However, some FSless applications may wish to store/read data to/from a memory address. A fuse layer may achieve this, but it may cause huge overhead. It would be great to use CLP/CLG as a library providing memory interfaces.

Possible implementation

Providing a C++ class which includes pointers to memory buffers that would otherwise be stored in the archive directory in the current implementation. This class also allows users to call compress, decompress, and search directly instead of going through the command line.

[kirkrodrigues]

The motivation makes sense. The current archive format has several files with different write patterns:

• logtype.dict, var.dict, logtype.segindex, var.segindex
  • Write pattern: Mostly sequential but has a header that requires seeking
  • Read pattern: Sequential if the archive has been closed for writing, mostly sequential otherwise
• s/<id>
  • Write pattern: Sequential, chunk by chunk, where each chunk is a column (timestamps, logtype IDs, or variables)
  • Read pattern: Sequential, chunk by chunk
• metadata
  • Write pattern: Random
  • Read pattern: Sequential
• metadata.db
  • Write pattern: Random (sqlite-implementation dependent)
  • Read pattern: Random (sqlite-implementation dependent)

Some questions:

Compression

1. How would the proposed class handle the write workloads given that the buffers need to grow over time? I imagine we want to avoid static allocations, but dynamic allocations are non-trivial if we need to grow + copy.
2. To support writing (and reading) metadata.db in-memory, we'd need to implement a SQLITE VFS interface, right?

Search

1. Could you describe the interface changes you'd need to make? I can imagine you need to change the reader::Archive interface to take the new class as another input type. How about the internals of reader::Archive and reader::File?