Staging data ingestion and query optimizations #452
Description
- Defer the
memtable.pushflow outside of the write path. For example make the memtable push async / use a separate thread. - The
disk.pushshould be madeaysnc, so server doesn't wait for the event to be written always. Also discussed in Mitigating blocking IO in highly batched payloads #441 - Concat of events (in the memtable) happens at 16384 events in memory. We need to either get rid of concat completely or move it out of ingest / write flow.