refactor!: move storage module into logstore

[roeap]

Description

TLDR;

• storage -> logstore::storage
• separate runtime mod for io runtime
• code movement

Another pre-factor PR, this time we move the storage module into logstore.

While working on adding a caching layer for log reads I realised that we are exposing quite a bit of functionality via the storage module and interactions with the logstore module require quite a bit of code parsing. With kernel coming in we will have to deal with yet another storage abstraction.

The good news, in practice we actually have a quite narrow waist when creating the tables, so we can just hook into the approach we take for wrapping the io runtime. Still I believe, that logically the ObjectStore sits behind teh log store, or at least the log store is responsible for providing it.

In an immediate follow up I would like to add the cached store (@rtyler - in case you have some time to run bench marks? 😄) and with that make do a pass over our config for storage to make sure it stays manageable as we add yet another set of options.

[codecov]

Codecov Report

Attention: Patch coverage is 42.20624% with 241 lines in your changes missing coverage. Please review.

Project coverage is 71.99%. Comparing base (50e6c0a) to head (03c0e37).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/logstore/storage/runtime.rs	10.14%	186 Missing ⚠️
crates/core/src/logstore/storage/mod.rs	77.60%	23 Missing and 5 partials ⚠️
crates/core/src/logstore/storage/retry_ext.rs	72.22%	7 Missing and 8 partials ⚠️
crates/core/src/logstore/storage/utils.rs	30.00%	7 Missing ⚠️
crates/core/src/logstore/mod.rs	78.57%	3 Missing ⚠️
crates/core/src/operations/merge/filter.rs	50.00%	0 Missing and 1 partial ⚠️
python/src/utils.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3382      +/-   ##
==========================================
+ Coverage   71.82%   71.99%   +0.17%     
==========================================
  Files         145      145              
  Lines       45972    45774     -198     
  Branches    45972    45774     -198     
==========================================
- Hits        33018    32957      -61     
+ Misses      10859    10743     -116     
+ Partials     2095     2074      -21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ion-elgreco]

Just an fyi, the IO runtime wrapping will hopefully replaced by a kind of Spawservice, the initial PR of Tustvold became stale but I'll try to push it forward from there with Andrew, apache/arrow-rs#7253 (comment)

[ion-elgreco]

Lgtm! :) Im curious to see your caching layer and on what abstraction level it will be because there are some quite through implementations out there at least on an ObjectStore level

[roeap]

Just an fyi, the IO runtime wrapping will hopefully replaced by a kind of Spawservice,

Awesome, looking forward to that :)

there are some quite through implementations out there at least on an ObjectStore level

@ion-elgreco - do you have some links maybe? 😄. The main idea is though to keep it quite simple on that layer - our most effective caching is how we managed the parsed actions :).

Another optimisation, that is not part of this (but I'm hoping to get to one day) is to cache parquet footers - this is something we can implement on the DF level ... That said, the PR should be coming up shortly, and we can continue there :).

[ion-elgreco]

@roeap the overview is here apache/arrow-rs-object-store#14

Caches results of requests locally using memory / disk (see ObjectStoreMemCache in influxdb3_core), and this one in slatedb @criccomini (thanks @ion-elgreco for the pointer)