Feature/disk usage optimization

[galadd]

This PR implements delta encoding for beacon state storage, reducing
disk usage by ~80% while significantly improving state I/O performance.

• 80% disk usage reduction (> 90% in reduction from full state to delta state)

• 70% faster state reads and writes (with cached base)

[CLAassistant]

All committers have signed the CLA.

[Tumas]

Please fix linter errors in your code. You can use scripts/ci/clippy.bash script to identify the errors.

[galadd]

@Tumas

[Tumas]

Please rebase your branch onto the latest develop and fix the failing CI build:
https://github.com/galadd/grandine/actions/runs/20126660680/job/57757774694

You can track the build status for your changes on your fork page (status of the latest commit):
https://github.com/galadd/grandine/tree/feature/disk-usage-optimization

[galadd]

@Tumas Everything looks good now

[galadd]

@Tumas changes resolved

[galadd]

Hi, just checking in on this. Happy to address any feedbacks

[Tumas]

This will panic when calculating delta between genesis state and the next state after. Use u64::saturating_sub

[galadd]

Actually, it wraps around. My logic doesn't cater for that when creating the delta (error from me). But u64::saturating_sub can't work because it results to 0. So more like,

let start_idx = (end_idx + P::EpochsPerHistoricalVector::U64 - diff_len - 1) % P::EpochsPerHistoricalVector::U64;

and then use mod_index, as opposed to get in creating delta

[galadd]

applies to every other vector

[Tumas]

This commit seems to have introduced a regressesion. You can find it by adding and running some sanity tests for your code:

   //fork_choice_control/src/storage.rs: 

    #[cfg(feature = "eth2-cache")]
    #[test_case(
        Config::mainnet().into(),
        eth2_cache_utils::mainnet::GENESIS_BEACON_STATE.force(),
        eth2_cache_utils::mainnet::BEACON_BLOCKS_UP_TO_SLOT_128.force()
    )]
    // #[test_case(
    //     Config::mainnet().into(),
    //     eth2_cache_utils::mainnet::ALTAIR_BEACON_STATE.force(),
    //     eth2_cache_utils::mainnet::ALTAIR_BEACON_BLOCKS_FROM_128_SLOTS.force()
    // )]
    // #[test_case(
    //     Config::mainnet().into(),
    //     eth2_cache_utils::mainnet::CAPELLA_BEACON_STATE.force(),
    //     eth2_cache_utils::mainnet::CAPELLA_BEACON_BLOCKS_FROM_244816_SLOTS.force()
    // )]
    fn test_deltas_roundtrip<P: Preset>(
        config: Arc<Config>,
        base_state: &Arc<BeaconState<P>>,
        blocks: &[Arc<SignedBeaconBlock<P>>],
    ) -> Result<()> {
        let pubkey_cache = Arc::new(PubkeyCache::default());

        let mut state = base_state.clone_arc();
        let state = state.make_mut();

        for block in blocks.iter().skip(1) {
            combined::untrusted_state_transition(&config, &pubkey_cache, state, block)?;

            let delta = delta(&base_state, &Arc::new(state.clone()))?;
            let recovered = apply_delta(&base_state, delta)?;

            assert_eq!(*state, recovered);
        }

        Ok(())
    }

To run these tests modify make tests command in the Makefile:

cargo test --release --features stub-grandine-version $(FEATURES) --features eth2-cache $(EXCLUDES) -p fork_choice_control delta -- --nocapture

and clone the https://github.com/grandinetech/eth2-cache to your local Grandine's project root

[galadd]

Ping for review when ready

[galadd]

@Tumas, I was thinking of using a derive macro. This might make it more maintainable during develop phase of a new fork

The way it would work:
Add attribute to encode delta to the main struct

use delta_derive::DeltaEncode;

#[derive(Clone, Debug, Default, Deserialize, Serialize, Ssz, DeltaEncode)] // Added DeltaEncode
#[derivative(PartialEq, Eq)]
#[serde(bound = "", deny_unknown_fields)]
pub struct BeaconState<P: Preset> {

this way, during develop phase, compiler can easily point missing fields. Thoughts?

[sauliusgrigaitis]

Hi, right now we are experimenting with an alternative solution that has potential to reduce the disk usage much more. For example instead of repeatedly storing validator indexes (it's append only list), we can extract those validator indexes to an external storage and never store it in states on disk.

We will get back to your PR after we finish experiments on our side.

BTW, have you tested how much storage space actually your commit saves?

[galadd]

Hi, Regarding the storage savings, I have gathered concrete benchmarks using the project's existing libmdbx storage layer and compression settings. I tested this against Phase Electra states over a 256-epoch window (8 checkpoints).

Benchmark Data (Electra Phase):

• Full State (Compressed): ~136 MB
• Delta State (Compressed): ~11.1 MB

Comparison over 256 Epochs:

1. Old Approach (Full State every 32 epochs):

• Stores 8 full states.
• Total Storage: 8×136 MB≈1,088 MB

2. New Approach (Base + Deltas):

• Stores 1 Base State + 7 Deltas.
• Total Storage: 136 MB+(7×11.1 MB)≈214 MB

This represents an 80% reduction in disk usage for state storage. While extracting the validator registry (as you suggested) would reduce the Base State size, the Delta approach still provides a massive 12x compression for the high-frequency changes (like balances) that occur every 32 epochs.

I believe a hybrid approach, extracting the static validator set and applying my Delta logic to the mutable fields, would yield the maximum possible savings.