Erasure coding test vectors

[davxy]

GP update: gavofyork/graypaper#429

[sourabhniyogi]

Thank you for updating GP to match what polkajam is actually doing, and making the test set a bit bigger.

We must care about efficiency of the erasure encoding and decoding.

Can you suggest a way of improving this: (which generates your test
https://github.com/jam-duna/jamtestnet/blob/main/erasurecoding/src/lib.rs#L29-L44
so that its efficient? By "efficient" I mean that I want just one call to encode, not k calls to encode, which you get from this actually simple solution (and is surely how this package is intended to be used):

        let shard_size = d_bytes.len() / C;
        let k = d_bytes.len() / W_E;
        let mut original_shards: Vec<Vec<u8>> = vec![Vec::with_capacity(2 * k); C];
        let mut recovery_shards: Vec<Vec<u8>> = vec![Vec::with_capacity(2 * k); V - C];

        let mut encoder = ReedSolomonEncoder::new(C, V - C, shard_size)?;
        for i in 0..C {
            let shard: Vec<u8> = d_bytes[i * shard_size..(i + 1) * shard_size].to_vec();
            encoder.add_original_shard(&shard)?;
            original_shards[i].extend_from_slice(&shard);
        }
        let encoded = encoder.encode()?;
        for (j, shard) in encoded.recovery_iter().enumerate() {
                recovery_shards[j].extend_from_slice(shard);
        }

But it doesn't get the same result as your test vectors. You mentioned shard sizes but I wasn't sure what to do with that. I'm probably missing something basic: What is it?

[jaymansfield]

JavaJAM passes these. Thanks @davxy.

[davxy]

JavaJAM passes these. Thanks @davxy.

Hey @jaymansfield what lib are you using for erasure coding?

[jaymansfield]

JavaJAM passes these. Thanks @davxy.

Hey @jaymansfield what lib are you using for erasure coding?

reed-solomon-simd 3.0.1

[davxy]

JavaJAM passes these. Thanks @davxy.

Hey @jaymansfield what lib are you using for erasure coding?

reed-solomon-simd 3.0.1

Alright, it seems like everyone is using it. At some point I'll try to implement a simple reference version in Python , since that library is doing some performance optimization voodoo that I haven't really looked into. We want to make sure it's actually compliant with what the GP specifies.