Fix Clay__HashData SIMD lane-symmetric hash collisions

[y2kiah]

Summary

Clay__HashData can produce colliding (and in the worst case, uniformly zero) hashes for legitimate, differently-sized inputs. This causes Clay's measurement cache to serve stale/wrong measurements for unrelated text, which leads to wrong text element layout.

Reproduction

(SIMD/x86_64): feed any repeating pattern of 8 bytes like 0123456701234567 into the hash function. Clay__HashData returns 0 as the hash for the different-length strings, causing the text-measurement cache to hand back a stale, wrong-width measurement.

Root cause

Both SIMD paths (x86 SSE, ARM NEON) mix input in two 64-bit lanes using identical per-lane add-rotate-xor ops, with no cross-lane mixing, combining lanes only at the end via result[0] ^ result[1]. Any input whose 16-byte chunks have equal low/high 8-byte halves — true for any 8-byte-periodic repeating pattern — keeps both lanes bit-identical throughout, so the final XOR is always exactly 0, regardless of pattern length or repeat count.

The scalar fallback uses an unrelated single-accumulator loop with no lane structure, doesn't reproduce this collision, and is left unchanged.

Fix

Both SIMD paths (x86 and ARM NEON) now mix in the original input length as one additional block immediately before finalization, split asymmetrically across the two lanes ({length, ~length}).

Verification

Two standalone C programs were run to verify, with the following output.

======================================================================
clay_hash_test.c -- native SIMD path
======================================================================
=== Build mode: x86_64 SSE SIMD ===
len    Clay__HashData (built)   NeonSim (scalar port)   
16     11230645361934756109     8182459649240422126     
32     8766597023212744882      5277923167795733852     
48     8963489681130709830      3220050529044920501     
64     3360829255815755830      5298420970737688413     
96     4543706634573728051      5282079190206392173     
128    10583999745537992412     7238826314991203593     
160    14688236849559211743     16870611772397850935    
176    5980081241396710808      5186844497610871812     
191    10485997848091019275     12883912852546777573    
192    9333076233928157203      12883912856796725285    
193    16597335431922017344     16597335431922017344    

Collisions within the built Clay__HashData (matches len=192):
  none

Collisions within NeonSim (matches len=192):
  none
======================================================================
clay_hash_test.c -- scalar fallback (-DCLAY_DISABLE_SIMD)
======================================================================
=== Build mode: SCALAR FALLBACK (CLAY_DISABLE_SIMD) ===
len    Clay__HashData (built)   NeonSim (scalar port)   
16     17723556200358001352     8182459649240422126     
32     17930966903814737913     5277923167795733852     
48     8052987633300233099      3220050529044920501     
64     14922823345160574309     5298420970737688413     
96     16016956546453714778     5282079190206392173     
128    10345154571493098764     7238826314991203593     
160    18346089376547262574     16870611772397850935    
176    8155023632474834976      5186844497610871812     
191    4342859391124470245      12883912852546777573    
192    5855613563044138420      12883912856796725285    
193    6914003496482745747      16597335431922017344    

Collisions within the built Clay__HashData (matches len=192):
  none

Collisions within NeonSim (matches len=192):
  none
======================================================================
clay_hash_fix_test.c -- candidate-fix comparison + distribution check
======================================================================
========== SIMD lane-pair path (x86 SSE / ARM NEON shape) ==========

--- baseline (no fix) -- reproduces the bug ---
  COLLISION: len=16 and len=32 both hash to 0
  COLLISION: len=16 and len=48 both hash to 0
  COLLISION: len=16 and len=64 both hash to 0
  COLLISION: len=16 and len=96 both hash to 0
  COLLISION: len=16 and len=128 both hash to 0
  COLLISION: len=16 and len=160 both hash to 0
  COLLISION: len=16 and len=176 both hash to 0
  COLLISION: len=16 and len=192 both hash to 0
  COLLISION: len=32 and len=48 both hash to 0
  COLLISION: len=32 and len=64 both hash to 0
  COLLISION: len=32 and len=96 both hash to 0
  COLLISION: len=32 and len=128 both hash to 0
  COLLISION: len=32 and len=160 both hash to 0
  COLLISION: len=32 and len=176 both hash to 0
  COLLISION: len=32 and len=192 both hash to 0
  COLLISION: len=48 and len=64 both hash to 0
  COLLISION: len=48 and len=96 both hash to 0
  COLLISION: len=48 and len=128 both hash to 0
  COLLISION: len=48 and len=160 both hash to 0
  COLLISION: len=48 and len=176 both hash to 0
  COLLISION: len=48 and len=192 both hash to 0
  COLLISION: len=64 and len=96 both hash to 0
  COLLISION: len=64 and len=128 both hash to 0
  COLLISION: len=64 and len=160 both hash to 0
  COLLISION: len=64 and len=176 both hash to 0
  COLLISION: len=64 and len=192 both hash to 0
  COLLISION: len=96 and len=128 both hash to 0
  COLLISION: len=96 and len=160 both hash to 0
  COLLISION: len=96 and len=176 both hash to 0
  COLLISION: len=96 and len=192 both hash to 0
  COLLISION: len=128 and len=160 both hash to 0
  COLLISION: len=128 and len=176 both hash to 0
  COLLISION: len=128 and len=192 both hash to 0
  COLLISION: len=160 and len=176 both hash to 0
  COLLISION: len=160 and len=192 both hash to 0
  COLLISION: len=176 and len=192 both hash to 0
  len=192 hash: 0

--- asymmetric fix: append {length, ~length} ---
  no collisions among lengths 16..192
  len=192 hash: 9333076233928157203

--- distribution: baseline (n=20000 random strings, len 1..250) ---
  exact-hash collisions among samples: 19
  bucket occupancy: min=18 max=160 expected_mean=39.06 variance=146.69

--- distribution: asymmetric fix (n=20000 random strings, len 1..250) ---
  exact-hash collisions among samples: 19
  bucket occupancy: min=20 max=78 expected_mean=39.06 variance=54.84

Caveat

I don't have ARM hardware to run the real compiled NEON path; the fix there was validated against a scalar simulation run in the test files.

[NogginBops]

The fix for this issue is actually much simpler than this. Detailed in #468, the solution is to make the initial values different for the left and right lanes.

In my own code I used the first 8 constants from the BLAKE hash function instead of just the first four:

// Pinched these constants from the BLAKE implementation
__m128i v0 = _mm_set_epi64x(0x6a09e667f3bcc908ULL, 0xbb67ae8584caa73bULL);
__m128i v1 = _mm_set_epi64x(0x3c6ef372fe94f82bULL, 0xa54ff53a5f1d36f1ULL);
__m128i v2 = _mm_set_epi64x(0x510e527fade682d1ULL, 0x9b05688c2b3e6c1fULL);
__m128i v3 = _mm_set_epi64x(0x1f83d9abfb41bd6bULL, 0x5be0cd19137e2179ULL);

See for the values: https://en.wikipedia.org/wiki/BLAKE_(hash_function)#Initialization_vector

[y2kiah]

That is a simpler solution. Closing this PR, and will look for the custom hash function pointer to eventually land.