tests: tighten three thresholds with empirical calibration

Each new bound was probed against the actual test landing zone before
tightening (one regression-test cycle per change), per the calibration
discipline. Two findings the agent flagged were verified unfounded
and left as-is.

tests/adsampling_regression.rs::adsampling_l2_recall_matches_hnsw
  Replaces the HNSW-as-oracle comparison with brute-force ground truth
  in doc_id space. The original input array is still in scope and was
  added with doc_ids = 0..n in insertion order, so vectors[i*dim..]
  maps directly to doc_id i. This avoids the previous "we don't have
  the internal->doc_id mapping" workaround that compared ADSampling
  to HNSW (a correlated oracle: both share the graph, so they tend to
  be wrong together; a graph-traversal regression in the shared path
  passes the assertion). Floor at 0.5 to accommodate HNSW's own
  imperfect recall at these parameters.

tests/correctness_regression.rs::nsw_recall_oracle
  Tightens the recall floor from 0.6 to 0.9. Empirical avg recall@5 at
  the test parameters (n=200, dim=16, m=16, ef_construction=50,
  ef_search=50) is 1.0; the 0.9 floor gives 10pp slack but catches a
  regression that drops recall from ~1.0 to ~0.85 (40pp below the
  prior threshold). Floor was probed via a one-shot example before
  tightening; that probe was discarded.

tests/opq_vs_pq.rs::opq_does_not_degrade_on_uncorrelated_data
  Tightens the OPQ-vs-PQ MAE ratio ceiling from 3.0x to 1.5x.
  Empirically OPQ on this fixture lands at ~0.97x PQ MAE (Procrustes
  finds a near-identity rotation; codebook re-fit is mildly
  beneficial). The previous 3x slack would pass even when a broken
  Procrustes solve returned codes 2-3x worse than PQ -- exactly the
  bug class the test names. Probed and verified to pass at the
  current implementation.

tests/recall_ground_truth.rs::ivf_pq_recall_vs_brute_force
  Floor kept at 0.20. Probed empirical recall is 0.21 at the test
  regime (N=300, DIM=32, num_codebooks=4, codebook_size=16, nprobe=4
  full coverage). 4 codebooks * 16 centroids over 8-d subspaces with
  300 training vectors is a high-quantization-error regime; the floor
  is not paper-grounded but it is calibrated to the empirical landing
  zone. Tightening without expanding the parameters would risk CI
  flakiness on within-seed variance. Comment now states this honestly
  rather than passing the threshold off as a tuned bound.

scann_recall_oracle (correctness_regression.rs:498-): Track F flagged
this for a possible MIPS-vs-cosine metric mismatch. Verified the
IVFAVQIndex::search uses dot product with descending sort
(src/ivf_avq/search.rs:220-224), which matches the test's negated-dot
oracle. No change needed; finding was unfounded on inspection.

Author: arclabs561

tests/adsampling_regression.rs

@@ -236,33 +236,39 @@ mod tests {
         for qi in 0..20 {
             let query = &test_vectors[qi * dim..(qi + 1) * dim];
 
-            // Brute-force ground truth using reordered vectors
-            let raw = index.vectors_raw();
+            // Brute-force ground truth in doc_id space. The original
+            // input is still in scope (line 209) and was added with
+            // doc_ids = 0..n in insertion order, so vectors[i * dim..]
+            // maps directly to doc_id `i`. This avoids the "we don't
+            // have internal->doc_id mapping" issue that previously
+            // forced this test to compare ADS against HNSW (a
+            // correlated oracle: both share the graph, so they tend to
+            // be wrong together).
             let mut gt: Vec<(u32, f32)> = (0..n)
                 .map(|i| {
-                    let v = &raw[i * dim..(i + 1) * dim];
+                    let v = &vectors[i * dim..(i + 1) * dim];
                     (i as u32, vicinity::distance::l2_distance(query, v))
                 })
                 .collect();
             gt.sort_by(|a, b| a.1.total_cmp(&b.1));
-            // Map internal IDs to doc_ids for ground truth comparison
-            // (search returns doc_ids, brute force uses internal IDs)
-            // Since we don't have the mapping externally, compare via HNSW search
-            let hnsw_results = index.search(query, k, ef).unwrap();
-            let ads_results = state.search_hnsw(&index, query, k, ef).unwrap();
+            let gt_top: HashSet<u32> = gt.iter().take(k).map(|(id, _)| *id).collect();
 
-            let hnsw_ids: HashSet<u32> = hnsw_results.iter().map(|r| r.0).collect();
+            let ads_results = state.search_hnsw(&index, query, k, ef).unwrap();
             let ads_ids: HashSet<u32> = ads_results.iter().map(|r| r.0).collect();
 
-            // Compare ADSampling against HNSW (not brute force -- both use the same graph)
-            let overlap = hnsw_ids.intersection(&ads_ids).count();
+            let overlap = gt_top.intersection(&ads_ids).count();
             total_ads_recall += overlap as f64 / k as f64;
         }
 
         let avg_parity = total_ads_recall / 20.0;
+        // ADSampling on L2-built HNSW should approximately track the
+        // brute-force oracle. The 0.5 floor accommodates the fact that
+        // HNSW itself has imperfect recall at these parameters; an
+        // ADSampling regression that returned random ids would be far
+        // below this floor.
         assert!(
-            avg_parity > 0.7,
-            "ADSampling should find >=70% of HNSW's results on L2, got {:.1}%",
+            avg_parity > 0.5,
+            "ADSampling recall@k vs brute force on L2 = {:.1}%, expected > 50%",
             avg_parity * 100.0
         );
     }

tests/correctness_regression.rs

@@ -276,9 +276,15 @@ mod nsw_tests {
         }
 
         let avg_recall = total_recall / num_queries as f32;
+        // Empirical baseline at these parameters (m=16, ef_construction=50,
+        // ef_search=50, n=200, dim=16) is recall@5 ~ 1.0. Floor at 0.9 gives
+        // 10pp slack for legit construction-time noise but catches a real
+        // regression that drops from ~1.0 to ~0.85. The previous 0.6 floor
+        // would pass at recall=0.61, masking a 40pp construction-quality
+        // collapse.
         assert!(
-            avg_recall >= 0.6,
-            "NSW recall={avg_recall:.3} below 0.6 (construction quality regression)"
+            avg_recall >= 0.9,
+            "NSW recall={avg_recall:.3} below 0.9 (construction quality regression)"
         );
     }
 

tests/opq_vs_pq.rs

@@ -226,8 +226,16 @@ fn opq_does_not_degrade_on_uncorrelated_data() {
     eprintln!("  PQ  distance MAE: {pq_mae:.6}");
     eprintln!("  OPQ distance MAE: {opq_mae:.6}");
 
+    // Empirically OPQ on this fixture lands at ~0.97x PQ MAE (slightly
+    // better than PQ even on uncorrelated data, because the Procrustes
+    // step finds a near-identity rotation and the codebook re-fit is
+    // mildly beneficial). A 1.5x ceiling preserves headroom for legit
+    // optimization variance while catching a real OPQ regression that
+    // makes it materially worse than vanilla PQ. The previous 3x slack
+    // would pass even when a broken Procrustes solve returned codes
+    // 2-3x worse than PQ -- exactly the bug class the test names.
     assert!(
-        opq_mae <= pq_mae * 3.0 + 0.01,
-        "OPQ MAE ({opq_mae:.6}) is more than 3x worse than PQ MAE ({pq_mae:.6}) on uncorrelated data"
+        opq_mae <= pq_mae * 1.5 + 0.01,
+        "OPQ MAE ({opq_mae:.6}) is more than 1.5x worse than PQ MAE ({pq_mae:.6}) on uncorrelated data"
     );
 }

tests/recall_ground_truth.rs

@@ -336,7 +336,16 @@ fn ivf_pq_recall_vs_brute_force() {
     idx.build().unwrap();
 
     let recall = avg_recall(&vectors, &qs, |q| idx.search(q, K).unwrap());
-    // IVF-PQ at low dimensions with few clusters has limited recall
+    // IVF-PQ at this regime (N=300, DIM=32, num_codebooks=4,
+    // codebook_size=16, nprobe=4=num_clusters) lands at ~0.21 recall
+    // empirically: 4 codebooks * 16 centroids over 8-d subspaces with
+    // 300 training vectors is a high-quantization-error regime, even
+    // with full-cluster coverage. The 0.20 floor is calibrated to the
+    // empirical landing zone, not paper-grounded; tightening it
+    // without expanding the test parameters (more codebooks / larger
+    // codebook_size) risks CI flakiness on within-seed variance.
+    // A regression that drops recall below 0.20 indicates a real
+    // codebook-corruption or cluster-routing bug, not noise.
     assert!(
         recall >= 0.20,
         "IVF-PQ avg recall@{K} = {recall:.3}, expected >= 0.20"