In RabitQuantizationStorage::dist_calculator, q_factor is computed as:
let q_factor = match self.distance_type {
DistanceType::L2 => dist_q_c,
DistanceType::Cosine | DistanceType::Dot => dist_q_c - 1.0,
_ => unimplemented!(...),
};
refer to https://github.com/lance-format/lance/blob/5681653e4462351cd3c92726c7654be89ca0332d/rust/lance-index/src/vector/bq/storage.rs#L427C9-L434C11
dist_q_c is the distance between the query vector and the partition centroid.
However, Cosine distance has already been normalized to L2 distance earlier:
refer to https://github.com/lance-format/lance/blob/5681653e4462351cd3c92726c7654be89ca0332d/rust/lance/src/io/exec/knn.rs#L507C1-L510C23
So for DistanceType::Cosine, q_factor should be dist_q_c (same as L2), not dist_q_c - 1.0.
Suggested fix:
let q_factor = match self.distance_type {
DistanceType::L2 | DistanceType::Cosine => dist_q_c,
DistanceType::Dot => dist_q_c - 1.0,
_ => unimplemented!(...),
};