Skip to content

bug(rust): wrong q_factor for Cosine in RabitQuantizationStorage::dist_calculator #5649

@Ke-Wng

Description

@Ke-Wng

In RabitQuantizationStorage::dist_calculator, q_factor is computed as:

let q_factor = match self.distance_type {
    DistanceType::L2 => dist_q_c,
    DistanceType::Cosine | DistanceType::Dot => dist_q_c - 1.0,
    _ => unimplemented!(...),
};

refer to https://github.com/lance-format/lance/blob/5681653e4462351cd3c92726c7654be89ca0332d/rust/lance-index/src/vector/bq/storage.rs#L427C9-L434C11

dist_q_c is the distance between the query vector and the partition centroid.

However, Cosine distance has already been normalized to L2 distance earlier:

refer to https://github.com/lance-format/lance/blob/5681653e4462351cd3c92726c7654be89ca0332d/rust/lance/src/io/exec/knn.rs#L507C1-L510C23

So for DistanceType::Cosine, q_factor should be dist_q_c (same as L2), not dist_q_c - 1.0.

Suggested fix:

let q_factor = match self.distance_type {
    DistanceType::L2 | DistanceType::Cosine => dist_q_c,
    DistanceType::Dot => dist_q_c - 1.0,
    _ => unimplemented!(...),
};

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions