vector_io providers do not calculate scores correctly when cosine distance being used

[jeremychoi]

System Info

N/A (I suspect the issue occurs regardless of the environment)

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

When using pgvector with vector_io as an example, the value of results.score seems to be calculated incorrectly in the following code (https://github.com/llamastack/llama-stack/blob/main/llama_stack/providers/remote/vector_io/pgvector/pgvector.py#L136
) if dist represents a cosine distance rather than a Euclidean distance:

score = 1.0 / float(dist) if dist != 0 else float("inf")

As a result, the score can be greater than 1, which may cause thresholds to behave inaccurately.

For example, here’s a reproducer:

results = client.vector_io.query(
                vector_db_id=vector_db_id,
                query=query_text,
                params=query_params
            )
... snipped 
            for i, chunk in enumerate(results.chunks):
                score = results.scores[i]
                logging.info(f"Score: {score}")

This produces the following log:

INFO:root:Score: 1.3162429427090052

This issue may also affect other vector_io providers that use cosine distance.

Error logs

INFO:root:Score: 1.3162429427090052

Expected behavior

The score should be calculated as 1 - cosine_distance instead in case of cosine_distnace, and ranged between 0 to 1 properly if necessary with normalization.

[franciscojavierarceo]

will investigate thanks @jeremychoi!

[ChristianZaccaria]

/assign @ChristianZaccaria . If it's okay with @franciscojavierarceo :)

[r3v5]

System Info

N/A (I suspect the issue occurs regardless of the environment)

Information

• The official example scripts[x] My own modified scripts

🐛 Describe the bug

When using pgvector with vector_io as an example, the value of results.score seems to be calculated incorrectly in the following code (https://github.com/llamastack/llama-stack/blob/main/llama_stack/providers/remote/vector_io/pgvector/pgvector.py#L136 ) if dist represents a cosine distance rather than a Euclidean distance:

score = 1.0 / float(dist) if dist != 0 else float("inf")

As a result, the score can be greater than 1, which may cause thresholds to behave inaccurately.

For example, here’s a reproducer:

results = client.vector_io.query(
                vector_db_id=vector_db_id,
                query=query_text,
                params=query_params
            )
... snipped 
            for i, chunk in enumerate(results.chunks):
                score = results.scores[i]
                logging.info(f"Score: {score}")

This produces the following log:

INFO:root:Score: 1.3162429427090052

This issue may also affect other vector_io providers that use cosine distance.

Error logs

INFO:root:Score: 1.3162429427090052

Expected behavior

The score should be calculated as 1 - cosine_distance instead in case of cosine_distnace, and ranged between 0 to 1 properly if necessary with normalization.

I have researched and found out that if we want to create normalized scores from cosine similarity that will be on scale [0,1] then we need to compute Angular similarity that lives in this scale. This will be the correct way to handle that I suppose. What do you think guys?
Another reference https://math.stackexchange.com/questions/3241174/scale-cosine-similarity-between-vectors-to-range-0-1

[r3v5]

There are actually two additional approaches how to get normalized scores without computing Angular similarity:

1. score = (1 + cosine_similarity) / 2

Code example:

import numpy as np
from numpy.linalg import norm

v1 = np.array([1, 0, 0])
v2 = np.array([0, 1, 0])

cos_sim = np.dot(v1, v2) / (norm(v1) * norm(v2))
score = (1 + cos_sim) / 2
print("Cosine Similarity:", cos_sim)
print("Normalized Score:", score)

2. cosine_distance = 1 - cosine_similarity
   score = 1 / (1 + cosine_distance)

Code example:

import numpy as np
from numpy.linalg import norm

v1 = np.array([1, 0, 0])
v2 = np.array([0, 1, 0])

cos_sim = np.dot(v1, v2) / (norm(v1) * norm(v2))
d = 1 - cos_sim
score = 1 / (1 + d)
print("Cosine Similarity:", cos_sim)
print("Normalized Score:", score)

See reference

[ChristianZaccaria]

@r3v5 correct, your second comment is the approach we're taking; it uses both approaches to get normalized scores. The PR contains more details: #3375

[r3v5]

@r3v5 correct, your second comment is the approach we're taking; it uses both approaches to get normalized scores. The PR contains more details: #3375

Yes, I am in the process of reviewing it. Will publish reviews soon! Thanks for your PR btw!

[ehhuang]

Sorry for jumping in late. I don't think vector_io should dictate how scores are normalized/computed as different embedding models can use different similarity calculations. Instead we should have configurable options.

[franciscojavierarceo]

Sorry for jumping in late. I don't think vector_io should dictate how scores are normalized/computed as different embedding models can use different similarity calculations. Instead we should have configurable options.

+1

[ChristianZaccaria]

Hi @ehhuang, thanks for the valuable input. I think I understand the concern. Let me know if this would make sense or if I'm missing anything:

1. Add centralized score normalization: create a new ScoreNormalizer class to ensure consistency, i.e., [cosine distance -> similarity], [dot product -> similarity], [L2 distance -> similarity], etc.
2. Add configurable distance_metric to VectorDB class, making this configurable at vector DB registration. Then, when scores are returned, the ScoreNormalizer applies the normalization strategy depending on the configured metric defined by the user.

This ensures the user can specify the metric of choice, and scores are normalized based on the specified metric.

[ehhuang]

I think we should focus on vector_stores APIs as vector_io/vector_db are to be deprecated.

On POST vector_stores, we can add the distance_metric as you suggested.

I'm not sure I understand the need for normalization. And how would you normalize dot product or L2 for instance?

[ChristianZaccaria]

I think we should focus on vector_stores APIs as vector_io/vector_db are to be deprecated.
On POST vector_stores, we can add the distance_metric as you suggested.

This makes sense to me 👍

I'm not sure I understand the need for normalization.

I don't think normalization is strictly needed, but it's a way to standardize all scores across all vector providers. This way, interpretation of scores remain consistent i.e., a score of 1 = best score/similarity, while a 0 = lowest score.

Without normalization there could be confusion for users where i.e., cosine distance [0,2] 0 is the best score, while 2 is the lowest score. Also other providers don't use costine distance but rather cosine similarity ranging from [-1,1]. - It's a way to remove ambiguity is my understanding.

And how would you normalize dot product or L2 for instance?

If we take for example faiss, they use dot product as the similarity metric. If we normalize the vector embeddings beforehand, the inner product becomes cosine similarity. Reference.
For L2 I'm not sure yet, but we could start with cosine as being the preferred and most used metric type. wdyt?

[ChristianZaccaria]

@ehhuang, just a friendly ping. What do you think of the above? TLDR:

1. Focus on vector_stores as vector_io/db are to be deprecated.
2. Add on POST vector_stores the distance_metric.
3. Normalization not strictly needed, but it's great for standardizing the scores and remove ambiguity on the meaning of the score.