Question about how BM25 embed function works

[anima-kit]

Hi all! I just created a tutorial for the "full-text" (BM25) search using Milvus. There's a concept that I'm unsure of and I'd like to gain clarity:

Let's say I'm following along with setting up a full-text search as in https://milvus.io/docs/full-text-search.md. I understand how the BM25 score works, but I want to make sure I understand how the BM25 embed function turns text into sparse vectors. From following the full-text search and sparse vectors (https://milvus.io/docs/sparse_vector.md) tutorials, the following is my understanding. Is it correct? Are there nuances that I'm missing?

Milvus does a lot of preprocessing to our documents behind the scenes, including tokenization and stop word removal. From this preprocessing, it obtains the set of all tokens within all documents. The sparse vector for each document will then have some number of dimensions equal to the number of tokens present in the document. Each dimension will be made up of an index-value pair corresponding to the token index in the set of all tokens and the value for the token. This value will depend on the BM25 score for that token and document (is it exactly the score or some other dependence?).

This seems to make sense and I can see why these vectors will be "sparse". Each document probably only contains a handful of the total set of tokens in all documents, while the tokens that are ubiquitous in every document will give scores close to zero.

If this is true, what happens when another data entry is inserted into the collection, are the sparse vectors for each document updated?

[xiaofan-luan]

Sparse vectors are mainly composed of two parts: TF (term frequency) and IDF (inverse document frequency).

When a document is ingested, it is first tokenized and preprocessed. From this, we extract the TF values, which form the sparse vector representation.

The IDF values depend on global corpus statistics and are updated in real time.

During search:

The query is tokenized, and its TF values are computed.

These are combined with the current IDF values.

The resulting query vector is then compared against the stored document term frequencies to calculate similarity (e.g., using cosine similarity or dot product).

This way, TF captures the importance of terms within a single document, while IDF adjusts for how common or rare those terms are across the entire corpus, yielding a balanced and effective sparse representation for retrieval.

[TueVNguyen]

1. Yes, my corpus focuses on English (we have already filtered it using greedy methods). As you mentioned, it’s likely a pretraining corpus for LLMs, so it may contain many rare words and unusual patterns. I think I can pre-normalize the text corpus first, then index it with BM25.

2. My CPUs have 256 cores, with 2TB of RAM and 36TB of disk space.

3. No, I always import first, then create the index, and finally load it.

4. Thanks for your support. I totally agree with the current implementation of Milvus; I just don’t understand why it is so slow.

5. I'm switching to other methods (i.e I compute the bm25 for all of my documents, and putting it as sparse vector and using IP metric) and It worked (only need 2h to import all corpus).

[xiaofan-luan]

@TueVNguyen

Could you reproduce this issue and generate a pprof or perf flame graph?
There are might be some performance issue or large number of tokens, we need to understand performance bottleneck

[xiaofan-luan]

and you said after some data the cluster is crashed. is there a fatal message for that?

[TueVNguyen]

@TueVNguyen

Could you reproduce this issue and generate a pprof or perf flame graph? There are might be some performance issue or large number of tokens, we need to understand performance bottleneck

sorry I can't, it take a lot of time in my side.
But you can try with these setup:
Split 512 tokens/ chunk for each document using jinaai/jina-embeddings-v3 for 3 corpus:

1. Wikipedia-2023
2. Fineweb-edu-subset-2025
3. S2ORC corpus: filter subset >= 2015.

[xiaofan-luan]

why this is related to BM25?
seems that you are using sparse embedding, not Bm25

[zhengbuqian]

https://milvusio.medium.com/full-text-search-in-milvus-whats-under-the-hood-9058016ea84e

here is a more detailed explanation of why we don't need to update the sparse vector in the db even after we have inserted tons of new documents(possibly with complete different term distribution) @anima-kit

[anima-kit]

Thanks for pointing to this, the step-by-step example confirms a lot of the ideas I had about how BM25 score and sparse vectors are calculated. I also see now why IP is used for vector comparison. Great article!

[yanliang567]

@zhuwenxing please keep an eye on the performance issue