Add way to tokenize and segment words on outside of database

[zhcn000000]

Problem Statement

Can I use the hf model's tokenizer or other spliter to manually segment words outside or inside the database and insert them as the sparse vectors required for bm25 search? It can improve cjk language and multi language document support and morden language dictionary

Proposed Solution

When building an index, there is an option to input additional sparse vector columns, such as manually segmenting them into sparse vectors externally and then inserting the tsvector column, or like the bm25vector column in the vchord_bm25 extension, or like generating dense vectors externally using an embedding model and then inserting the vector column, without using an internal index dictionary.

-- Example usage (if applicable)

Alternatives Considered

Using zhparser requires the installation of additional extensions and the creation of text search configuration within the database. The dictionary is also not modern enough

[SteveLauC]

Can I use the hf model's tokenizer or other spliter to manually segment words outside or inside the database and insert them as the sparse vectors required for bm25 search?

Currently, no. pg_textsearch only allows you to build index for the text type (and its sibling types)

[zhcn000000]

I think we can add the intger sparse vector column for bm25 search, and based on this, establish a bm25 search index, using a dictionary that has been manually segmented and input externally for bm25 indexing and require input another sparse vector to query for document, If this is done, it can enhance the BM25 retrieval capability for multilingual documents by split words with language models like bert, not just the built-in single-language tokenizer. At the same time, models available on Hugging Face can be used to perform more accurate tokenization, or optimized tokenization for specific domains to generate sparse vectors for BM25 retrieval.