Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weaviate supports Chinese bm25 #12223

Open
5 tasks done
dajianguo opened this issue Dec 30, 2024 · 0 comments · May be fixed by #12258
Open
5 tasks done

Weaviate supports Chinese bm25 #12223

dajianguo opened this issue Dec 30, 2024 · 0 comments · May be fixed by #12258
Labels
👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database.

Comments

@dajianguo
Copy link
Contributor

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Weaviate is not supports Chinese bm25 in last version。
I looked up Weaviate's official website and found that Weaviate added support for jieba word segmentation after 1.24. The previous version segmented words based on spaces, which does not work for Chinese. The following is my modification method. If there is no problem, I will submit it,Of course it is configurable in docker-compose.yaml.
1、Upgraded Weaviate version
2、Modify the default word segmenter
image

Reference Links:
https://weaviate.io/developers/weaviate/config-refs/schema#gse-and-trigram-tokenization-methods
https://pkg.go.dev/github.com/go-ego/gse#section-readme

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
@dosubot dosubot bot added the 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database. label Dec 30, 2024
dajianguo added a commit to dajianguo/dify that referenced this issue Dec 31, 2024
dajianguo added a commit to dajianguo/dify that referenced this issue Dec 31, 2024
@dajianguo dajianguo linked a pull request Dec 31, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant