Skip to content

[vector][python] Read vindex vector indexes#8293

Open
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/vector-vindex-python-read
Open

[vector][python] Read vindex vector indexes#8293
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/vector-vindex-python-read

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

This PR flattens paimon-vector into a single jar module backed by paimon-vector-index-java 0.1.0, removing the old paimon-vector-jni submodule. It also adds Python read support for vindex vector indexes through paimon-vindex 0.1.0 and covers Java-write/Python-read interop.

Changes

  • Convert paimon-vector from a parent module with paimon-vector-index and paimon-vector-jni children into one module that depends on paimon-vector-index-java 0.1.0.
  • Move the vector global index factories/readers/writers into the top-level paimon-vector source tree and load the native library from the released Java artifact.
  • Add a Python paimon-vindex vector global index reader and wire it into vector search reader dispatch for ivf-flat, ivf-pq, ivf-hnsw-flat, and ivf-hnsw-sq.
  • Add optional Python packaging/dev dependency entries for paimon-vindex==0.1.0.
  • Add Java/Python mixed tests: Java writes a vindex-backed vector index and Python reads/searches it; run_mixed_tests.sh now runs that path on Python >= 3.9.

Testing

  • python -m py_compile paimon-python/pypaimon/table/source/vector_search_read.py paimon-python/pypaimon/globalindex/vindex/vindex_vector_global_index_reader.py paimon-python/pypaimon/tests/e2e/java_py_read_write_test.py
  • bash -n paimon-python/dev/run_mixed_tests.sh
  • git diff --check
  • mvn -pl paimon-vector -am -Pfast-build -DfailIfNoTests=false -Dtest=NativeVectorGlobalIndexerFactoryTest,SeekableStreamVectorIndexInputTest,NativeVectorGlobalIndexTest,org.apache.paimon.JavaPyE2ETest#testVindexVectorIndexWrite test -Drun.e2e.tests=true
  • PYTHONPATH=/tmp/paimon-vindex-site:${PYTHONPATH:-} python -m pytest java_py_read_write_test.py::JavaPyReadWriteTest::test_read_vindex_vector_index -v
  • mvn -pl paimon-vector -am -DskipTests compile

Notes

paimon-vector has not been released yet, so this intentionally does not preserve compatibility with the old two-submodule layout.

@JingsongLi JingsongLi force-pushed the codex/vector-vindex-python-read branch from a77efc3 to 6c29f19 Compare June 19, 2026 14:26
@JingsongLi JingsongLi force-pushed the codex/vector-vindex-python-read branch from 6c29f19 to d441944 Compare June 19, 2026 14:33
@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants