Update indexing and quantization docs by jzheng106 · Pull Request #344 · lancedb/docs

jzheng106 · 2026-08-26T18:31:13Z

Substantial changes to indexing documentation, as part of Jason's internship project. Reorganized and rewrote "overview", "vector index", and "quantization" tabs in https://docs.lancedb.com/indexing . Added descriptions of algorithms and improved API references, usage examples, and overall readability/flow. Msg me on slack with questions

wkalt · 2026-08-26T21:58:11Z

-| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
 | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. |
+| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
 | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. |


I'm not sure if changing this was in scope, but I find these descriptions pretty weak. They would be better if they gave some kind of comparison between each other. For example if we are saying these are all useful because they fall at different points on the compression/speed/accuracy plane, we should describe how they actually relate. Why would I choose PQ over RQ, or SQ over PQ, is not really answered by this table.

Another way to do this is something like,

If you need X [petabyte scale low-latency search on static dataset], choose Y because x, y, z
If you need A, choose B because a, b, c

etc. I have no idea what the real values to plug in here are though.

"I think these descriptions are pretty weak" I agree. Unfortunately, until recently many technical descriptions were written similar to this, and I didn't really know how to approach them (I also didn't have a clearly defined scope for this documentation task) so I just largely took my best guess. I agree that there could be further improvements. Maybe I'll make a note of this and add it to a future commit?

I think the Use Case column makes a stab at this doesn't it? I don't know if it reflects reality though. I read it as...

PQ is the baseline.
RQ achieves better compression than PQ with minimal loss to accuracy.
SQ sacrifices compression but can speed up index builds
Flat discards compression entirely in favor of accuracy

I think the use cases don't do it for me because they don't compare all the options on the same plane. How do I compare (fast indexing, consistent value ranges) to (max compression, particular dimension requirements). Assuming I am reading the right column.

wkalt · 2026-08-26T21:58:48Z

@@ -0,0 +1,172 @@
+9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800   1) ---


is this included intentionally?

no. oops. you're right to delete the whole file

westonpace

Thanks for working on this. I think this PR improves things but we can still improve further, especially for a naive user.

We've also removed a lot of information. I think that can be good to focus the guide. I'll try and do a second pass soon to see if we removed anything that might have been important.

westonpace · 2026-08-28T13:00:36Z

-| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
 | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. |
+| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
 | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. |


I think the Use Case column makes a stab at this doesn't it? I don't know if it reflects reality though. I read it as...

PQ is the baseline.
RQ achieves better compression than PQ with minimal loss to accuracy.
SQ sacrifices compression but can speed up index builds
Flat discards compression entirely in favor of accuracy

westonpace · 2026-08-28T13:45:21Z

+**Using `bypass_vector_index`:**

-Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload.
+Use `bypass_vector_index` to compute an exact **kNN** result. Note that exact queries may be prohibitively slow on production scales.


Do we discuss recall@k anywhere anymore?

I think "how do I know if my vector index is properly configured" or "how do I know how well my vector index is performing" is a common question we should be answering on this page. I'm not sure the old page answered it well either but I think we should strive to include that somewhere (and calculating recall@k is part of the answer)

westonpace · 2026-08-28T13:48:08Z

-</Warning>
+| Parameter | Description |
+| :--- | :--- |
+| `m` | The number of neighbors to select for each vector in the HNSW graph. |


Do we have any guidance on this parameter? What is the default?

westonpace

In this pass I did a look for anything we might be missing in this conversion. There's a few things I think it would be nice to keep.

westonpace · 2026-09-10T12:28:36Z

-
-For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows.
-
-By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to


use the same distance metric when creating the index and searching it

This part is important and probably something we should highlight (I don't think the new version says this at all).

If the user creates a vector index with (for example) cosine distance and then they search the vector distance with l2 then we will silently fallback to a full scan (without the index) which is surprising and costly. It's led to complaints from users on more than one occasion.

I'd argue it is a bug (we should fail unless bypass_vector_index is specified), but it is one we seem determined not to fix.

The user must either specify the same distance type that was used to train the index or they must not speciy distance type at all.

Can we add that guidance into the docs somewhere?

westonpace · 2026-09-10T12:29:11Z

-The default distance is `l2` (Euclidean).
-
-<Note>
-You can call `create_index()` with different parameters to create a new index -- this replaces any existing index.


I don't think the new docs explain that you can replace existing indexes

westonpace · 2026-09-10T12:30:44Z

-    <CodeBlock filename="Python" language="Python" icon="python">
-    {VectorIndexNestedField}
-    </CodeBlock>
-</CodeGroup>


Do we still have an example of a nested field somewhere?

Co-authored-by: Weston Pace <weston.pace@gmail.com>

…c to distance_type. Add warning about setting the distance_type correctly. Clarify nprobes default.

jzheng106 requested review from BubbleCal, Xuanwo, justinrmiller, westonpace and wjones127 August 26, 2026 18:31

wkalt reviewed Aug 26, 2026

View reviewed changes

westonpace requested changes Aug 28, 2026

View reviewed changes

mintlify Bot deployed to staging - docs August 28, 2026 17:21 View deployment

mintlify Bot deployed to staging - docs August 28, 2026 21:41 View deployment

mintlify Bot deployed to staging - docs August 28, 2026 21:42 View deployment

mintlify Bot deployed to staging - docs August 28, 2026 21:43 View deployment

mintlify Bot deployed to staging - docs August 28, 2026 21:44 View deployment

mintlify Bot deployed to staging - docs August 28, 2026 21:45 View deployment

mintlify Bot deployed to staging - docs August 31, 2026 19:03 View deployment

mintlify Bot deployed to staging - docs August 31, 2026 20:36 View deployment

mintlify Bot deployed to staging - docs August 31, 2026 21:54 View deployment

jzheng106 requested review from westonpace and wkalt August 31, 2026 21:56

westonpace requested changes Sep 10, 2026

View reviewed changes

westonpace reviewed Sep 10, 2026

View reviewed changes

Comment thread docs/indexing/vector-index.mdx Outdated

mintlify Bot deployed to staging - docs September 10, 2026 16:46 View deployment

mintlify Bot deployed to staging - docs September 10, 2026 16:51 View deployment

jzheng106 requested a review from westonpace September 10, 2026 17:06

mintlify Bot deployed to staging - docs September 10, 2026 17:09 View deployment

jzheng106 added 5 commits September 11, 2026 06:06

light changes

6498b5e

more small changes

2b68810

half done with ivf implementation

5175f21

quantization light pass

452663a

vector index light pass

9b10c49

jzheng106 and others added 17 commits September 11, 2026 06:06

almost done

087616f

Apply batched suggestions from code review

ab21e07

Co-authored-by: Weston Pace <weston.pace@gmail.com>

Apply suggestion from @westonpace

df517b6

Co-authored-by: Weston Pace <weston.pace@gmail.com>

Apply suggestion from @westonpace

97a165d

Co-authored-by: Weston Pace <weston.pace@gmail.com>

Apply suggestion from @westonpace

5a6c68a

Co-authored-by: Weston Pace <weston.pace@gmail.com>

Apply suggestion from @westonpace

d65da25

Co-authored-by: Weston Pace <weston.pace@gmail.com>

Apply suggestion from @westonpace

1370a9c

Co-authored-by: Weston Pace <weston.pace@gmail.com>

a

68d3ffb

a

08e92d2

a

4989406

a

5e5557c

a

89797fe

hnsw stuff

21d3b91

a

a84dd22

fixed snippets/indexing.mdx

3125807

Apply suggestion from @westonpace

46c06b3

Co-authored-by: Weston Pace <weston.pace@gmail.com>

a

61bb452

westonpace force-pushed the update-indexing-docs branch from 9d6f1a4 to 61bb452 Compare September 11, 2026 13:09

mintlify Bot deployed to staging - docs September 11, 2026 13:10 View deployment

westonpace added 3 commits September 11, 2026 06:40

remove snippet source so the snippets are not re-generated

866adbf

Remove entries from index choosing table with no advice. Change metri…

8762626

…c to distance_type. Add warning about setting the distance_type correctly. Clarify nprobes default.

remove all mention of auto tuning

313a1c6

mintlify Bot deployed to staging - docs September 11, 2026 14:16 View deployment

Fix up text for num_partitions / target_partition_size defaults

6e99006

mintlify Bot deployed to staging - docs September 11, 2026 16:02 View deployment

Make target_partition_size default a little more vague

8d560f3

mintlify Bot deployed to staging - docs September 11, 2026 16:55 View deployment

westonpace approved these changes Sep 11, 2026

View reviewed changes

westonpace merged commit db47318 into main Sep 11, 2026
4 checks passed

westonpace deleted the update-indexing-docs branch September 11, 2026 20:02

		@@ -0,0 +1,172 @@
		9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 1) ---


		For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows.

		By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to

Uh oh!

Conversation

jzheng106 commented Aug 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants