how index refresh rate happen in milvus

[vihariazure]

hi team
i have below doubts i checked the documentation still having few confusion

1. lets say i have built the index by 1k vectors. i passed query and got the results. now i have inserted another 100 records

during 100 records insertion, now the whole index will be rebuilt again using 1100 vectors or only this 100vectors are inserted to existing index.

2. while inserting , can we pass the query

3. in the first case, while the query is passed only 1000 record will considered or 1000 + (100 not inserted into index ) is considered

[xiaofan-luan]

Milvus support two type of segments, growing and sealed.
When segment is not sealed/flushed, then you can insert into it.
Once it sealed, segment become immutable. New growing segment will be used to insert data.
when search happened we have to scatter gathering results from segment.
Background compaction happened to merge all the sealed segment to improve searvh

[zhu3359]

@xiaofan-luan will the growing segment be searched？

[xiaofan-luan]

@xiaofan-luan will the growing segment be searched？

Yes, Growing segment is searchable with a temporary index.
It's slower than sealed index but since growing segment is smaller that doesn't really matter

[zhu3359]

@xiaofan-luan how dose milvus growing segment build vector index? In faiss, when build index like ivfflat, we should load all vectors and train the loaded data to produce the index, growing data is not supported; but in milvus when a new vector come, will the growing segment re-train all the vectors?

[xiaofan-luan]

Growing Segment is usually. small(< 128MB).
Right now what happened is we will sample and find centroid when data reaches (10% off the segment ), before 10% it's simply brute force search.
After 10% there will be an IVF index or an ScaNN index and the growing index is actually preserved space for adding more points into the ivf index

[zhu3359]

got it， thx