Feature Request: Progress Bars

The lack of feedback during Collection.upsert and Collection.create_index is a bad DX.

It would be great to get some progress bars but I haven't been able to get them working properly in notebooks and shell environments.

If anyone has experience adding them this would be a great community contribution

upsert

vecs/src/vecs/collection.py

Line 108 in 87ed2d3

for chunk in flu(vectors).chunk(chunk_size):

but it can't always assume that vectors is sized. Maybe a runtime check to see if *vectors* has a known length and providing it to the progress bar if known would be best

create_index

vecs/src/vecs/collection.py

Lines 346 to 350 in 87ed2d3

    
           stmt = postgresql.insert(clone_table).from_select( 
        
               self.table.c, select(self.table) 
        
           ) 
        
           stmt = stmt.on_conflict_do_nothing() 
        
           sess.execute(stmt)

this one is a little more involved as it'll have to introduce client side keyset pagination on the id primary key to get feedback to python

@olirice

@olirice I would like to work on it.

can please point me to contributing guideline for this repo I was find it

great!

we don't have official contributing guidelines

Broadly you'll need

python 3.7+ (and be compatible with 3.7)
docker

setup a virtualenv

python -m venv venv
source venv/bin/activate
pip install -e ".[dev,docs,text_embedding]"

install the pre-commit hooks

pre-commit install

at that point you should be able to run

pytest

and have it pass. Note: on the first run a docker container will be spinning up so it could take a couple minutes. After that It'll be much faster.

We have 100% code coverage as a requirement so any new lines/functions must be covered. You can check if what you introduce is covered using

pytest --cov=vecs --cov-report=html -vv

to generate a report, followed by

open htmlcov/index.html

to see the coverage report

all new functions/methods should conform to the current documentation style

let me know if you have any trouble

@olirice

Hi @olirice, I’d like to work on this if it is still relevant.

To keep the first PR focused, I propose adding optional progress reporting to Collection.upsert:

Preserve the existing behavior by default.
Display a known total when the input implements Sized.
Support generators and other unsized iterables without consuming them in advance.
Ensure the progress display works in both terminal and notebook environments.
Add tests for enabled, disabled, sized, and unsized inputs while maintaining 100% coverage.
Update the relevant documentation in the existing style.

Would you be open to an initial PR scoped to Collection.upsert, with create_index handled separately? Also, is there a preferred progress-bar library, or should the implementation avoid adding a required dependency?

Picked this up, scoped to upsert as discussed here: #118. create_index progress left for a follow-up.

@marcosbeta23

@marcosbeta23 sorry, I missed your comment before opening #118. You asked first, and your scope is exactly what I ended up building (opt-in flag on upsert, Sized total or indeterminate bar, tqdm.auto, optional extra, tests for the four cases).

Happy either way: I can close #118 so you take it from your own branch, or you're welcome to build on it, review it, or take it over directly and I'll add you as a co-author. Whatever @olirice prefers.

@srijanarya

Hi @srijanarya, no worries at all! I appreciate the apology, these things happen in open source when issues are active.Since you've already built out the exact scope I proposed, let's definitely team up on it. I'd love to review your PR (#118) and collaborate with you there. Once we ensure everything is good to go, adding me as a co-author sounds like a great plan.

Navigation Menu

Repository navigation

Feature Request: Progress Bars #10

upsert

create_index

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	stmt = postgresql.insert(clone_table).from_select(
	self.table.c, select(self.table)
	)
	stmt = stmt.on_conflict_do_nothing()
	sess.execute(stmt)

Feature Request: Progress Bars #10

Description

upsert

create_index

Activity

sahusiddharth commented on Dec 31, 2023

olirice commented on Jan 2, 2024

marcosbeta23 commented on Sep 3, 2026

srijanarya commented on Sep 5, 2026

srijanarya commented on Sep 5, 2026

marcosbeta23 commented on Sep 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions