Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Conversation v2 #342

Merged
merged 38 commits into from
Jan 17, 2025
Merged
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
f880eac
fix notebook example
dillonalaird Jan 3, 2025
d848c15
remove old tools
dillonalaird Jan 3, 2025
e73f400
fix edge case for checking sim
dillonalaird Jan 3, 2025
b81ba9e
formatting fix
dillonalaird Jan 3, 2025
82b50a9
fixed docs for video tracking tools
dillonalaird Jan 3, 2025
dcef748
updated embs for fixed docs
dillonalaird Jan 3, 2025
9194369
take in user modified code
dillonalaird Jan 6, 2025
78e088f
update planner tool prompt
dillonalaird Jan 7, 2025
c00ce23
change default fps to 5
dillonalaird Jan 8, 2025
bedd6e0
fix type error
dillonalaird Jan 8, 2025
84f725a
fix doc in example
dillonalaird Jan 8, 2025
2895ea8
do not change default fps
dillonalaird Jan 8, 2025
8e27ab5
change default fps to 5
dillonalaird Jan 8, 2025
5f1b1ac
dont crash if parser fails
dillonalaird Jan 9, 2025
5bc59cd
clean up docs
dillonalaird Jan 9, 2025
addf579
flake8
dillonalaird Jan 10, 2025
e8295e3
fix check load
dillonalaird Jan 10, 2025
dc6f299
update vision agent conversation
dillonalaird Jan 10, 2025
a6ae1db
Merge branch 'main' into conversation-updates
dillonalaird Jan 10, 2025
bb62037
fix o1 for lmm class
dillonalaird Jan 11, 2025
76f8dc5
fixed names
dillonalaird Jan 11, 2025
b7e4940
updated docs
dillonalaird Jan 11, 2025
ae407c7
add image size args for lmm
dillonalaird Jan 13, 2025
7ec43c9
fixed resizing
dillonalaird Jan 16, 2025
684733c
added configs
dillonalaird Jan 16, 2025
facdd1e
add config for agents
dillonalaird Jan 16, 2025
f80bdef
update readme
dillonalaird Jan 16, 2025
f5f50ce
fix tool docs
dillonalaird Jan 16, 2025
2c3faf4
fix tool docs
dillonalaird Jan 16, 2025
6f4f6ef
fix bug with strip calls
dillonalaird Jan 16, 2025
3dd6516
update configs
dillonalaird Jan 17, 2025
e931d46
run multi judge
dillonalaird Jan 17, 2025
d75e6c0
remove write code, only test now
dillonalaird Jan 17, 2025
1d57422
fix prompts
dillonalaird Jan 17, 2025
3a2665c
update index
dillonalaird Jan 17, 2025
fe20346
mypy fixes
dillonalaird Jan 17, 2025
2b63ab4
add config module
dillonalaird Jan 17, 2025
2585624
fix type error
dillonalaird Jan 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix tool docs
dillonalaird committed Jan 16, 2025
commit 2c3faf4433427afa972acb3b60d0df3577c8a9ca
18 changes: 10 additions & 8 deletions vision_agent/.sim_tools/df.csv
Original file line number Diff line number Diff line change
@@ -244,7 +244,8 @@ desc,doc,name
1.0.

Parameters:
prompt (str): The prompt to ground to the image.
prompt (str): The prompt to ground to the image. Use exclusive categories that
do not overlap such as 'person, car' and NOT 'person, athlete'.
image (np.ndarray): The image to ground the prompt to.
fine_tune_id (Optional[str]): If you have a fine-tuned model, you can pass the
fine-tuned model ID here to use it.
@@ -281,7 +282,8 @@ desc,doc,name
is useful for tracking and counting without duplicating counts.

Parameters:
prompt (str): The prompt to ground to the video.
prompt (str): The prompt to ground to the image. Use exclusive categories that
do not overlap such as 'person, car' and NOT 'person, athlete'.
frames (List[np.ndarray]): The list of frames to ground the prompt to.
chunk_length (Optional[int]): The number of frames to re-run florence2 to find
new objects.
@@ -317,14 +319,14 @@ desc,doc,name
]
",florence2_sam2_video_tracking
"'florence2_object_detection' is a tool that can detect multiple objects given a text prompt which can be object names or caption. You can optionally separate the object names in the text with commas. It returns a list of bounding boxes with normalized coordinates, label names and associated confidence scores of 1.0.","florence2_object_detection(prompt: str, image: numpy.ndarray, fine_tune_id: Optional[str] = None) -> List[Dict[str, Any]]:
'florence2_object_detection' is a tool that can detect multiple
objects given a text prompt which can be object names or caption. You
can optionally separate the object names in the text with commas. It returns a list
of bounding boxes with normalized coordinates, label names and associated
confidence scores of 1.0.
'florence2_object_detection' is a tool that can detect multiple objects given a
text prompt which can be object names or caption. You can optionally separate the
object names in the text with commas. It returns a list of bounding boxes with
normalized coordinates, label names and associated confidence scores of 1.0.

Parameters:
prompt (str): The prompt to ground to the image.
prompt (str): The prompt to ground to the image. Use exclusive categories that
do not overlap such as 'person, car' and NOT 'person, athlete'.
image (np.ndarray): The image to used to detect objects
fine_tune_id (Optional[str]): If you have a fine-tuned model, you can pass the
fine-tuned model ID here to use it.