You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This package provides a ROS 2 node that interfaces with the [Idea-Research GroundingDINO Model](https://github.com/IDEA-Research/GroundingDINO)for open-set object detection.
5
+
This package provides ROS2 integration with [Idea-Research GroundingDINO Model](https://github.com/IDEA-Research/GroundingDINO)and [Grounded-SAM-2, RobotecAI fork](https://github.com/RobotecAI/Grounded-SAM-2) for object detection, segmentation, and gripping point calculation. The `GroundedSamAgent` and `GroundingDinoAgent` are ROS2 service nodes that can be readily added to ROS2 applications. It also provides tools that can be used with [RAI LLM agents](../../../docs/tutorials/walkthrough.md) to construct conversational scenarios.
6
6
7
+
In addition to these building blocks, this package includes utilities to facilitate development, such as a ROS2 client that demonstrates interactions with agent nodes.
7
8
8
9
## Installation
9
10
10
-
In your workspace you need to have an `src` folder containing this package `rai_perception` and the `rai_interfaces` package.
11
+
While installing `rai_perception` via Pip is being actively worked on, to incorporate it into your application, you will need to set up a ROS2 workspace.
git archive --format=tar --prefix=rai_perception/ HEAD:src/rai_extensions/rai_perception | tar -xf -
26
+
mv rai_perception ../rai_perception
27
+
cd ..
28
+
rm -rf temp
18
29
```
19
30
20
-
##Build and run
31
+
### ROS2 Dependencies
21
32
22
-
In the base directory of the `RAI` package install dependencies:
33
+
Add required ROS dependencies. From the workspace root, run
23
34
24
-
```
25
-
poetry install --with perception
35
+
```bash
36
+
rosdep install --from-paths src --ignore-src -r
26
37
```
27
38
28
-
Source the ros installation
39
+
### Build and Run
29
40
30
-
```
31
-
source /opt/ros/${ROS_DISTRO}/setup.bash
32
-
```
41
+
Source ROS2 and build:
33
42
34
-
Run the build process:
43
+
```bash
44
+
# Source ROS2 (humble or jazzy)
45
+
source /opt/ros/${ROS_DISTRO}/setup.bash
35
46
36
-
```
47
+
# Build workspace
48
+
cd~/rai_perception_ws
37
49
colcon build --symlink-install
50
+
51
+
# Source ROS2 packages
52
+
source install/setup.bash
38
53
```
39
54
40
-
Source the environment
55
+
### Python Dependencies
41
56
42
-
```
43
-
source setup_shell.sh
57
+
`rai_perception` depends on `rai-core` and `sam2`. There are many ways to set up a virtual environment and install these dependencies. Below, we provide an example using Poetry.
58
+
59
+
**Step 1:** Copy the following template to `pyproject.toml` in your workspace root, updating it according to your directory setup:
> To manage ROS 2 + Poetry environment with less friction: Keep build tools (colcon) at system level, use Poetry only for runtime dependencies of your packages.
103
+
52
104
<!--- --8<-- [end:sec1] -->
53
105
54
-
Agents create two ROS 2 Nodes: `grounding_dino` and `grounded_sam` using [ROS2Connector](../../../docs/API_documentation/connectors/ROS_2_Connectors.md).
106
+
`rai-perception` agents create two ROS 2 nodes: `grounding_dino` and `grounded_sam` using [ROS2Connector](../../../docs/API_documentation/connectors/ROS_2_Connectors.md).
@@ -68,83 +120,109 @@ These agents can be triggered by ROS2 services:
68
120
## RAI Tools
69
121
70
122
`rai_perception` package contains tools that can be used by [RAI LLM agents](../../../docs/tutorials/walkthrough.md)
71
-
enhance their perception capabilities. For more information on RAI Tools see
123
+
to enhance their perception capabilities. For more information on RAI Tools see
72
124
[Tool use and development](../../../docs/tutorials/tools.md) tutorial.
73
125
74
-
<!--- --8<-- [start:sec3] -->
126
+
<!--- --8<-- [start:sec2] -->
75
127
76
128
### `GetDetectionTool`
77
129
78
-
This tool calls the grounding dino service to use the model to see if the message from the provided camera topic contains objects from a comma separated prompt.
130
+
This tool calls the GroundingDINO service to detect objects from a comma-separated prompt in the provided camera topic.
79
131
80
-
<!--- --8<-- [end:sec3] -->
132
+
<!--- --8<-- [end:sec2] -->
81
133
82
134
> [!TIP]
83
135
>
84
136
> you can try example below with [rosbotxl demo](../../../docs/demos/rosbot_xl.md) binary.
85
-
> The binary exposes `/camera/camera/color/image_raw` and `/camera/camera/depth/image_raw` topics.
137
+
> The binary exposes `/camera/camera/color/image_raw` and `/camera/camera/depth/image_rect_raw` topics.
86
138
87
-
<!--- --8<-- [start:sec4] -->
139
+
<!--- --8<-- [start:sec3] -->
88
140
89
141
**Example call**
90
142
91
143
```python
144
+
import time
92
145
from rai_perception.tools import GetDetectionTool
93
146
from rai.communication.ros2 import ROS2Connector, ROS2Context
I have detected the following items in the picture - chair, human
165
+
I have detected the following items in the picture plant, table lamp, table lamp, bed, desk
107
166
```
108
167
109
168
### `GetDistanceToObjectsTool`
110
169
111
-
This tool calls the grounding dino service to use the model to see if the message from the provided camera topic contains objects from a comma separated prompt. Then it utilises messages from depth camera to create an estimation of distance to a detected object.
170
+
This tool calls the GroundingDINO service to detect objects from a comma-separated prompt in the provided camera topic. Then it utilizes messages from the depth camera to estimate the distance to detected objects.
112
171
113
172
**Example call**
114
173
115
174
```python
116
-
from rai_perception.tools importGetDetectionTool
175
+
from rai_perception.tools importGetDistanceToObjectsTool
117
176
from rai.communication.ros2 import ROS2Connector, ROS2Context
177
+
import time
118
178
119
179
with ROS2Context():
120
180
connector=ROS2Connector(node_name="test_node")
121
-
connector.node.declare_parameter("conversion_ratio", 1.0) # scale parameter for the depth map
181
+
connector.node.declare_parameter("conversion_ratio", 1.0) # scale parameter for the depth map
182
+
183
+
# Wait for topic discovery to complete
184
+
print("Waiting for topic discovery...")
185
+
time.sleep(3)
186
+
122
187
x = GetDistanceToObjectsTool(connector=connector)._run(
I have detected the following items in the picture human: 3.77m away
199
+
I have detected the following items in the picture desk: 2.43m away
134
200
```
135
201
136
202
## Simple ROS2 Client Node Example
137
203
138
-
An example client is provided with the package as `rai_perception/talker.py`
204
+
The `rai_perception/talker.py` example demonstrates how to use the perception services for object detection and segmentation. It shows the complete pipeline: GroundingDINO for object detection followed by GroundedSAM for instance segmentation, with visualization output.
205
+
206
+
This example is useful for:
139
207
140
-
You can see it working by running:
208
+
- Testing perception services integration
209
+
- Understanding the ROS2 service call patterns
210
+
- Seeing detection and segmentation results with bounding boxes and masks
0 commit comments