Autonomous Tangram Solver

Author: Allen Liu

This repo contains the packages necessary for a robot arm to autonomously picks up tangram pieces and solves the tangram puzzle.

Final Video

Final.Project.-.HD.1080p.mp4

Install Dependencies

Boost

sudo apt install libbost-all-dev

OpenCV

sudo apt install libopencv-dev

Armadillo

sudo apt install libarmadillo-dev

librealsense2

sudo apt install librealsense2-dev librealsense2-utils

ROS2 Iron

Configure locale

locale  # check for UTF-8

sudo apt update && sudo apt install locales
sudo locale-gen en_US en_US.UTF-8
sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
export LANG=en_US.UTF-8

locale  # verify settings

Add universe repo

sudo apt install software-properties-common
sudo add-apt-repository universe

Add ROS2 Key

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null

Install ros dev tooks

sudo apt update && sudo apt install ros-dev-tools

Upgrade machine (Caution)

sudo apt update && sudo apt upgrade

Install ROS2 Iron Desktop

sudo apt install ros-iron-desktop-full

Building all packages

Setup backages

export WORKSPACE=<path/to/your/workspace>

mkdir -p ${WORKSPACE}/src
cd ${WORKSPACE}
vcs import --input src < https://raw.githubusercontent.com/nu-jliu/Autonomous_Tangram_Solver/refs/heads/main/tangram.repos
rosdep install --from-paths src --ignore-src -r -y --skip-keys boost --skip-keys OpenCV

Build

cd ${WORKSPACE}
colcon clean workspace -y
colcon build --symlink-install --event-handlers console_direct+

Launch the Program

To launch the entire software run the following command in a terminal

cd ${WORKSPACE}
source install/setup.bash
ros2 launch tangram_bot tangram.launch.py

Then the program will start by self-calibration. After you see robot arm moving out of the camera frame, you shall place the target tangram puzzle shape in the front of the camera, then run the following command, the robot will solve the puzzle automatically.

ros2 action send_goal /action/execute tangram_msgs/action/ExecuteAction {} -f

Software Architecture

The entire software architecture for the project is shown in the figure below, this diagram shows data is transmitted between ROS nodes.

Packages

The software developed for this project is composed of 8 interconnected packages, each performing a specific function to enable the robotic system to solve tangram puzzles. The packages are described as follows:

• hand_eye_calibration: Implements the steps for hand-eye calibration to determine the positional relationship between the camera frame and the robot frame.
• image_segmentation: Executes all vision models used in this project to segment images based on a trained model.
• maxarm_control: Handles low-level position control of the robot arm, ensuring precise movements.
• piece_detection: Detects the pose (position and orientation) of all tangram pieces required for solving the puzzle.
• puzzle_solver: Generates the goal pose for each tangram piece to solve the puzzle.
• tangram_bot: Generates and executes robot actions to pick up and place tangram pieces, completing the puzzle assembly.
• tangram_msgs: Contains all custom interfaces for Inter-Process Communication (IPC) between different system components.
• tangram_utils: Provides library functions for tangram geometric calculations, supporting operations like pose transformations and shape analysis.