No software engineers' "fruit salad" here, instead, let me describe a fun project that I've started working on recently. I plan to build and document more projects like this in the future. Some of them may be open-sourced, while others only partially so. This section will be transferred to my blog when it is ready.

Back at uni, I had this idea that ended up as a final project for a course (with an additional medication dispenser designed by my friend that was supposed to go on top of the "sandwich" on the bot). Basically, turn a smartphone into a small mobile robot that comes to life when you install the phone in the contraption at the front, which allows the robot to look around. The hardware communicates with an app running on the smartphone via Bluetooth. It displays eyes on the screen and uses the front camera and OpenCV to follow the user. I wasn't familiar with ROS at the time and there were no multimodal LLMs available (2016). The project wasn't intended to be something serious, but rather a fun finalization of the course. The course didn't require projects to be actually physically implemented, but I was able to put the bot together. However, as is often the case with time-constrained uni projects, after the PowerPoint presentation I could only drive the prototype around using an Xbox 360 pad. The smartphone part was not ready and the app functionality was reduced to a JPEG picture of eyes on the screen.

A few years ago, I was presented with a sheet metal parts of discontinued Husarion product called Ralph. At the time, I had some plans for them, but ultimately the parts ended up in a box and then in a storage.

Recently, I came across them and decided to resurrect my old idea from uni in a slightly different form. I had to source a few parts like batteries, 18650 battery case, wheels, motors and IMU. I had an original CORE2 (STM32F407ZG). Instead of additional ESP32 I went for CORE2 + Raspberry Pi3A+ "sandwich". The Bosch bno055 (Adafruit) replaced MPU9250 as IMU.

My first step was to recreate the sheet metal parts and the entire robot in Fusion 360, a CAD/CAM/CAE/PCB design tool I am familiar with. I want to be able to make different modifications along the way and visualize them beforehand. Another benefit of having a model is that you can use one in the Gazebo/O3DE robot simulators.

Renders:

Assembled robot:

My goal is to create a self-balancing robot that on the one hand, can navigate autonomously using ROS2 and multimodal LLMs, while also having a Tamagotchi-esque features that make it a fun companion. Achievements in the field of AI have given rise to AI companions. The next stop, Physical AI. While the AI train is moving fast, and the technocrats work relentlessly to replace humans with physical AI (humanoid robots), I would like induce this 90s sense of excitement and hope for the future, by creating a small, cute robot from something we all carry in our pockets - a smartphone.

The primary focus of this project is the software. Through this project I aim to learn how to connect LLM clients with ROS2 hardware using Model Context Protocol. The robot will respond to voice commands and be able to answer questions about its surroundings. The robot will use the front camera, VL53L1X (or similar) distance sensor, and two Raspberry Pi cameras connected to the Raspberry Pi SBC. The system will be distributed across the CORE2 (real time system), Raspberry Pi SBC, a ROS2 docker container running on the laptop, an Android App and LLM clients running in the cloud. I also plan to have a web interface like rerun or foxglove for monitoring topics, camera feed, and other useful information.

For firmware, I am using Mbed OS Community Edition. It is a community-led continuation of Mbed OS, an embedded operation system (RTOS) for ARM Cortex-M microcontrollers. The original Mbed OS has been EOLed by Arm around August 2024. I have been using Mbed OS for around 7 years of my career at Husarion, so the bbot project is also my form of tribute to Mbed OS.

The board communicates with the SBC over UART using the micro-ROS library.

Progress:

• Create the template for CORE2 target for MbedOS CE.
• Create the template for micro-ROS + MbedOS CE for CORE 2.
• Implement the bbot firmware for the CORE2 target. <- I am here
• Add support for VL53L1X distance sensor.

Progress:

• Install Ubuntu Server 24.04 and ROS2 Kilted (included support for the Zenoh rmw).
• Build the Debian package for micro-ROS Agent using Docker and QEMU.
• Install the package and make sure the agent works and communicates with the CORE2 firmware.
• Cameras attached to CSI ports of the Raspberry Pi are working. <- I am here
• Create the flasher script for flashing the CORE2 firmware over UART.

I am working on other personal projects that are more of a serious nature, and I hope to eventually turn some of them commercial:

• Project Code V - a 3DGS/Zip-NeRF scanner, scanning techniques and a platform for a specific use case.
• Project Code H - a gamification ecosystem where neurodivergent (ADHD, ADD, ...) individuals are first class citizens.