GPT-4o: Open-source robotic arm removes spilled liquid on command
Two roboticists spent only around 120 US dollars to build an open-source robotic arm that can wipe away spilled liquid on voice command.
The open-source robotic arm grabs a sponge on command and cleans up spilled liquid.
(Image: Jannik Grothusen (Screenshot))
Two roboticists from the University of Califonia Berkeley (UC Berkeley) and the Swiss Federal Institute of Technology in Zurich (EZH Zurich) have taught an open-source robotic arm to wipe spilled liquids off a table. In just four days, they managed to control the robot with OpenAI's multilingual, multimodal generative transformer GPT-4o. The robot provides information about what it sees before it starts the task. It then outputs its procedure step by step using speech.
The self-built open source robot arm costs just 120 US dollars. It uses an SO-100 robot arm developed by The Robot Studio. The robotic arm was specially developed for tasks such as cleaning up spilled liquids. It can pick up a sponge with a two-finger gripper. The arm only requires six servos, an adapter board for control with a computer and some 3D-printed parts. The parts are largely printed from carbon fiber to give the robotic arm the necessary lightness and strength. Assembly should only take 30 minutes. The plans for the robotic arm are publicly available on GitHub and can be rebuilt.
Consistent use of open source
To be able to control the robot by voice, the two developers created a visual language model with GPT-4o. The two roboticists used the open source framework LangChain AI for programming. The aim was for the robot arm to be able to be instructed by voice to carry out a cleaning process and at the same time explain the necessary steps in advance via voice output. This should make it easier for the user to control and provide a better understanding of the robot's actions.
Videos by heise
In order to continuously improve the movement of the robot arms, the roboticists used the open source framework Actor-Critic Training from LeRobot. Through reinforcement learning, the robot arm learns effective movement patterns that can be adapted and improved.
The robotic arm trained each individual movement strategy based on around 100 demonstrations of handling a sponge while mopping up. Using a camera, it is able to localize spilled liquid and can then wipe it up with a sponge. A video of this can be seen on X.
(olb)
