CoFRIDA: Robot paints pictures together with a human
In order for a robot to paint a picture together with a human, the robot must know what the human intends to do.
The CoFRIDA robotic system can paint a picture together with a human.
(Image: Carnegie Mellon University)
A team of scientists at the Robotics Institute (RI) at Carnegie Mellon University (CMU) has developed a robotic system called "Collaborative FRIDA" (CoFRIDA – Collaborative Framework and Robotics Initiative for Developing Arts), which can work together with humans in artistic activities such as painting pictures.
"It's like the drawing equivalent of a writing prompt," says Jim McCann, associate professor at the RI. "If you're stuck and you don't know what to do, it can put something on the page for you. It can break the barrier of an empty page. It's a really interesting way of enhancing human creativity."
Videos by heise
The CoFRIDA robotic system can be used in two different ways, the CMU researchers write in their scientific paper "CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Painting", which has been published as a preprint on Arxiv. On the one hand, the multi-axis robot arm, which can be equipped with a brush or a permanent marker, can be given instructions in text form as to what the robot should paint. Instructions in picture form are also possible. On the other hand, the robot can help the human artist based on a painting that has already been started, for example by CoFRIDA and the human taking turns to continue painting the picture.
Co-painting
This process, known as co-painting, is anything but trivial, as CoFRIDA must understand the overarching goals of the human artist so that the robot can make meaningful strokes and continue painting the picture. The researchers created their own self-monitored training data set for this purpose. It contained drawings from which the researchers removed individual elements so that the original motif was still recognizable. For example, the scientists removed the edge of the wheels of a car and its windows, but retained the outline of the vehicle.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
"We tried to simulate different stages of the drawing process," explains Jun-Yan Zhu, one of the scientists involved in the CoFRIDA project. "It is easy to arrive at a final sketch, but it is quite difficult to imagine the intermediate stages of this process.
Using the dataset of semi-finished and finished images, the researchers created a text-image model called InstructPix2Pix. This allows CoFRIDA to add brush strokes to images that have already been started. The robot system can continue the sketch in different directions. However, the decision as to what the result will look like remains with the human, who thus also retains artistic control.
(olb)
