AI language model: Claude from Anthropic can now move the mouse pointer
Anthropic has presented a major update for its AI, which can now interact directly with the PC. For example, it can move the mouse pointer.
(Image: Maciej Walczak/Shutterstock.com)
Following a major update, Anthropic's most powerful language model can now also interact directly with a computer and move the mouse pointer, for example. The AI company announced this in a blog post and added that Claude 3.5 Sonnet is the first AI model ever to be able to do this. It will also be able to recognize the content of a screen and press buttons. Anthropic is confident that the update will first make the function available to developers, whose feedback will quickly improve the capabilities. The first companies are already working with it.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
In a video, an Anthropic researcher demonstrates what the new function is capable of. In the example, a form with various address data is to be filled in and sent. The necessary data can be found on the computer, but first has to be found by the AI model. After the task has been formulated, the software searches a screenshot of the current screen content, but is unable to find it. A search window is then opened, and the search term is entered. The find is saved again as a screenshot and the form is opened. The requested data is entered there one by one before the submit button is pressed.
Anthropic expects the new function to be improved quickly over the coming months. It still has massive limitations in some cases and has problems with actions that people find extremely easy or cannot perform them. This is also because the AI technology takes numerous screenshots, which are then analyzed. Scrolling, dragging and dropping and zooming are therefore difficult or impossible. Anyone trying out the function should therefore start with less risky tasks first, advises Anthropic. Also because it is not clear from Anthropic's statements what happens to these screenshots, the technology should not be used for particularly sensitive tasks – such as bank transfers –. It is therefore also unclear how close the function is technically to Microsoft's Recall, to which the screenshots were fatal. Because the use of the computer through an AI model also opens up new ways to carry out known dangers such as spam, disinformation or fraud, the company will proactively ensure secure distribution.
Apart from the new function, the performance of the model has also been significantly improved with the update, writes Anthropic. In some cases, noticeably higher values were achieved in various benchmarks. The update represents a significant leap forward, especially for AI-supported programming. In addition to Sonnet, Anthropic has also improved the Haiku model, the smallest of the company's three. This is to be made available this month and is still ideally suited for products with which users interact directly.
(mho)