Intimate conversations with AI chatbots were visible to everyone

Beyond the office drones from Microsoft and the like, there are AI chatbots that also offer intimate and romantic conversations and corresponding images. In two such apps, an unsecured broker instance of middleware meant that all messages, all photos sent, and the IP addresses of users could be viewed by anyone with a corresponding link. The sums of money that some users put into the chatbots were also visible and particularly exciting.

The affected apps are "Chattee Chat - AI Companion" and "GiMe Chat - AI Companion", both from Hong Kong-based manufacturer Imagime Interactive Limited. The security vulnerability existed from the end of August to mid-September and was discovered by security researchers from the news portal Cybernews. The specialists have now made their investigation public.

Many users, few protective measures

In the case of "Chattee", the vulnerability affected a larger group of users: the app had 300,000 downloads in the Apple App Store and was the 121st most popular app in the Entertainment category. It is now no longer available here or in the Play Store, nor is "GiMe", which was significantly less popular. This could be due to an unsecured broker instance of the middleware Kafka, which was originally developed by LinkedIn and has been part of the Apache Software Foundation since 2012.

In the case of the compromised AI apps, it coordinated data streams of private messages between users and various instances of AI girlfriends and other companions, according to Cybernews. This included links to real photos and videos submitted by users, as well as photos and videos generated by AI. Anyone who had access here was able to track which user sent which content to which chatbot and which messages the users in turn received from the chatbots. Both iOS and Android users were affected. Data that directly reveals the identity of the user was not included, but IP addresses and unique device identifiers (UDID) were. These can often be used to identify the person behind the leaks with the help of previous leaks.

Intimate and sexual content

This would also enable cybercriminals to blackmail the affected users in many cases – because, according to a security researcher, there is virtually nothing among the leaked information that could be described as "safe for work". He is alluding to the very intimate and often sexual content of the chats in question. He also refers to the high discrepancy between the trust that users place in such apps and the security precautions that those responsible take to protect them.

According to the security researchers, such risks are also the reason a Kafka Broker should always be equipped with access controls and authentication options. However, this was not the case with the two apps. Anyone with the relevant link was able to connect to the app's content delivery network and access the relevant files and read the messages exchanged between users and chatbots. This was possible until 19 September, when the vulnerability was closed. Cybernews assumes that the data of a total of 400,000 users was previously publicly accessible.

The insights into payments that some users made for in-app purchases are also remarkable. For example, one user spent a total of 18,000 US dollars on in-app currency. According to Cybernews, however, he was the exception rather than the rule, even though there were several other users with similarly high amounts. It remains questionable whether they would have willingly paid for the use of the apps with their anonymity.

(nen)