Hugging Face: Malicious ML models uncovered on development platform

IT researchers have discovered maliciously manipulated machine learning models in the repositories of Hugging Face. The specific ML models found have opened a reverse shell on infected systems. Developers and interested parties can download data sets and pre-trained AI models from Hugging Face, including “slimmed-down” versions of AI models from Deepseek, Mistral or Qwen, which can be used on weaker hardware at home.

A blog post by IT security researchers from Reversinglabs analyzes the malware dubbed “nullifAI”. Malicious actors have infiltrated Hugging Face with manipulated AI models. They misuse a required function, Pickle File Serialization, also known as Pickling. Pickle is a Python module that is often used to serialize and deserialize ML model data. The Pickle format is considered unsafe because it allows Python code to be executed during ML model deserialization. Hugging Face also points out this risk in its documentation.

Slipped past malware detection

The IT researchers came across two Hugging Face models that contained malicious code and were not flagged as unsafe by the Hugging Face security mechanisms. The ML model examined gave the impression of a proof-of-concept model to test this novel attack method. A screenshot shows the scan results of Hugging Face: ProtectAI and ClamAV found no problem, HF Picklescan apparently did not even recognize the file as a pickle.

The two malicious files are in PyTorch format, which is essentially a compressed pickle. By default, ZIP compression is used for this, but these models were compressed with 7z. As a result, the standard PyTorch function torch.load() cannot load them. The IT researchers assume that the HF Picklescan did not mark the file as insecure for this reason. Picklescan is a very limited scanner anyway: it uses a blacklist of dangerous functions; the IT security experts at Checkmarx have also discovered other functions that can cause code to be executed but are not on this blacklist. Furthermore, Picklescan is apparently unable to analyze defective pickles.

As a further obfuscation tactic, the object serialization of both files is defective shortly after the malicious payload. This leads to errors when decompiling the object. As with previously known malware, the attackers also use the concealment variant here to evade detection through defective formats. Although they function as desired in the target system, they cannot be correctly analyzed by the parsers in security software.

The IT researchers cite the execution of malicious commands, the opening of network connections, the creation of new processes or access to cameras, microphones, and file systems as examples of what attackers can do by abusing this deserialization gap. The specific samples contained a reverse shell customized for each platform, which connects to a hardcoded IP address.

The IT researchers are still discussing further tests with which they have verified their theses. However, they and Hugging Face have not been able to come up with any simple or tangible solutions to the pickle problem. The middle ground of not banning Pickle and using security mechanisms to analyze the Pickle files and continuously improve the scans remains. Indicators of compromise (IOCs) can be found at the end of the blog entry.

(dmk)