Further down the road to open source AI: Red Hat forges ahead

Contents

Version 2.15 of the AI platform OpenShift AI includes a number of new functions, including a model registry for centralized model management. This allows predictive and generative AI models, metadata and model artifacts to be shared, versioned, provided and controlled. This was announced by the manufacturer Red Hat. OpenShift AI is an AI and machine learning platform for the development and operation of AI-supported applications in a hybrid cloud environment. Its tasks include data collection and preparation, model training, fine-tuning, model deployment, monitoring and hardware acceleration.

Recognizing deviations – and unbiased AI

Another new feature in OpenShift AI 2.15 is Data Drift Detection. The function recognizes changes in the distribution of input data for ML models already in use. Drift Detection warns data scientists as soon as the live data of the model predictions deviate significantly from the training data. As the input data is continuously monitored, the reliability of the model can be checked and the model can be adapted to the real data if necessary. Another new feature, Bias Detection, helps to develop fair and unbiased AI. The tool should not only show whether the models are unbiased based on the training data, but also monitor the fairness of the models in practice.

Another new feature is fine-tuning with Low Rank Adaption (LoRA). This enables faster fine-tuning of LLMs such as Llama 3. The new support for NVIDIA NIM accelerates the distributed deployment of AI applications using microservices. NIM is part of the NVIDIA AI Enterprise Software Platform and improves the distribution of GenKI applications. In addition to NVIDIA, OpenShift also supports AMD GPUs. This enables access to an AMD ROCm Workbench image to use AMD GPUs for model development. The new feature also allows access to images that can be used for serving and training/tuning use cases with AMD GPUs. This provides additional options for using GPUs to improve performance for compute-intensive activities. The new version of Red Hat OpenShift AI also works with KServe Model Cars: Open Container Initiative (OCI) repositories can be added as an option for storing and accessing containerized model versions.

Infrastructure-independent language models

Red Hat sees itself as an application-neutral platform provider and is therefore required to allow the platform-independent operation of AI models. However, there is an increasing problem here: the large language models are strongly tied to the hardware on which they run. To solve this problem, Red Hat recently acquired Neural Magic, a spin-off from MIT, and now wants to launch a kind of "Open Hybrid AI" on the market. To accomplish this, the LLMs must be abstracted to such an extent that they can run on any platform – called "virtual LLMs" (vLLMs) regarding virtual machines. The concept for this was developed at the University of California in Berkeley and is intended to allow AI models to run on various hardware platforms, including processors from AMD, Intel and Nvidia as well as custom chips from Amazon Web Services and Google.

Small models against extreme power requirements

However, this is not the only problem: the most powerful LLMs require a huge amount of computing power – and demand is increasing exponentially. On the other hand, the large universal models are oversized for most company applications. Red Hat relies on "Small Language Models" (SML), i.e. models that are suitable for simple AI use cases. A large model is trained so specifically that certain applications can be processed satisfactorily with it. Red Hat uses InstructLab for this. Together with RHEL and a Granite model from Red Hat's parent company IBM, it can be used to train almost any model so specifically that it fits a given use case exactly. These models require significantly less computing power so that they can run on your own server or even on one of the new AI PCs. Red Hat is working closely with Intel on this.

IBM has integrated its Granite-7B language model into InstructLab so that anyone can add new skills and knowledge and customize it for an organization's specific needs – without losing any of what the model was previously trained with. In terms of performance, IBM says that the original training of the Granite code models for translation from COBOL to Java required 14 rounds of fine-tuning, which took a total of nine months. With InstructLab, the team was able to add newly tuned COBOL capabilities and achieve better performance with just one round and a time requirement of one week.

Shortcomings in technology, pluses in transparency

Overall, the use of open source for AI is still viewed critically. This concerns independence from large companies, for example. "Without the big AI companies, such as Microsoft, Facebook, Google and Tesla, AI cannot develop any further – This is particularly true for the use of AI with open-source models," says Hans Roth, Red Hat's Senior Vice President and General Manager EMEA. This is mainly because the development and maintenance of LLMs requires immense resources that the community cannot provide.

But regardless of this, open source is not yet an equivalent alternative to proprietary products. The LF AI & Data Foundation, a sub-organization of the Linux Foundation that specializes in AI, has been working intensively on this topic. In a comparative study, the current open-source models performed worse than closed source models in most tasks. This also applies to security. An analysis of over 100,000 open-source models on Hugging Face and GitHub using code vulnerability scanners such as Bandit, FlawFinder and Semgrep revealed that over 30 percent of the models have serious vulnerabilities. Closed models generally have far fewer security risks.

A particular problem with the use of open source for artificial intelligence is the confusing classification. The organization complains that many providers engage in "openwashing": they claim that their product is open source, even though their products run counter to the recognized principles and freedoms of open source. According to the LF AI & Data Foundation's definition, an open-source AI system is one that meets the following conditions: "It can be used for any purpose and without permission, the functionality of the system can be freely studied, the system can be modified for any purpose, including changes to the output, and the system can be released with or without modifications for others to use for any purpose. These requirements apply both to a fully functional system and to individual elements of a system." Models that meet these criteria are Pythia (Eleuther AI), OLMo (AI2), Amber, CrystalCoder (LLM360) and T5 (Google). However, there are a few others that could also be included if they were to change their licenses or legal conditions. These include BLOOM (BigScience), Starcoder2 (BigCode) and Falcon (TII). In any case, Llama2 (Meta), Grok (X/Twitter), Phi-2 (Microsoft) and Mixtral (Mistral) are incompatible with the open-source principles.

With this definition, the Foundation indirectly addresses one of the biggest problems of closed source models: The lack of transparency of the algorithms and the data used. This may even lead to considerable legal problems. Open-source providers see themselves at a clear advantage here. "All of our software is guaranteed to be free from third-party claims – and this also applies to our AI solutions," is Hans Roth's strong argument in favor of open source.

(vbr)