Seite 4: How OpenAssistant differs from ChatGPT

Inhaltsverzeichnis

The crucial question for many users and developers might be: How does the free conversation assistant compare to OpenAI? According to the publishers, it can compete with GPT-3.5 Turbo, the direct predecessor of GPT-4 (OpenAI's user interface ChatGPT sits on top of various models of the GPT series). According to the OpenAssistant paper, which is still in the draft stage, human users prefer their assistant sitting on the fine-tuned Pythia-12B to the answers generated by GPT-3.5 Turbo by 48.3 percent (more on this in Appendix E of the draft).

However, this preference can probably be attributed in part to the fact that the wizard censors less than ChatGPT and some positive outliers among the prompt-output pairs would have to be factored out (this reduces the preference to about 40 percent, which would still be a strong result). One major difference from ChatGPT is that OpenAssistant does not engage in self-censorship. The draft paper does not yet take this into account accordingly. Therefore, the claim that this is the first complete open-source chat language model that can compete with GPT-3.5 Turbo is probably a bit exaggerated, as the Australian AI researcher Jeremy Howard (founder of FastDotAI), among others, critically remarked on Twitter. Jeremy Howard goes deeper into the evaluation question in a thread.

A scientific evaluation on the capabilities of the model is currently not available, the evaluation in the appendix of the paper refers to human preferences alone, according to one of the main developers. With this limitation in mind, nothing stands in the way of testing it out yourself. Since the underlying models are largely licensed under the Apache 2.0 licence, commercial use of models derived from them as closed source is presumably ruled out: free use, modification and distribution, on the other hand, are permitted. Modified derivatives must, according to Apache 2.0, indicate in a conspicuous place that they are modified, and retain all original copyright notices.

Beware, even open-source AI is prone to fabrication

There are a few cautions to keep in mind: As with its big brother ChatGPT, the models created by the OpenAssistant team in training are subject to the warning that they may be partially off in math and programming tasks, and the output may be factually incorrect or misleading. "Answers can look convincing at first glance, but at the same time contain completely fabricated false statements," the team discloses in the repository in the model description. Only English-language examples are published in the draft.

OpenAssistant can be tested on the project website (logging in via email does not seem to work currently, probably a temporary problem). The models and datasets are available in the project's repository on Hugging Face. The draft research report is publicly viewable as a preprint on a Google Drive ("OpenAssistant Conversations – Democratizing Large Language Model Alignment").

(sih)