Smart Hans effect: Danger due to incorrect learning
Despite the correct result, AI may have learned something by making mistakes – the Clever Hans effect. Hans was a horse.
Not clever Hans, but a horse.
(Image: Daniel AJ Sokolov)
Clever Hans was a horse that could supposedly do math. To indicate a number, he would scuff his hooves. In fact, the results were correct, but the horse found another way to get the right result than doing the math. The horse could tell its trainer when to stop pawing. So you could say that the result could not be reproduced under different conditions – without the trainer –. The Kluge Hans effect: it can be transferred to AI models.
A research team led by TU Berlin investigated the effect regarding the unsupervised learning of AI models, the article on this was published in Nature Machine Intelligence. This unsupervised learning refers to a form of machine learning in which the models themselves are supposed to recognize patterns and correlations – without being given a direction or question or the prospect of a reward. Conversely, it is also possible to detect anomalies, i.e. deviations in data. This is how cancer can be found in medicine, for example.
AI pays attention to notes instead of Covid characteristics
This becomes problematic when a model does not look at the decisive image at all, but at an actually unimportant detail – such as the frame. In the study, the authors used X-ray images of lungs with and without Covid-19. In some cases, the AI did not classify the images based on the Covid features, but on other characteristics. These included the notes on the edge of the images. This resulted in a number of incorrect classifications. The assignments were monitored using a so-called explainable AI, which provides insight into the AI's decision.
Videos by heise
In another test with the image of a piece of wood, the researchers came to a similar conclusion. They conclude that unsupervised learning is particularly susceptible to the Kluger-Hans effect. They also speak of spurious correlations, i.e., assumed relationships that do not actually exist. In addition to false diagnoses, they also warn of potentially false recalls, as monitoring often takes place in production using anomaly detection.
The authors raise the question of when it should be checked whether there is a Clever Hans effect (also referred to as the Clever Hans 'CH' effect in the study). This could already be checked in unsupervised learning, but also in downstream models. They suggest that CH effects should already be considered in the unsupervised model so that all downstream models benefit from them. Explainable AI and human supervision therefore seem to be possibilities that can at least minimize the effect.
(emw)
