Eyes closed in the autonomous car: AI to make napping at the wheel safer

Contents

Anyone stuck in traffic in a highly automated SAE Level 3 vehicle today is theoretically allowed to take their hands off the steering wheel and their eyes off the asphalt. The car takes over braking, accelerating, and steering, while the person in the driver's seat can use the newfound freedom for a book or to look at their smartphone. But one limit remains ironclad: as soon as the eyes close, the system sounds the alarm.

The problem is less the legal situation and more the technical inability of the current generation of sensors. Today's cars simply cannot reliably distinguish whether a driver is briefly closing their tired eyes to relax or already drifting into deep sleep.

The automotive industry distinguishes between different levels of autonomy in this environment. With SAE Level 3 (highly automated), the system completely takes over the driving task under certain conditions, but the human must be ready to take over the steering wheel within a warning period. With SAE Level 4 (fully automated), the vehicle is so independent in specific scenarios, such as on the highway, that the human could theoretically switch off completely and even sleep: The car will bring the system into a safe state independently and according to plan if necessary.

AI analysis instead of mere video surveillance

This is where the Salsa research project comes in ("Smart, Adaptive, and Learnable Systems for All"). Behind it is a consortium of automotive manufacturers, suppliers, and scientists, driven by the University of Stuttgart and the Fraunhofer Institute for Optronics, System Technologies, and Image Exploitation (IOSB), among others. Its experts in Karlsruhe are working to make it technically possible to relax with closed eyes.

In extensive user studies, the IOSB researchers have therefore collected data to train an Artificial Intelligence (AI). This AI is intended to be able to do much more than just detect closed eyelids. It's about nuances in posture and minimal movements that are captured by interior cameras and analyzed in real-time.

The highlight of the experiments in the institute's realistic driving simulator is the combination of different measurement methods. While sensors, like those already installed in production vehicles, track the driver's posture, scientists simultaneously monitor brain activity using electroencephalography (EEG). In this process, electrodes on the scalp measure the electrical activity of the cerebrum. The data obtained serves as a reference to teach the AI to assess a person's actual state as precisely as possible. This way, the system is supposed to learn whether a person is merely regenerating or has already sunk into Morpheus' arms. This is crucial, as the requirements for taking over the driving task differ significantly depending on the state.

Window of Drowsiness

A central aspect of the research is so-called "Sleep Inertia," the phase of grogginess immediately after waking up. The IOSB studies according to initial publications show that physical and mental performance is significantly reduced after real sleep. A relaxed driver with closed eyes is therefore fully ready to act again after about ten seconds. In the case of a necessary takeover of vehicle control at Level 4, a sleeping person, on the other hand, needs at least two minutes before they can act safely again. This time discrepancy must be factored in by the vehicle's AI to avoid dangerous situations during the transition from autonomy to manual driving.

To ensure that this change at the wheel is not only safe but also comfortable, the project team investigated what a "gentle awakening" by the vehicle could look like. A mere shrill warning signal is often not enough or leads to startled reactions. Instead, the researchers identified a combination of vibrations in the seat and targeted sound elements as the optimal way to bring the driver back to the here and now. The scientists claim to have already successfully tested a corresponding prototype.

Practical Check in Schwabenlandhalle

The results of this work are more than just dry theory. They could form the basis for a new generation of driver state models that will ensure greater safety and acceptance in production vehicles in the future. Anyone who wants to get an idea of the current state of the art can do so on February 12th in the Schwabenlandhalle in Fellbach. There, the project consortium, which has received a financial boost of around 10 million euros from the Federal Ministry for Economic Affairs and Energy, will present its interim assessment. The path to a relaxed nap behind the wheel may still be long, but the technological guardrails are currently being laid in Karlsruhe and Stuttgart.

(tho)