Bus drivers who overlook an approaching racing bike while making a turn, cyclists who fail to notice a truck overtaking them: Every year, accidents involving cyclists and especially large vehicles make the headlines. Time and again, serious injuries and even fatalities are reported. “I am an ardent driver and cyclist myself,” Stabernack says. “And for me, every cyclist injured in a traffic accident is one too many.” Stabernack is Professor of Embedded Systems Architectures for Signal Processing, a joint appointment with the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (HHI) and the University of Potsdam. He has launched a research project to help prevent such accidents. “I can't understand why, year after year, new technologies are being developed for multimedia home entertainment – but not for road safety, where they could make a real difference. So I asked myself, 'Can I do something about it?' The answer was yes. So we launched the project, if only to demonstrate what’s possible.”
Artificial intelligence to link sensors
In cooperation with two industrial partners – Solectrix GmbH in Fürth and DResearch Fahrzeugelektronik GmbH in Berlin – Stabernack is developing a predictive sensor system to reliably detect the risk of a collision between a large commercial vehicle, such as a bus, and a cyclist traveling in the same direction. In case of an imminent danger, the system will warn both road users in good time. In fact, as Stabernack points out, simple warning systems are already being used. Many public buses have cameras installed that trigger an alarm in critical situations. However, these systems are not precise enough yet and often trigger false alarms. So for lack of trust, they are ignored or even switched off. “SySiKo” wants to do better. “We want nothing less than absolute safety,” Stabernack says. For that purpose, he and his team are combining various sensors in one system. The collected data are then analyzed using artificial intelligence to reliably detect potential collision risks.
For the prototype of their project, the researchers have installed an RGB camera, a lidar radar and a heat sensor in a small white box mounted to the outside of the test vehicle at about the level of the side mirror. In addition, three time-of-flight (TOF) cameras were attached with magnets on the outside of the vehicle. “We are testing the technology,” Stabernack explains. “How to integrate the system into vehicles, perhaps in mirrors or other body parts, will be dealt with later.” The researchers are using the test system to collect various data sets: RGB video images, radar and thermal data, and information provided by the TOF cameras. These cameras emit infrared light and measure how long it takes for the light to return. When a cyclist passes the vehicle and possibly overtakes it in its blind spot, these values will change – and make the cyclist “visible”. Even supporting sensors for or on cyclists are conceivable, such as real-time location systems using GPS, Bluetooth, wireless LAN or other radio signals, which are able to locate themselves spatially, Stabernack says. In the future, these sensors, which are not even expensive, could be mounted in bicycle helmets and communicate with the safety systems of vehicles – and thus save lives. If a bus and a bicycle get dangerously close to each other, the bus driver's alarm would go off and warn the cyclist, too, for instance through warning lights on the outside of the bus. “Light signals would indicate that the bus driver may not have noticed the cyclist and therefore not react properly.”
Driving the streets of Berlin in the test vehicle
Currently, Stabernack and his team are testing what data the sensors can deliver, and how well they can be analyzed. As yet, it is not clear which of them will be best suited for combined use. “We intend to evaluate the whole range of sensors – individually and in combination,” Stabernack says. “Therefore, we want to establish what additional detection reliability the various systems offer.”
To this end, the researchers spent many hours driving the streets of Berlin in their prepared test vehicle. But first of all they had to define what exactly they wanted the sensors to record. “Specifically, we asked bus drivers what they considered the most dangerous situations in road traffic,” explains Michal Stec, a member of the “SySiKo” team. “And we asked what they expected of a turn-off assisting system. How could it really help them?” Based on this prior research, a storyboard was created with all situations in urban traffic to be covered. Two sources of danger stood out by far: turning right at low speed with large steering wheel deflection, and changing lanes at higher speed with low steering wheel deflection. Besides, data had to be collected at as many times of the day, and year, as possible to find out whether the sensors provide reliable data at different temperatures and under different lighting conditions as well. Almost eighteen months of the project went into collecting data alone. Today, more than 100 hours of data are available.
“In the meantime, we have entered the training stage,” Stabernack explains. After all, the processing of the sensor data is performed with the help of artificial intelligence. “Although the term doesn't really fit,” Stabernack remarks. “Strictly speaking, these are artificial neural networks.” That is to say, they only detect what they were trained to. For this reason, all data need to be analyzed and labeled “manually” first. Cars, cyclists and other road users are marked on all data series. Then the AI starts practicing – “and it cannot be smarter than the person who taught it.”
Stabernack reckons that it could take years before “SySiKo” will assist drivers and cyclists on the road. Once the time-consuming AI training has been completed, the researchers will have to analyze which sensors contribute the most to detection reliability. It may well be that not all sensors are needed in the end. Quite obviously, the AI required to process the sensor data in real time has a huge hunger for energy. Not everything that can be done is suited for everyday use. Moreover, only what is affordable will find its way into mass-produced vehicles, and the market will “regulate” that. “Our industrial partners have already indicated that the price of the system must not exceed 200 euros,” Stabernack says. “But even a new cell phone may be 500 euros or more these days!” Stabernack is annoyed by the fact that many people seem to be willing to spend more on consumer electronics than on road safety. At the same time, this inspires him. “Our aim is to prevent these serious accidents one hundred percent of the time, and nothing will stop us!”
The Project
“SySiKo” – Vehicle-based system for the detection and signaling of a collision risk with weak road users in the blind spot area
Participants: University of Potsdam (Prof. Dr.-Ing. Benno Stabernack, Michal Stec); Solectrix GmbH, Fürth; DResearch Fahrzeugelektronik GmbH, Berlin
Funding: German Federal Ministry of Education and Research (BMBF)
Duration: 08/2019 – 07/2022
https://www.elektronikforschung.de/projekte/SySiKo
The Researchers
Prof. Dr.-Ing. Benno Stabernack studied electrical engineering at Technische Universität Berlin. He is the head of the Embedded Systems Group at Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (HHI), and has been Professor of Embedded Systems Architectures for Signal Processing at the University of Potsdam since 2016.
Mail: christian.stabernackuuni-potsdampde
Michal Stec studied electrical engineering at Technische Universität Berlin. Since 2019, he has been a research associate at the Chair of Embedded Systems Architectures for Signal Processing.
Mail: michal.stecuuni-potsdampde
This text was published in the university magazine Portal Wissen - Two 2021: Departure (PDF).