Eight area scan cameras and eight radars observe traffic on the A9 highway from two overhead signs (see the video above). The locally collected data is processed on-site; objects are identified and anonymized. Fusion of the sensor data creates a digital twin in real time. The graphic on the right shows the average speed at this exact moment, as well as how many vehicles are detected by the cameras and how many vehicles are expected to drive past the sensor stations per hour.

0
20
40
60
80
100
120
140
160
120 km/h
vehicles currently
detected


32.589 vehicles per hour

What is the status of Providentia++?

The first phase of the Providentia project showed that the digital highway of the future is in principle possible. The challenge: Radar and lidar sensors installed in modern vehicles can survey the vehicle’s surroundings but have a limited range. In addition, they are not able to see behind passing vehicles. Providentia offers a “cross-vehicle system” that complements the vehicle’s own sensor technology.

During the first three years, it was shown that it is technically possible to create a digital twin of the traffic on the A9 highway in real time, using radar systems and cameras mounted on overhead highway signs. This virtual image can then be sent to any networked vehicle, thus increasing the range of the sensors in the vehicle. With apps, this fused data can also be used to make lane recommendations or warn of accidents.

Bilder anzeigen

Cameras and radar systems observe the traffic on federal and urban roads and at the A9 highway near Garching.

Bilder anzeigen

FURTHER ARTICLES ABOUT THE PROJECT

July 06,2021

Resilience for sensors and algorithms: 3 errors in focus

The external infrastructure plays an important role in compensating for sensor errors and making connected driving safer and more reliable. AI expert Dinu Purice from Providentia++ partner Cognition Factory is working on algorithms to improve reliability.

MORE >

June 22,2021

Plausibility checks of sensors: Particularly efficient in networks

The more sensors are integrated in a network such as Providentia++, the easier it is to find errors. Intel has developed a plausibility check for the roadside infrastructure monitoring sensors. Researcher Dr. Florian Geißler on identifying and excluding bad measurements.

MORE >

February 26,2021

Autonomous driving: High-resolution 3D road maps as an additional sensor

High-resolution road maps are a prerequisite for automated and autonomous driving. They provide vehicles with the framework in which they can move. 3D Mapping Solutions is measuring the streets with millimeter precision and creating HD maps for Providentia++.

MORE >

Who drives the project?

The main tasks of the team of scientists from the chair for robotics, artificial intelligence and real time systems at the TU München are to expand the existing infrastructure, to improve the object detection systems, to accelerate the data fusion up to real time and to make the entire system robust against external influences. The video (in GERMAN) shows, how the scientists of the TU München work, what they are working on and what drives them foreward.

Why Providentia?

Providentia is an acronym formed from the German words for “Proactive Video-Based Use of Telecommunication Technologies in Innovative Traffic Scenarios.” Providentia is also the Roman goddess of providence and care. This is fitting, as the digital twin will make it possible for networked vehicles to predict which lane the vehicle should use in order to navigate through traffic as smoothly as possible and avoid traffic jams. It will also help prevent accidents, as the sensor technology will be able to anticipate hazards.

Providentia is an acronym formed from the German words for “Proactive Video-Based Use of Telecommunication Technologies in Innovative Traffic Scenarios.” Providentia is also the Roman goddess of providence and care. This is fitting, as the digital twin will make it possible for networked vehicles to predict which lane the vehicle should use in order to navigate through traffic as smoothly as possible and avoid traffic jams. It will also help prevent accidents, as the sensor technology will be able to anticipate hazards.

The first phase: Testing the digital twin on the highway

In the first phase of the Providentia project, high-resolution cameras and radar systems were mounted on two overhead signs on a test section of the A9 highway (marked in green in the picture). The sensor data was transmitted wirelessly via 5G, artificial intelligence was used to identify vehicle types and classes, and the data was then “fused” – in other words, a digital twin was created. An autonomous vehicle was developed that can use the information from the digital twin to independently change lanes on the highway, for example, or to slow down to avoid traffic jams or accidents.

In the first phase of the Providentia project, high-resolution cameras and radar systems were mounted on two overhead signs on a test section of the A9 highway (marked in green in the picture). The sensor data was transmitted wirelessly via 5G, artificial intelligence was used to identify vehicle types and classes, and the data was then “fused” – in other words, a digital twin was created. An autonomous vehicle was developed that can use the information from the digital twin to independently change lanes on the highway, for example, or to slow down to avoid traffic jams or accidents.

The second phase: From the highway to the residential area

By the end of 2019, the infrastructure on the A9 highway had been set up and the digital twin of the highway traffic had been developed. The next step: to fine-tune the technology, to expand the infrastructure into “urban space” (marked in blue in the picture), and to develop so-called value-added services. New transmitter and sensor towers will be erected on the edge of a feeder road into a residential area. Intersections, traffic circles, bus stops, train stations, and park-and-ride parking lots can be surveyed anonymously by the sensor technology.

Lidar systems, which generally cover a larger angle than radars, will be used in the project for the first time. In addition, value-added services will be created for drivers, highway operators, vehicle manufacturers, and academia. The follow-up project Providentia++ was launched in early 2020 and is expected to last two years. The current Providentia++ project is being funded with 9.16 million euros from the Federal Ministry of Transport and Digital Infrastructure (BMVI) and industry partners. The industry partners are the Technical University of Munich as consortium leader, Cognition Factory, Elektrobit Automotive, Valeo, fortiss, and Intel Deutschland. Additional associated partners are Huawei Technologies Deutschland, IBM Deutschland, brighter AI, and 3D Mapping Solutions.

 

 

 

What are the research areas?

OBJECT RECOGNITION in multi-sensor scenarios

Data on the current traffic situation is constantly being recorded by eight area scan cameras and eight radars mounted on two overhead signs on the A9 highway. An area scan camera generates about 400 megabits per second, amounting to nearly 35 gigabytes per day. Since the systems should ultimately be as real-time capable as possible, a back-end computer reduces this volume of data to the essentials – information about the object and its class and position. In the end, only a few hundred bytes are generated per object. It is precisely this small volume that makes it possible to transfer the data in real time.

 

DATA FUSION and the digital twin

The goal is to uniquely identify objects and determine their position, with as little effort as possible. There are various challenges related to the fusion of sensor data: Cameras and radars record image data at different frequencies – cameras at 30 frames per second, and radars at 10 frames per second. They must be synchronized with each other. Moreover, sensor data is not always transmitted without interference. Vibrations on the overhead signs and poor weather conditions can impair data quality. The challenge is to evaluate and weight the individual data sources, thus laying the foundation for local digital twins, which are created at the respective measuring stations, and global twins, which take all available sensor data into account.

ERROR TOLERANCE, reliability, and scalability

If individual sensors provide poor data or even fail, the overall system must still be able to provide reliable information. The goal is for the overall system to be able to decide autonomously which systems are used and which are left out of the calculations. This makes the digital twin error-tolerant and able to provide usable information at all times for vehicles that use value-added services from the digital twin. It is also interesting that in-vehicle assistance systems are able to check their own sensor data for plausibility via the external infrastructure. This makes the digital twin even more reliable. Warning of risk of collision, improving traffic flow, controlling traffic, and enabling real-time traffic analysis: These are services that are of interest to very different stakeholders, including vehicle owners, highway operators, and car manufacturers.

 

COMMUNICATION PROTOCOL

Data exchange between system units and vehicles in real time necessitates customized message protocols that adapt the available network capacity to the specific situation and that prioritize tasks. Every vehicle that is near the overhead highway signs receives customized information – in other words, its individual digital twin. If two vehicles from the same “region of interest” request a digital twin at the same time, they receive it via broadcast, otherwise via unicast. The assignment has to be accurate.

What is the vision of Providentia++?

The better sensors from vehicles and infrastructure are networked, the more seamlessly the traffic area can be recorded. This ultimately benefits not only motorists, but all other road users as well. Mobility expert Prof. Dr. Alois Knoll and his vision of the mobility of the future.

TO THE ARTICLE >

The better sensors from vehicles and infrastructure are networked, the more seamlessly the traffic area can be recorded. This ultimately benefits not only motorists, but all other road users as well. Mobility expert Prof. Dr. Alois Knoll and his vision of the mobility of the future.

TO THE ARTICLE >

PROVIDENTIA TEAM

Ziyad Aliyev, Project Manager, Elektrobit

Bernhard Blieninger, Scientist, fortiss

Christian Creß, Scientist, TU Munich

Markus Dillinger, Director 5G R&D, Huawei

Ralf Gräfe, Program Manager, Intel Labs Europe

Dr.-Ing. Gunnar Gräfe, CEO, 3D Mapping Solutions

Gereon Hinz, Visiting Scientist, STTech

Dr. Jens Honer, Senior Expert, Valeo

Xinyi Li, Scientist, TU Munich

Juri Kuhn, Scientist, fortiss

Venkatnarayanan Lakshminarasimhan, Scientist, TU Munich

Dr. Claus Lenz, CEO, Cognition Factory

Marie-Luise Neitz, Manager Third-Party Funded Projects, TU Munich

Andreas Schmitz, Public and Media Relations, TU Munich

Christoph Schöller, Scientist, fortiss

Leah Strand, Scientist, TU Munich

Thomas Strottner, Head of Business Development, brighter AI

Walter Zimmer, Scientist, TU Munich

load more

PARTNERS

 

(consortium leader)

load more
load more

ASSOCIATED PARTNERS

load more