chapman89
Founding Member
This is an interesting paper setting out the issues around adverse weather as the title implies and you might like to read the whole paper.
The Lidar section does not deal with VALEO unfortunately but does point out the issues in using other Lidar. I have extracted a few parts that I found of particular interest. The link is at the end of these extracts:
Perception and sensing for autonomous vehicles under adverse weather conditions: A survey
Author links open overlay panelYuxiaoZhangaAlexanderCarballobcdHantingYangaKazuyaTakedaacd
a
Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ward, Nagoya, 464-8601, Japan
b
Faculty of Engineering and Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu City, 501-1193, Japan
c
Institute of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
d
TierIV Inc., Nagoya University Open Innovation Center, 1-3, Mei-eki 1-chome, Nakamura-Ward, Nagoya, 450-6610, Japan
Received 29 April 2022, Revised 8 December 2022, Accepted 22 December 2022, Available online 9 January 2023, Version of Record 9 January 2023.
Abstract
Automated Driving Systems (ADS) open up a new domain for the automotive industry and offer new possibilities for future transportation with higher efficiency and comfortable experiences. However, perception and sensing for autonomous driving under adverse weather conditions have been the problem that keeps autonomous vehicles (AVs) from going to higher autonomy for a long time. This paper assesses the influences and challenges that weather brings to ADS sensors in a systematic way, and surveys the solutions against inclement weather conditions. State-of-the-art algorithms and deep learning methods on perception enhancement with regard to each kind of weather, weather status classification, and remote sensing are thoroughly reported. Sensor fusion solutions, weather conditions coverage in currently available datasets, simulators, and experimental facilities are categorized. Additionally, potential ADS sensor candidates and developing research directions such as V2X (Vehicle to Everything) technologies are discussed. By looking into all kinds of major weather problems, and reviewing both sensor and computer science solutions in recent years, this survey points out the main moving trends of adverse weather problems in perception and sensing, i.e., advanced sensor fusion and more sophisticated machine learning techniques; and also the limitations brought by emerging 1550 nm LiDARs. In general, this work contributes a holistic overview of the obstacles and directions of perception and sensing research development in terms of adverse weather conditions.
This first extract clearly presents the problem Mercedes Benz found while drying to develop ADAS and why it went looking for a different way of processing the amount of data being produced which led them to trial with Intel and Loihi before moving up to Brainchip the Artificial Intelligence experts and AKIDA technology for sensor fusion in real time at ultra low power.
Bijelic et al. (2020) from Mercedes-Benz AG present a large deep multimodal sensor fusion in unseen adverse weather. Their test vehicle is equipped with the following: a pair of stereo RGB cameras facing front; a near-infrared (NIR) gated camera whose adjustable delay capture of the flash laser pulse reduces the backscatter from particles in adverse weather (Bijelic et al., 2018b); a 77 GHz radar with 1∘" role="presentation" id="MathJax-Element-68-Frame">1∘ resolution; two Velodyne LiDARs namely HDL64 S3D and VLP32C; a far-infrared (FIR) thermal camera; a weather station with the ability to sense temperature, wind speed and direction, humidity, barometric pressure, and dew point; and a proprietary road-friction sensor. All the above are time-synchronized and ego-motion corrected with the help of the inertial measurement unit (IMU). Their fusion is entropy-steered, which means regions in the captures with low entropy can be attenuated, while entropy-rich regions can be amplified in the feature extraction. All the data collected by the exteroceptive sensors are concatenated for the entropy estimation process and the training was done by using clear weather only which demonstrated a strong adaptation. The fused detection performance was proven to be evidently improved than LiDAR or image-only under fog conditions. The blemish in this modality is that the amount of sensors exceeds the normal expectation of an ADS system. More sensors require more power supply and connection channels which is a burden to the vehicle itself and proprietary weather sensors are not exactly cost-friendly. Even though such an algorithm is still real-time processed, given the bulk amount of data from multiple sensors, the response and reaction time becomes something that should be worried about.
This next extract highlights the problem that AKIDA real time processing of Prophesee’s event based sensor overcomes and makes both essential in the automotive and robotic industries.
4.4.2. Reflections and shadows
Glare and strong light might not be removed easily, but reflections in similar conditions are relatively removable with the help of the absorption effect (Zheng et al., 2021b), reflection-free flash-only cues (Lei and Chen, 2021), and photo exposure correction (Afifi et al., 2021) techniques in the computer vision area. The principle follows reflection alignment and transmission recovery and it could relieve the ambiguity of the images well, especially in panoramic images which are commonly used in ADS (Hong et al., 2021). It is limited to recognizable reflections and fails in extremely strong lights where image content knowledge is not available. A special reflection is the mirage effect on hot roads. It has a weakness: the high-temperature area on the road is fixed and that fits the feature of a horizon which could be confusing (Young, 2015). Kumar et al. (2019)implemented horizon detection and depth estimation methods and managed to mark out a mirage in a video. The lack of mirage effects in datasets makes it hard to validate the real accuracy.
The same principle applies to shadow conditions as well, where the original image element is intact with a little low brightness in certain regions (Fu et al., 2021). Such image processing uses similar computer vision techniques as in previous paragraphs and can also take the route of first generating shadows and then removing them (Liu et al., 2021b). The Retinex algorithm can also be used for image enhancement in low-light conditions (Pham et al., 2020b).
This extract makes clear why it is absolutely critical that real time information is gathered by autonomous vehicles as to road surface conditions.
5.1.3. Road surface condition classification
Instant road surface condition changes are direct results of weather conditions, especially wet weather. The information on road conditions can sometimes be an alternative to weather classification. According to the research of Kordani et al. (2018) that at the speed of 80 km/h, the road friction coefficient of rainy, snowy, and icy road surface conditions are 0.4, 0.28 and 0.18 respectively, while average dry road friction coefficient is about 0.7. The dry or wet conditions can be determined in various ways besides road friction or environmental sensors (Shibata et al., 2020). Šabanovič et al. (2020) build a vision-based DNN to estimate the road friction coefficient because dry, slippery, slurry, and icy surfaces with decreasing friction can basically be identified as clear, rain, snow, and freezing weather correspondingly. Their algorithm detects not only the wet conditions but is able to classify the combination of wet conditions and pavement types as well. Panhuber et al. (2016) mounted a mono camera behind the windshield and observed the spray of water or dust caused by the leading car and the bird-view of the road features in the surroundings. They determine the road surface’s wet or dry condition by analyzing multiple regions of interest with different classifiers in order to merge into a robust result of 86% accuracy.
Road surface detection can also be performed in an uncommon way: audio. The sounds of vehicle speed, tire-surface interactions, and noise under different road conditions or different levels of wetness could be unique, so it is reasonable for Abdić et al. (2016) to train a deep learning network with over 780,000 bins of audio, including low speed when sounds are weak, even at 0 speed because it can detect the sound made by other driving-by vehicles. There are concerns about the vehicle type or tire type’s effects on the universality of such a method and the uncertain difficulty degree of the installation of sound collecting devices on vehicles.
This extract points to the convenient truths that:
2.4. Ultrasonic sensors
Ultrasonic sensors are commonly installed on the bumpers and all over the car body serving as parking assisting sensors and blindspot monitors (Carullo and Parvis, 2001). The principle of ultrasonic sensors is pretty similar to radar, both measuring the distance by calculating the travel time of the emitted electromagnetic wave, only ultrasonic operates at ultrasound band, around 40 to 70 kHz. In consequence, the detecting range of ultrasonic sensors normally does not exceed 11 m (Frenzel, 2021), and that restricts the application of ultrasonic sensors to close-range purposes such as backup parking. Efforts have been done to extend the effective range of ultrasonic and make it fit for long-range detecting (Kamemura et al., 2008). For example, Tesla’s “summon” feature uses ultrasonic to navigate through park space and garage doors (Tesla, 2021a).
Ultrasonic is among the sensors that are hardly considered in the evaluation of weather influences, but it does show some special features. The speed of sound traveling in air is affected by air pressure, humidity, and temperature (Varghese et al., 2015). The fluctuation of accuracy caused by this is a concern to autonomous driving unless enlisting the help of algorithms that can adjust the readings according to the ambient environment which generates extra costs. Nonetheless, ultrasonic does have its strengths, given the fact that its basic function is less affected by harsh weather compared to LiDAR and camera. The return signal of an ultrasonic wave does not get decreased due to the target’s dark color or low reflectivity, so it is more reliable in low visibility environments than cameras, such as high-glare or shaded areas beneath an overpass.
Additionally, the close proximity specialty of ultrasonic can be used to classify the condition of the road surface. Asphalt, grass, gravel, or dirt road can be distinguished from their back-scattered ultrasonic signals (Bystrov et al., 2016), so it is not hard to imagine that the snow, ice, or slurry on the road can be identified and help AV weather classification as well.
The following extract makes clear that the solutions known to these researchers are still to be found if ADAS or AV is to manage predictable
extreme light and weather conditions are to be managed within an acceptable power envelope in real time.
8. Conclusion
In this work, we surveyed the influence of adverse weather conditions on 5 major ADS sensors. Sensor fusion solutions were listed. The core solution to adverse weather problems is perception enhancement and various machine learning and image processing methods such as de-noising were thoroughly analyzed. Additional sensing enhancement methods including classification and localization were also among the discussions. A research tendency towards robust sensor fusions, sophisticated networks and computer vision models is concluded. Candidates for future ADS sensors such as FMCW LiDAR, HDR camera and hyperspectral camera were introduced. The limitations brought by the lack of relevant datasets and the difficulty of 1550 nm LiDAR were thoroughly explained. Finally, we believe that V2X and IoT have a brighter prospect in future weather research. This survey covered almost all types of common weather that pose negative effects on sensors’ perception and sensing abilities including rain, snow, fog, haze, strong light, and contamination, and listed out datasets, simulators, and experimental facilities that have weather support.
With the development of advanced test instruments and new technologies in LiDAR architectures, signs of progress have been largely made in the performance of perception and sensing in common wet weather. Rain and fog conditions seem to be getting better with the advanced development in computer vision in recent years, but still have some space for improvement on LiDAR. Snow, on the other hand, is still at the stage of dataset expansion and perception enhancement against snow has some more to dig in. Hence, point cloud processing under extreme snowy conditions, preferably with interaction scenarios either under controlled environments or on open roads is part of our future work. Two major sources of influence, strong light and contamination are still not rich in research and solutions. Hopefully, efforts made towards the robustness and reliability of sensors can carry adverse weather conditions research to the next level.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Funding
The author (Y.Z) would like to take this opportunity to thank the “Nagoya University Interdisciplinary Frontier Fellowship” supported by Nagoya University and JST, Japan, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2120, and JSPS KAKENHI, Japan Grant Number JP21H04892 and JP21K12073.
The authors thank Prof. Ming Ding from Nagoya University for his help. We would also like to extend our gratitude to Sensible4, the University of Michigan, Tier IV Inc., Ouster Inc., Perception Engine Inc., and Mr. Kang Yang for their support. In addition, our deepest thanks to VTT Technical Research Center of Finland, the University of Waterloo, Pan Asia Technical Automotive Center Co., Ltd, and the Civil Engineering Research Institute for Cold Region of Japan.
The Lidar section does not deal with VALEO unfortunately but does point out the issues in using other Lidar. I have extracted a few parts that I found of particular interest. The link is at the end of these extracts:
Perception and sensing for autonomous vehicles under adverse weather conditions: A survey
Author links open overlay panelYuxiaoZhangaAlexanderCarballobcdHantingYangaKazuyaTakedaacd
a
Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ward, Nagoya, 464-8601, Japan
b
Faculty of Engineering and Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu City, 501-1193, Japan
c
Institute of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
d
TierIV Inc., Nagoya University Open Innovation Center, 1-3, Mei-eki 1-chome, Nakamura-Ward, Nagoya, 450-6610, Japan
Received 29 April 2022, Revised 8 December 2022, Accepted 22 December 2022, Available online 9 January 2023, Version of Record 9 January 2023.
Abstract
Automated Driving Systems (ADS) open up a new domain for the automotive industry and offer new possibilities for future transportation with higher efficiency and comfortable experiences. However, perception and sensing for autonomous driving under adverse weather conditions have been the problem that keeps autonomous vehicles (AVs) from going to higher autonomy for a long time. This paper assesses the influences and challenges that weather brings to ADS sensors in a systematic way, and surveys the solutions against inclement weather conditions. State-of-the-art algorithms and deep learning methods on perception enhancement with regard to each kind of weather, weather status classification, and remote sensing are thoroughly reported. Sensor fusion solutions, weather conditions coverage in currently available datasets, simulators, and experimental facilities are categorized. Additionally, potential ADS sensor candidates and developing research directions such as V2X (Vehicle to Everything) technologies are discussed. By looking into all kinds of major weather problems, and reviewing both sensor and computer science solutions in recent years, this survey points out the main moving trends of adverse weather problems in perception and sensing, i.e., advanced sensor fusion and more sophisticated machine learning techniques; and also the limitations brought by emerging 1550 nm LiDARs. In general, this work contributes a holistic overview of the obstacles and directions of perception and sensing research development in terms of adverse weather conditions.
This first extract clearly presents the problem Mercedes Benz found while drying to develop ADAS and why it went looking for a different way of processing the amount of data being produced which led them to trial with Intel and Loihi before moving up to Brainchip the Artificial Intelligence experts and AKIDA technology for sensor fusion in real time at ultra low power.
Bijelic et al. (2020) from Mercedes-Benz AG present a large deep multimodal sensor fusion in unseen adverse weather. Their test vehicle is equipped with the following: a pair of stereo RGB cameras facing front; a near-infrared (NIR) gated camera whose adjustable delay capture of the flash laser pulse reduces the backscatter from particles in adverse weather (Bijelic et al., 2018b); a 77 GHz radar with 1∘" role="presentation" id="MathJax-Element-68-Frame">1∘ resolution; two Velodyne LiDARs namely HDL64 S3D and VLP32C; a far-infrared (FIR) thermal camera; a weather station with the ability to sense temperature, wind speed and direction, humidity, barometric pressure, and dew point; and a proprietary road-friction sensor. All the above are time-synchronized and ego-motion corrected with the help of the inertial measurement unit (IMU). Their fusion is entropy-steered, which means regions in the captures with low entropy can be attenuated, while entropy-rich regions can be amplified in the feature extraction. All the data collected by the exteroceptive sensors are concatenated for the entropy estimation process and the training was done by using clear weather only which demonstrated a strong adaptation. The fused detection performance was proven to be evidently improved than LiDAR or image-only under fog conditions. The blemish in this modality is that the amount of sensors exceeds the normal expectation of an ADS system. More sensors require more power supply and connection channels which is a burden to the vehicle itself and proprietary weather sensors are not exactly cost-friendly. Even though such an algorithm is still real-time processed, given the bulk amount of data from multiple sensors, the response and reaction time becomes something that should be worried about.
This next extract highlights the problem that AKIDA real time processing of Prophesee’s event based sensor overcomes and makes both essential in the automotive and robotic industries.
4.4.2. Reflections and shadows
Glare and strong light might not be removed easily, but reflections in similar conditions are relatively removable with the help of the absorption effect (Zheng et al., 2021b), reflection-free flash-only cues (Lei and Chen, 2021), and photo exposure correction (Afifi et al., 2021) techniques in the computer vision area. The principle follows reflection alignment and transmission recovery and it could relieve the ambiguity of the images well, especially in panoramic images which are commonly used in ADS (Hong et al., 2021). It is limited to recognizable reflections and fails in extremely strong lights where image content knowledge is not available. A special reflection is the mirage effect on hot roads. It has a weakness: the high-temperature area on the road is fixed and that fits the feature of a horizon which could be confusing (Young, 2015). Kumar et al. (2019)implemented horizon detection and depth estimation methods and managed to mark out a mirage in a video. The lack of mirage effects in datasets makes it hard to validate the real accuracy.
The same principle applies to shadow conditions as well, where the original image element is intact with a little low brightness in certain regions (Fu et al., 2021). Such image processing uses similar computer vision techniques as in previous paragraphs and can also take the route of first generating shadows and then removing them (Liu et al., 2021b). The Retinex algorithm can also be used for image enhancement in low-light conditions (Pham et al., 2020b).
This extract makes clear why it is absolutely critical that real time information is gathered by autonomous vehicles as to road surface conditions.
5.1.3. Road surface condition classification
Instant road surface condition changes are direct results of weather conditions, especially wet weather. The information on road conditions can sometimes be an alternative to weather classification. According to the research of Kordani et al. (2018) that at the speed of 80 km/h, the road friction coefficient of rainy, snowy, and icy road surface conditions are 0.4, 0.28 and 0.18 respectively, while average dry road friction coefficient is about 0.7. The dry or wet conditions can be determined in various ways besides road friction or environmental sensors (Shibata et al., 2020). Šabanovič et al. (2020) build a vision-based DNN to estimate the road friction coefficient because dry, slippery, slurry, and icy surfaces with decreasing friction can basically be identified as clear, rain, snow, and freezing weather correspondingly. Their algorithm detects not only the wet conditions but is able to classify the combination of wet conditions and pavement types as well. Panhuber et al. (2016) mounted a mono camera behind the windshield and observed the spray of water or dust caused by the leading car and the bird-view of the road features in the surroundings. They determine the road surface’s wet or dry condition by analyzing multiple regions of interest with different classifiers in order to merge into a robust result of 86% accuracy.
Road surface detection can also be performed in an uncommon way: audio. The sounds of vehicle speed, tire-surface interactions, and noise under different road conditions or different levels of wetness could be unique, so it is reasonable for Abdić et al. (2016) to train a deep learning network with over 780,000 bins of audio, including low speed when sounds are weak, even at 0 speed because it can detect the sound made by other driving-by vehicles. There are concerns about the vehicle type or tire type’s effects on the universality of such a method and the uncertain difficulty degree of the installation of sound collecting devices on vehicles.
This extract points to the convenient truths that:
- AKIDA technology boasts the capacity to process ultrasonic sensors in real time allowing sensor fusion,
- VALEO produces ultrasonic sensors and has a purpose built factory for their production along with the next gen Scala 3 Lidar, and
- Brainchip and VALEO have an ASX announced EAP relationship for ADAS and AV development and Brainchip is trusted by VALEO.
2.4. Ultrasonic sensors
Ultrasonic sensors are commonly installed on the bumpers and all over the car body serving as parking assisting sensors and blindspot monitors (Carullo and Parvis, 2001). The principle of ultrasonic sensors is pretty similar to radar, both measuring the distance by calculating the travel time of the emitted electromagnetic wave, only ultrasonic operates at ultrasound band, around 40 to 70 kHz. In consequence, the detecting range of ultrasonic sensors normally does not exceed 11 m (Frenzel, 2021), and that restricts the application of ultrasonic sensors to close-range purposes such as backup parking. Efforts have been done to extend the effective range of ultrasonic and make it fit for long-range detecting (Kamemura et al., 2008). For example, Tesla’s “summon” feature uses ultrasonic to navigate through park space and garage doors (Tesla, 2021a).
Ultrasonic is among the sensors that are hardly considered in the evaluation of weather influences, but it does show some special features. The speed of sound traveling in air is affected by air pressure, humidity, and temperature (Varghese et al., 2015). The fluctuation of accuracy caused by this is a concern to autonomous driving unless enlisting the help of algorithms that can adjust the readings according to the ambient environment which generates extra costs. Nonetheless, ultrasonic does have its strengths, given the fact that its basic function is less affected by harsh weather compared to LiDAR and camera. The return signal of an ultrasonic wave does not get decreased due to the target’s dark color or low reflectivity, so it is more reliable in low visibility environments than cameras, such as high-glare or shaded areas beneath an overpass.
Additionally, the close proximity specialty of ultrasonic can be used to classify the condition of the road surface. Asphalt, grass, gravel, or dirt road can be distinguished from their back-scattered ultrasonic signals (Bystrov et al., 2016), so it is not hard to imagine that the snow, ice, or slurry on the road can be identified and help AV weather classification as well.
The following extract makes clear that the solutions known to these researchers are still to be found if ADAS or AV is to manage predictable
extreme light and weather conditions are to be managed within an acceptable power envelope in real time.
8. Conclusion
In this work, we surveyed the influence of adverse weather conditions on 5 major ADS sensors. Sensor fusion solutions were listed. The core solution to adverse weather problems is perception enhancement and various machine learning and image processing methods such as de-noising were thoroughly analyzed. Additional sensing enhancement methods including classification and localization were also among the discussions. A research tendency towards robust sensor fusions, sophisticated networks and computer vision models is concluded. Candidates for future ADS sensors such as FMCW LiDAR, HDR camera and hyperspectral camera were introduced. The limitations brought by the lack of relevant datasets and the difficulty of 1550 nm LiDAR were thoroughly explained. Finally, we believe that V2X and IoT have a brighter prospect in future weather research. This survey covered almost all types of common weather that pose negative effects on sensors’ perception and sensing abilities including rain, snow, fog, haze, strong light, and contamination, and listed out datasets, simulators, and experimental facilities that have weather support.
With the development of advanced test instruments and new technologies in LiDAR architectures, signs of progress have been largely made in the performance of perception and sensing in common wet weather. Rain and fog conditions seem to be getting better with the advanced development in computer vision in recent years, but still have some space for improvement on LiDAR. Snow, on the other hand, is still at the stage of dataset expansion and perception enhancement against snow has some more to dig in. Hence, point cloud processing under extreme snowy conditions, preferably with interaction scenarios either under controlled environments or on open roads is part of our future work. Two major sources of influence, strong light and contamination are still not rich in research and solutions. Hopefully, efforts made towards the robustness and reliability of sensors can carry adverse weather conditions research to the next level.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Funding
The author (Y.Z) would like to take this opportunity to thank the “Nagoya University Interdisciplinary Frontier Fellowship” supported by Nagoya University and JST, Japan, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2120, and JSPS KAKENHI, Japan Grant Number JP21H04892 and JP21K12073.
The authors thank Prof. Ming Ding from Nagoya University for his help. We would also like to extend our gratitude to Sensible4, the University of Michigan, Tier IV Inc., Ouster Inc., Perception Engine Inc., and Mr. Kang Yang for their support. In addition, our deepest thanks to VTT Technical Research Center of Finland, the University of Waterloo, Pan Asia Technical Automotive Center Co., Ltd, and the Civil Engineering Research Institute for Cold Region of Japan.
Lidar manufacturer offensive: Luminar hires Tesla's former boss for autopilot and Nvidia manager
Electric cars at the CES trade fair: BYD with luxury brand, BMW Vision, Mercedes plans Ladenetz
