D
Deleted member 118
Guest
US-China chip war: America is winning
The rivalry is entering a new phase, with global players increasingly caught in the crossfire.
BRN Discussion Ongoing
- Thread starter TechGirl
- Start date
Hi Bravo,Howdy Brain Fam,
Hope everyone is having a great weekend. Let's hope I can make it even better!
I just watched the Cerence 25th Annual Needham Growth Conference which was filmed on the 10th Jan 2023. It's a 40 min approx video presentation that you have to sign up for to watch (full name and email address required for access). This link is here if you're interested in watching. https://wsw.com/webcast/needham
I'm itching to share a bit of information from the presentation because I believe there were numerous points raised throughout the presentation that indicate quite strongly the possible use of our technology in Cerence's embedded voice solution IMO.
For some background, Cerence is a global leader in conversational AI and they state that they are the only company in the world to offer the "complete stack" including conversational AI, audio, speech to text AI. Cerence state that every second newly defined SOP (start of production) car uses their technology, and they’re working with some very big names such as BYD, NIO, GM, Ford, Toyota, Volkswagen, Stellantis, Mercedes, BMW.
In the presentation they discussed how in November they held their second Analyst Day in which they outlined their new strategy called "Destination Next". They said that from a technology perspective this strategy or transition means they are going to be evolving from a voice only driver-centric solution via their Cerence Assistant or Co-pilate to a truly immersive in-cabin experience. Stefan Ortmanns (CEO Cerence) said early in the presentation something like "which means we're bringing in more features and applications beyond conversational AI, for example, wellness sensing, for example surrounding awareness, emotional AI or the interaction inside and outside the car with passengers and we have all these capabilities for creating a really immersive companion”. He also said something about the underlying strategy being based on 3 pillars, "scalable AI, teachable AI, and the immersive in-cabin experience", which has been bought about as a result of a "huge appetite for AI".
At about 6 mins Stefan Ortmanns says they have recently been shifting gear to bring in more proactive AI and he said something along these lines "What does it mean? So you bring everything you get together, so you have access to the sensors in the car, you an embedded solution, you have a cloud solution, and you also have this proactive AI, for example the road conditions or the weather conditions. And if you can bring everything together you have a personalised solution for the diver and also for the passengers and this is combines with what we call the (??? mighty ?? intelligence). And looking forward for the immersive experience, you need to bring in more together, it's not just about speech, it's about AI in general right so, with what I said wellness sensing, drunkenness detection, you know we're working on all this kind of cool stuff. We're working on emotional AI to have a better engagement with the passengers and also with the driver. And this is our future road map and we have vetted this with 50-60 OEM's across the globe and we did it together with a very well know consultancy firm."
At about 13 mins they describe how there will be very significant growth in fiscal years 23/24 because of the bookings they have won over the last 18 to 24 months that will go into production at the end of this year and very early in 2024 and a lot of them will have the higher tech stack that Stephan talked about.
At roughly 25 mins Stefan Ortmanns is asked how they compete with big tech like Alexa, Google, Apple, and how are they are co-exisiting because there are certain OEMS's using Alexa and certain ones using Cerence as well. In terms of what applications is Cerence providing Stephan replied stating something like "Alexa is a good example, so what you're seeing in the market is that OEM's are selecting us for their OEM branded solution and we are providing the wake word for interacting with Alexa, that's based on our core technology".
Now here comes the really good bit. At 29 mins the conversation turns to partnership statements, and they touch on NVDIA and whether Cerence view NVDIA as a competitor or partner (sounds familiar). This question was asked in relation to NVDIA having its own chauffeur product which enables some voice capabilities with its own hardware and software however Cerence has also been integrated into NVDIA's DRIVE platform. In describing this relationship, the Stefan Ortmanns says something like "So you're right. They have their own technology, but our technology stack is more advanced. And here we're talking about specifically Mercedes where they're positioned with their hardware and with our solution. There's also another semi-conductor big player, Qualcomm namely now they are working with Volkswagen group and they're also using our technology. So we're very flexible and open with our partners".
Following on form that they discuss how Cerence is also involved in the language processing for BMW which has to be "seamlessly integrated" with "very low latency".
So, a couple of points I wanted to throw in to emphasise why I'm thinking all of this so strongly indicates the use of BrainChip's technology being a part of Cerence's stack.
Anyway, naturally I would love to hear what @Fact Finder has to say about this. As we all know he is the world's utmost guru in being able to sort the chaff from the wheat and always stands at the ready to pour cold water on any outlandish dot joining attempts when the facts don't stack up.
- Cerence presented Mercedes as the premium example in which to demonstrate how advanced their voice technology is in comparison to NVDIA's. Since this presentation is only a few days old, I don't think they'd be referring to Mercedes old voice technology but rather the new advanced technology developed for the Vision EQXX. And I don't think Cerence would be referring to Mercedes at all if they weren't still working with them.
- This is after Mercedes worked with BrainChip on the “wake word” detection for the Vision EQXX which made it 5-10 times more efficient. So, it only seems logical if Cerence's core technology is to provide the wake word that they should incorporate BrainChip’s technology to make the wake word detection 5-10 times faster.
- In November 2022 Nils Shanz, who was responsible for user interaction and voice control at Mercedes and who also worked on the Vision EQXX voice control system was appointed Chief Product Officer at Cerence.
- Previous post in which Cerence describe their technology as "self-learning",etc #6,807
- Previous post in which Cerence technology is described as working with an internet connection #35,234 and #31,305
- I’m no engineer but I would have thought the new emotion detection AI and contextual awareness AI that are connected to the car’s sensors must be embedded into Cerence’s technology for it all to work seamlessly.
Of course, these are only the conclusions I have arrived at after watching the presentation and I would love to hear what everyone else’s impression are. Needless to say, I hope I'm right.
B
As you know, Cerence has been on our radar and I had filed them under competitors under the "friend or Foe" principle, but the truth is that that they seem to be agnostic as far as NNs are concerned, simply listing "artificial intelligence" in a grocery list of functions.
US2022415318A1 VOICE ASSISTANT ACTIVATION SYSTEM WITH CONTEXT DETERMINATION BASED ON MULTIMODAL DATA
A vehicle system for classifying spoken utterance within a vehicle cabin as one of system-directed and non-system directed may include at least one microphone to detect at least one acoustic utterance from at least one occupant of the vehicle, at least one camera to detect occupant data indicative of occupant behavior within the vehicle corresponding to the acoustic utterance, and a processor programmed to receive the acoustic utterance, receive the occupant data, determine whether the occupant data is indicative of a vehicle feature, classify the acoustic utterance as a system-directed utterance in response to the occupant data being indicative of a vehicle feature, and process the acoustic utterance.
[0016] The vehicle 104 may be configured to include various types of components, processors, and memory, and may communicate with a communication network 110 . The communication network 110 may be referred to as a “cloud” and may involve data transfer via wide area and/or local area networks, such as the Internet, Global Positioning System (GPS), cellular networks, Wi-Fi, Bluetooth, etc. The communication network 110 may provide for communication between the vehicle 104 and an external or remote server 112 and/or database 114 , as well as other external applications, systems, vehicles, etc. This communication network 110 may provide navigation, music or other audio, program content, marketing content, internet access, speech recognition, cognitive computing, artificial intelligence, to the vehicle 104 .
Mt09
Regular
BrainChip on LinkedIn: Edge Impulse and BrainChip Partner to Further AI Development with Support…
Edge Impulse announced today official support for BrainChip's neural processor AI IP, making BrainChip the first strategic IP partner on the Edge Impulse…
CEO of Inivation wishes us all the best?
Big Rob T gave it a like.
Soz if posted b4!
Attachments
Not quite techno but why haven’t I ever seen this before? By AkidaEvening Chippers,
Breaking news...
World first, Pioneer DJ mixing table utilising Brainchips Akida neuromorphic chip on the International Space Station.
Personally can't imagine having to wash the external windows , whilst attached via umbilical, without some groovy tunes.
.
* With any luck, may pull Fact Finder back, to give me a dressing down.
Seemed to work last time.
All in good humour.
ARi - Matasin, Live Series, Ep.003 ( Melodic Techno Progressive House Mix) 7th Jan 2023.
If a savvy individual could post link, Thankyou in advance.
This may be our only hope of retrieving Fact Finder.
Cheers for all the great finds and posts today.
Regards,
Esq.
thelittleshort
Top Bloke
@Diogenese what do you reckon? Could we be also involved in the SiFive - Intel ‘Horse Creek’ collaboration? Is the ChatGPT description below legit?
Rise from the ashes
Regular
Chatgpt also has a cutoff date that it scraped info from whatever source it gets its information from.
2021 not sure of the month though.
Just throwing that out here in case it helps anyone somehow.
SiFive Horse Creek was showcased in October 2022, so they would have been seeing each other for some time before that.
https://www.cnx-software.com/2022/1...rm-sifive-p550-risc-v-cpu-8gb-ddr5-pcie-gen5/
There is no mention of NNs or AI accelerators in this article from 20221010:
https://www.cnx-software.com/2022/1...rm-sifive-p550-risc-v-cpu-8gb-ddr5-pcie-gen5/
Horse Creek platform specifications:
- CPU – SiFive P500 quad-core RISC-V processor @ up to 2.2 GHz with a 13-stage, 3-issue, out-of-order (OoO) pipeline, private L2 cache, and common L3 cache
- Memory – DDR5-5600 interface
- PCIe – PCIe Gen5 through Intel’s PCIe PHY with 8 lanes, Synopsys PCIe Root Hub controller
- Other peripheral interfaces – I3C, Quad and Octal SPI, UART, peripheral DMA
- Package – 19×19 FBGA
- Process – Intel 4 technology
Our affair with SiFive goes back to April 2022
https://brainchip.com/brainchip-sifive-partner-deploy-ai-ml-at-edge/
but we did not start going out with Intel until December 2022.
| https://brainchip.com/brainchip-joins-intel-foundry-services-to-advance-neuromorphic-ai-at-the-edge/ |
There is nothing to indicate that Horse Creek uses Akida. [Now I'm talking like ChatGPT (where I get all my answers from)].
As for the GPT response, it is couched in broad generalizations without any real detail, almost like it was under NDA.
Last edited:
Rise from the ashes
Regular
Founder of Qualcomm and a couple of other well knowns.
Short video but I like to know what these guys are like in a relaxed setting.
"When the money hits the table that's when you find out the real character of people" so very true
Last edited:
Deadpool
Did someone say KFC
Evening Chippers,
Breaking news...
World first, Pioneer DJ mixing table utilising Brainchips Akida neuromorphic chip on the International Space Station.
Personally can't imagine having to wash the external windows , whilst attached via umbilical, without some groovy tunes.
* With any luck, may pull Fact Finder back, to give me a dressing down.
Seemed to work last time.
All in good humour.
ARi - Matasin, Live Series, Ep.003 ( Melodic Techno Progressive House Mix) 7th Jan 2023.
If a savvy individual could post link, Thankyou in advance.
This may be our only hope of retrieving Fact Finder.
Cheers for all the great finds and posts today.
Regards,
Esq.
thelittleshort
Top Bloke
Maybe there was a bit of flirting before we starting dating officially
... but we were always home before 11 o'clock.
Rise from the ashes
Regular
There's a comment in that video that goes something like " big shots they think they built the business uh their customers built the business"
So all those who whinge about NDA's should really have a think about that.
Hey @Esq.111 not bad, not bad at all bit more tempo in that one.
Terroni2105
Founding Member
Hi Dio, are you saying, opposed to your initial thought, it is possible that Cerence is a friend instead of a foe? Because I was wondering @Bravo if the large consulting firm that Cerence used to vett the OEMs might have been our recent podcast mate Accenture 🫢
Last edited:
Rise from the ashes
Regular
Above tweet had nothing to do with your post but like Akida I'm trying to be more efficient.
thelittleshort
Top Bloke
Observation
Media Alert: BrainChip Demonstrates Neuromorphic AI Capabilities of its Akida Platform at tinyML Summit 2022
BrainChip exhibiting the latest capabilities of its Akida™ neuromorphic technology in person at tinyML Summit 2022
brainchip.com
BrainChip and NVISO Partner on Human Behavioral Analytics in Automotive and Edge AI Devices
BrainChip and NVISO Partner on Human Behavioral Analytics in Automotive and Edge AI Devices
www.businesswire.com
Hi Terroni,
The fact that they do not seem to have any in-house NN tech leaves open the possibility that they may have or could adopt Akida.
from @Bravo 's post:
"At about 6 mins Stefan Ortmanns says they have recently been shifting gear to bring in more proactive AI and he said something along these lines "What does it mean? So you bring everything you get together, so you have access to the sensors in the car, you an embedded solution, you have a cloud solution, and you also have this proactive AI, for example the road conditions or the weather conditions. And if you can bring everything together you have a personalised solution for the diver and also for the passengers and this is combines with what we call the (??? mighty ?? intelligence). And looking forward for the immersive experience, you need to bring in more together, it's not just about speech, it's about AI in general right so, with what I said wellness sensing, drunkenness detection, you know we're working on all this kind of cool stuff. We're working on emotional AI to have a better engagement with the passengers and also with the driver. And this is our future road map and we have vetted this with 50-60 OEM's across the globe and we did it together with a very well know consultancy firm."
...
At roughly 25 mins Stefan Ortmanns is asked how they compete with big tech like Alexa, Google, Apple, and how are they are co-exisiting because there are certain OEMS's using Alexa and certain ones using Cerence as well. In terms of what applications is Cerence providing Stephan replied stating something like "Alexa is a good example, so what you're seeing in the market is that OEM's are selecting us for their OEM branded solution and we are providing the wake word for interacting with Alexa, that's based on our core technology".
Now here comes the really good bit. At 29 mins the conversation turns to partnership statements, and they touch on NVDIA and whether Cerence view NVDIA as a competitor or partner (sounds familiar). This question was asked in relation to NVDIA having its own chauffeur product which enables some voice capabilities with its own hardware and software however Cerence has also been integrated into NVDIA's DRIVE platform. In describing this relationship, the Stefan Ortmanns says something like "So you're right. They have their own technology, but our technology stack is more advanced. And here we're talking about specifically Mercedes where they're positioned with their hardware and with our solution. There's also another semi-conductor big player, Qualcomm namely now they are working with Volkswagen group and they're also using our technology. So we're very flexible and open with our partners".
Following on form that they discuss how Cerence is also involved in the language processing for BMW which has to be "seamlessly integrated" with "very low latency"."
The thing is that Cerence do not tell us much about their processor. The system is a processor with camera and microphone, programmed to determine whether the speaker is talking to the vehicle. Their product is really software.
There is no real detail about how the camera and microphone signals are classified - a job for which Akida is ideal. So it is entirely possible that Akida is used by Cerence, particularly, as Bravo points out, in view of the Mercedes connexion.
US2022415318A1 VOICE ASSISTANT ACTIVATION SYSTEM WITH CONTEXT DETERMINATION BASED ON MULTIMODAL DATA
A vehicle system for classifying spoken utterance within a vehicle cabin as one of system-directed and non-system directed may include at least one microphone to detect at least one acoustic utterance from at least one occupant of the vehicle, at least one camera to detect occupant data indicative of occupant behavior within the vehicle corresponding to the acoustic utterance, and a processor programmed to receive the acoustic utterance, receive the occupant data, determine whether the occupant data is indicative of a vehicle feature, classify the acoustic utterance as a system-directed utterance in response to the occupant data being indicative of a vehicle feature, and process the acoustic utterance.
a processor programmed to receive the acoustic utterance, receive the occupant data, determine whether the occupant data is indicative of a vehicle feature, classify the acoustic utterance as a system-directed utterance in response to the occupant data being indicative of a vehicle feature, and process the acoustic utterance.
[0046] At block 410 , the processor 106 may process the utterance to determine the content of the utterance, e.g., the command or phrase spoken by the occupant. The processing may also determine other characteristics of the utterance, such as the tone, direction, occupant position within the vehicle, the specific occupant based on voice recognition, etc. Signal processing techniques including filtering, noise cancelation, amplification, beamforming, to name a few, may be implemented to process the utterance. In some instances, the tone of the utterance alone may be used to classify the utterance as system-directed or non-system-directed.
Terroni2105
Founding Member
Thanks Dio
much appreciatedTime to bury the "hatchet" FF. I've ( we've all ) missed you immensely ( immensurably even, I think that's a word ), so time to turn the other cheek, and get back into what you were born to do, i.e. ( Inspire with courage ) and project your viewpoints again and again, and with Gusto-oooo. Stuff is happening, ( right now IMO ) and we need your tick of approval and sharp appraisal qualities, and pronto, kind Sir.Evening Chippers,
Breaking news...
World first, Pioneer DJ mixing table utilising Brainchips Akida neuromorphic chip on the International Space Station.
Personally can't imagine having to wash the external windows , whilst attached via umbilical, without some groovy tunes.
* With any luck, may pull Fact Finder back, to give me a dressing down.
Seemed to work last time.
All in good humour.
ARi - Matasin, Live Series, Ep.003 ( Melodic Techno Progressive House Mix) 7th Jan 2023.
If a savvy individual could post link, Thankyou in advance.
This may be our only hope of retrieving Fact Finder.
Cheers for all the great finds and posts today.
Regards,
Esq.
The couch, can wait, and there's just toooo much crap on the goggle box IMHO, NB... I'm an expert in gogglebox wasted time, I can assure you, it is mind numbing, to say the least. So please "get back on board"...You just might miss all the fun.
And don't forget >>>>>>>
Akida Ballista >>>>>> Just a whole lotta good FACTS emerging - waiting for your perusal and comment <<<<<
hotty...
chapman89
Founding Member
This is an interesting paper setting out the issues around adverse weather as the title implies and you might like to read the whole paper.
The Lidar section does not deal with VALEO unfortunately but does point out the issues in using other Lidar. I have extracted a few parts that I found of particular interest. The link is at the end of these extracts:
Perception and sensing for autonomous vehicles under adverse weather conditions: A survey
Author links open overlay panelYuxiaoZhangaAlexanderCarballobcdHantingYangaKazuyaTakedaacd
a
Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ward, Nagoya, 464-8601, Japan
b
Faculty of Engineering and Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu City, 501-1193, Japan
c
Institute of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
d
TierIV Inc., Nagoya University Open Innovation Center, 1-3, Mei-eki 1-chome, Nakamura-Ward, Nagoya, 450-6610, Japan
Received 29 April 2022, Revised 8 December 2022, Accepted 22 December 2022, Available online 9 January 2023, Version of Record 9 January 2023.
Abstract
Automated Driving Systems (ADS) open up a new domain for the automotive industry and offer new possibilities for future transportation with higher efficiency and comfortable experiences. However, perception and sensing for autonomous driving under adverse weather conditions have been the problem that keeps autonomous vehicles (AVs) from going to higher autonomy for a long time. This paper assesses the influences and challenges that weather brings to ADS sensors in a systematic way, and surveys the solutions against inclement weather conditions. State-of-the-art algorithms and deep learning methods on perception enhancement with regard to each kind of weather, weather status classification, and remote sensing are thoroughly reported. Sensor fusion solutions, weather conditions coverage in currently available datasets, simulators, and experimental facilities are categorized. Additionally, potential ADS sensor candidates and developing research directions such as V2X (Vehicle to Everything) technologies are discussed. By looking into all kinds of major weather problems, and reviewing both sensor and computer science solutions in recent years, this survey points out the main moving trends of adverse weather problems in perception and sensing, i.e., advanced sensor fusion and more sophisticated machine learning techniques; and also the limitations brought by emerging 1550 nm LiDARs. In general, this work contributes a holistic overview of the obstacles and directions of perception and sensing research development in terms of adverse weather conditions.
This first extract clearly presents the problem Mercedes Benz found while drying to develop ADAS and why it went looking for a different way of processing the amount of data being produced which led them to trial with Intel and Loihi before moving up to Brainchip the Artificial Intelligence experts and AKIDA technology for sensor fusion in real time at ultra low power.
Bijelic et al. (2020) from Mercedes-Benz AG present a large deep multimodal sensor fusion in unseen adverse weather. Their test vehicle is equipped with the following: a pair of stereo RGB cameras facing front; a near-infrared (NIR) gated camera whose adjustable delay capture of the flash laser pulse reduces the backscatter from particles in adverse weather (Bijelic et al., 2018b); a 77 GHz radar with 1∘" role="presentation" id="MathJax-Element-68-Frame">1∘ resolution; two Velodyne LiDARs namely HDL64 S3D and VLP32C; a far-infrared (FIR) thermal camera; a weather station with the ability to sense temperature, wind speed and direction, humidity, barometric pressure, and dew point; and a proprietary road-friction sensor. All the above are time-synchronized and ego-motion corrected with the help of the inertial measurement unit (IMU). Their fusion is entropy-steered, which means regions in the captures with low entropy can be attenuated, while entropy-rich regions can be amplified in the feature extraction. All the data collected by the exteroceptive sensors are concatenated for the entropy estimation process and the training was done by using clear weather only which demonstrated a strong adaptation. The fused detection performance was proven to be evidently improved than LiDAR or image-only under fog conditions. The blemish in this modality is that the amount of sensors exceeds the normal expectation of an ADS system. More sensors require more power supply and connection channels which is a burden to the vehicle itself and proprietary weather sensors are not exactly cost-friendly. Even though such an algorithm is still real-time processed, given the bulk amount of data from multiple sensors, the response and reaction time becomes something that should be worried about.
This next extract highlights the problem that AKIDA real time processing of Prophesee’s event based sensor overcomes and makes both essential in the automotive and robotic industries.
4.4.2. Reflections and shadows
Glare and strong light might not be removed easily, but reflections in similar conditions are relatively removable with the help of the absorption effect (Zheng et al., 2021b), reflection-free flash-only cues (Lei and Chen, 2021), and photo exposure correction (Afifi et al., 2021) techniques in the computer vision area. The principle follows reflection alignment and transmission recovery and it could relieve the ambiguity of the images well, especially in panoramic images which are commonly used in ADS (Hong et al., 2021). It is limited to recognizable reflections and fails in extremely strong lights where image content knowledge is not available. A special reflection is the mirage effect on hot roads. It has a weakness: the high-temperature area on the road is fixed and that fits the feature of a horizon which could be confusing (Young, 2015). Kumar et al. (2019)implemented horizon detection and depth estimation methods and managed to mark out a mirage in a video. The lack of mirage effects in datasets makes it hard to validate the real accuracy.
The same principle applies to shadow conditions as well, where the original image element is intact with a little low brightness in certain regions (Fu et al., 2021). Such image processing uses similar computer vision techniques as in previous paragraphs and can also take the route of first generating shadows and then removing them (Liu et al., 2021b). The Retinex algorithm can also be used for image enhancement in low-light conditions (Pham et al., 2020b).
This extract makes clear why it is absolutely critical that real time information is gathered by autonomous vehicles as to road surface conditions.
5.1.3. Road surface condition classification
Instant road surface condition changes are direct results of weather conditions, especially wet weather. The information on road conditions can sometimes be an alternative to weather classification. According to the research of Kordani et al. (2018) that at the speed of 80 km/h, the road friction coefficient of rainy, snowy, and icy road surface conditions are 0.4, 0.28 and 0.18 respectively, while average dry road friction coefficient is about 0.7. The dry or wet conditions can be determined in various ways besides road friction or environmental sensors (Shibata et al., 2020). Šabanovič et al. (2020) build a vision-based DNN to estimate the road friction coefficient because dry, slippery, slurry, and icy surfaces with decreasing friction can basically be identified as clear, rain, snow, and freezing weather correspondingly. Their algorithm detects not only the wet conditions but is able to classify the combination of wet conditions and pavement types as well. Panhuber et al. (2016) mounted a mono camera behind the windshield and observed the spray of water or dust caused by the leading car and the bird-view of the road features in the surroundings. They determine the road surface’s wet or dry condition by analyzing multiple regions of interest with different classifiers in order to merge into a robust result of 86% accuracy.
Road surface detection can also be performed in an uncommon way: audio. The sounds of vehicle speed, tire-surface interactions, and noise under different road conditions or different levels of wetness could be unique, so it is reasonable for Abdić et al. (2016) to train a deep learning network with over 780,000 bins of audio, including low speed when sounds are weak, even at 0 speed because it can detect the sound made by other driving-by vehicles. There are concerns about the vehicle type or tire type’s effects on the universality of such a method and the uncertain difficulty degree of the installation of sound collecting devices on vehicles.
This extract points to the convenient truths that:
2.4. Ultrasonic sensors
Ultrasonic sensors are commonly installed on the bumpers and all over the car body serving as parking assisting sensors and blindspot monitors (Carullo and Parvis, 2001). The principle of ultrasonic sensors is pretty similar to radar, both measuring the distance by calculating the travel time of the emitted electromagnetic wave, only ultrasonic operates at ultrasound band, around 40 to 70 kHz. In consequence, the detecting range of ultrasonic sensors normally does not exceed 11 m (Frenzel, 2021), and that restricts the application of ultrasonic sensors to close-range purposes such as backup parking. Efforts have been done to extend the effective range of ultrasonic and make it fit for long-range detecting (Kamemura et al., 2008). For example, Tesla’s “summon” feature uses ultrasonic to navigate through park space and garage doors (Tesla, 2021a).
Ultrasonic is among the sensors that are hardly considered in the evaluation of weather influences, but it does show some special features. The speed of sound traveling in air is affected by air pressure, humidity, and temperature (Varghese et al., 2015). The fluctuation of accuracy caused by this is a concern to autonomous driving unless enlisting the help of algorithms that can adjust the readings according to the ambient environment which generates extra costs. Nonetheless, ultrasonic does have its strengths, given the fact that its basic function is less affected by harsh weather compared to LiDAR and camera. The return signal of an ultrasonic wave does not get decreased due to the target’s dark color or low reflectivity, so it is more reliable in low visibility environments than cameras, such as high-glare or shaded areas beneath an overpass.
Additionally, the close proximity specialty of ultrasonic can be used to classify the condition of the road surface. Asphalt, grass, gravel, or dirt road can be distinguished from their back-scattered ultrasonic signals (Bystrov et al., 2016), so it is not hard to imagine that the snow, ice, or slurry on the road can be identified and help AV weather classification as well.
The following extract makes clear that the solutions known to these researchers are still to be found if ADAS or AV is to manage predictable
extreme light and weather conditions are to be managed within an acceptable power envelope in real time.
8. Conclusion
In this work, we surveyed the influence of adverse weather conditions on 5 major ADS sensors. Sensor fusion solutions were listed. The core solution to adverse weather problems is perception enhancement and various machine learning and image processing methods such as de-noising were thoroughly analyzed. Additional sensing enhancement methods including classification and localization were also among the discussions. A research tendency towards robust sensor fusions, sophisticated networks and computer vision models is concluded. Candidates for future ADS sensors such as FMCW LiDAR, HDR camera and hyperspectral camera were introduced. The limitations brought by the lack of relevant datasets and the difficulty of 1550 nm LiDAR were thoroughly explained. Finally, we believe that V2X and IoT have a brighter prospect in future weather research. This survey covered almost all types of common weather that pose negative effects on sensors’ perception and sensing abilities including rain, snow, fog, haze, strong light, and contamination, and listed out datasets, simulators, and experimental facilities that have weather support.
With the development of advanced test instruments and new technologies in LiDAR architectures, signs of progress have been largely made in the performance of perception and sensing in common wet weather. Rain and fog conditions seem to be getting better with the advanced development in computer vision in recent years, but still have some space for improvement on LiDAR. Snow, on the other hand, is still at the stage of dataset expansion and perception enhancement against snow has some more to dig in. Hence, point cloud processing under extreme snowy conditions, preferably with interaction scenarios either under controlled environments or on open roads is part of our future work. Two major sources of influence, strong light and contamination are still not rich in research and solutions. Hopefully, efforts made towards the robustness and reliability of sensors can carry adverse weather conditions research to the next level.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Funding
The author (Y.Z) would like to take this opportunity to thank the “Nagoya University Interdisciplinary Frontier Fellowship” supported by Nagoya University and JST, Japan, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2120, and JSPS KAKENHI, Japan Grant Number JP21H04892 and JP21K12073.
The authors thank Prof. Ming Ding from Nagoya University for his help. We would also like to extend our gratitude to Sensible4, the University of Michigan, Tier IV Inc., Ouster Inc., Perception Engine Inc., and Mr. Kang Yang for their support. In addition, our deepest thanks to VTT Technical Research Center of Finland, the University of Waterloo, Pan Asia Technical Automotive Center Co., Ltd, and the Civil Engineering Research Institute for Cold Region of Japan.
The Lidar section does not deal with VALEO unfortunately but does point out the issues in using other Lidar. I have extracted a few parts that I found of particular interest. The link is at the end of these extracts:
Perception and sensing for autonomous vehicles under adverse weather conditions: A survey
Author links open overlay panelYuxiaoZhangaAlexanderCarballobcdHantingYangaKazuyaTakedaacd
a
Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ward, Nagoya, 464-8601, Japan
b
Faculty of Engineering and Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu City, 501-1193, Japan
c
Institute of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
d
TierIV Inc., Nagoya University Open Innovation Center, 1-3, Mei-eki 1-chome, Nakamura-Ward, Nagoya, 450-6610, Japan
Received 29 April 2022, Revised 8 December 2022, Accepted 22 December 2022, Available online 9 January 2023, Version of Record 9 January 2023.
Abstract
Automated Driving Systems (ADS) open up a new domain for the automotive industry and offer new possibilities for future transportation with higher efficiency and comfortable experiences. However, perception and sensing for autonomous driving under adverse weather conditions have been the problem that keeps autonomous vehicles (AVs) from going to higher autonomy for a long time. This paper assesses the influences and challenges that weather brings to ADS sensors in a systematic way, and surveys the solutions against inclement weather conditions. State-of-the-art algorithms and deep learning methods on perception enhancement with regard to each kind of weather, weather status classification, and remote sensing are thoroughly reported. Sensor fusion solutions, weather conditions coverage in currently available datasets, simulators, and experimental facilities are categorized. Additionally, potential ADS sensor candidates and developing research directions such as V2X (Vehicle to Everything) technologies are discussed. By looking into all kinds of major weather problems, and reviewing both sensor and computer science solutions in recent years, this survey points out the main moving trends of adverse weather problems in perception and sensing, i.e., advanced sensor fusion and more sophisticated machine learning techniques; and also the limitations brought by emerging 1550 nm LiDARs. In general, this work contributes a holistic overview of the obstacles and directions of perception and sensing research development in terms of adverse weather conditions.
This first extract clearly presents the problem Mercedes Benz found while drying to develop ADAS and why it went looking for a different way of processing the amount of data being produced which led them to trial with Intel and Loihi before moving up to Brainchip the Artificial Intelligence experts and AKIDA technology for sensor fusion in real time at ultra low power.
Bijelic et al. (2020) from Mercedes-Benz AG present a large deep multimodal sensor fusion in unseen adverse weather. Their test vehicle is equipped with the following: a pair of stereo RGB cameras facing front; a near-infrared (NIR) gated camera whose adjustable delay capture of the flash laser pulse reduces the backscatter from particles in adverse weather (Bijelic et al., 2018b); a 77 GHz radar with 1∘" role="presentation" id="MathJax-Element-68-Frame">1∘ resolution; two Velodyne LiDARs namely HDL64 S3D and VLP32C; a far-infrared (FIR) thermal camera; a weather station with the ability to sense temperature, wind speed and direction, humidity, barometric pressure, and dew point; and a proprietary road-friction sensor. All the above are time-synchronized and ego-motion corrected with the help of the inertial measurement unit (IMU). Their fusion is entropy-steered, which means regions in the captures with low entropy can be attenuated, while entropy-rich regions can be amplified in the feature extraction. All the data collected by the exteroceptive sensors are concatenated for the entropy estimation process and the training was done by using clear weather only which demonstrated a strong adaptation. The fused detection performance was proven to be evidently improved than LiDAR or image-only under fog conditions. The blemish in this modality is that the amount of sensors exceeds the normal expectation of an ADS system. More sensors require more power supply and connection channels which is a burden to the vehicle itself and proprietary weather sensors are not exactly cost-friendly. Even though such an algorithm is still real-time processed, given the bulk amount of data from multiple sensors, the response and reaction time becomes something that should be worried about.
This next extract highlights the problem that AKIDA real time processing of Prophesee’s event based sensor overcomes and makes both essential in the automotive and robotic industries.
4.4.2. Reflections and shadows
Glare and strong light might not be removed easily, but reflections in similar conditions are relatively removable with the help of the absorption effect (Zheng et al., 2021b), reflection-free flash-only cues (Lei and Chen, 2021), and photo exposure correction (Afifi et al., 2021) techniques in the computer vision area. The principle follows reflection alignment and transmission recovery and it could relieve the ambiguity of the images well, especially in panoramic images which are commonly used in ADS (Hong et al., 2021). It is limited to recognizable reflections and fails in extremely strong lights where image content knowledge is not available. A special reflection is the mirage effect on hot roads. It has a weakness: the high-temperature area on the road is fixed and that fits the feature of a horizon which could be confusing (Young, 2015). Kumar et al. (2019)implemented horizon detection and depth estimation methods and managed to mark out a mirage in a video. The lack of mirage effects in datasets makes it hard to validate the real accuracy.
The same principle applies to shadow conditions as well, where the original image element is intact with a little low brightness in certain regions (Fu et al., 2021). Such image processing uses similar computer vision techniques as in previous paragraphs and can also take the route of first generating shadows and then removing them (Liu et al., 2021b). The Retinex algorithm can also be used for image enhancement in low-light conditions (Pham et al., 2020b).
This extract makes clear why it is absolutely critical that real time information is gathered by autonomous vehicles as to road surface conditions.
5.1.3. Road surface condition classification
Instant road surface condition changes are direct results of weather conditions, especially wet weather. The information on road conditions can sometimes be an alternative to weather classification. According to the research of Kordani et al. (2018) that at the speed of 80 km/h, the road friction coefficient of rainy, snowy, and icy road surface conditions are 0.4, 0.28 and 0.18 respectively, while average dry road friction coefficient is about 0.7. The dry or wet conditions can be determined in various ways besides road friction or environmental sensors (Shibata et al., 2020). Šabanovič et al. (2020) build a vision-based DNN to estimate the road friction coefficient because dry, slippery, slurry, and icy surfaces with decreasing friction can basically be identified as clear, rain, snow, and freezing weather correspondingly. Their algorithm detects not only the wet conditions but is able to classify the combination of wet conditions and pavement types as well. Panhuber et al. (2016) mounted a mono camera behind the windshield and observed the spray of water or dust caused by the leading car and the bird-view of the road features in the surroundings. They determine the road surface’s wet or dry condition by analyzing multiple regions of interest with different classifiers in order to merge into a robust result of 86% accuracy.
Road surface detection can also be performed in an uncommon way: audio. The sounds of vehicle speed, tire-surface interactions, and noise under different road conditions or different levels of wetness could be unique, so it is reasonable for Abdić et al. (2016) to train a deep learning network with over 780,000 bins of audio, including low speed when sounds are weak, even at 0 speed because it can detect the sound made by other driving-by vehicles. There are concerns about the vehicle type or tire type’s effects on the universality of such a method and the uncertain difficulty degree of the installation of sound collecting devices on vehicles.
This extract points to the convenient truths that:
- AKIDA technology boasts the capacity to process ultrasonic sensors in real time allowing sensor fusion,
- VALEO produces ultrasonic sensors and has a purpose built factory for their production along with the next gen Scala 3 Lidar, and
- Brainchip and VALEO have an ASX announced EAP relationship for ADAS and AV development and Brainchip is trusted by VALEO.
2.4. Ultrasonic sensors
Ultrasonic sensors are commonly installed on the bumpers and all over the car body serving as parking assisting sensors and blindspot monitors (Carullo and Parvis, 2001). The principle of ultrasonic sensors is pretty similar to radar, both measuring the distance by calculating the travel time of the emitted electromagnetic wave, only ultrasonic operates at ultrasound band, around 40 to 70 kHz. In consequence, the detecting range of ultrasonic sensors normally does not exceed 11 m (Frenzel, 2021), and that restricts the application of ultrasonic sensors to close-range purposes such as backup parking. Efforts have been done to extend the effective range of ultrasonic and make it fit for long-range detecting (Kamemura et al., 2008). For example, Tesla’s “summon” feature uses ultrasonic to navigate through park space and garage doors (Tesla, 2021a).
Ultrasonic is among the sensors that are hardly considered in the evaluation of weather influences, but it does show some special features. The speed of sound traveling in air is affected by air pressure, humidity, and temperature (Varghese et al., 2015). The fluctuation of accuracy caused by this is a concern to autonomous driving unless enlisting the help of algorithms that can adjust the readings according to the ambient environment which generates extra costs. Nonetheless, ultrasonic does have its strengths, given the fact that its basic function is less affected by harsh weather compared to LiDAR and camera. The return signal of an ultrasonic wave does not get decreased due to the target’s dark color or low reflectivity, so it is more reliable in low visibility environments than cameras, such as high-glare or shaded areas beneath an overpass.
Additionally, the close proximity specialty of ultrasonic can be used to classify the condition of the road surface. Asphalt, grass, gravel, or dirt road can be distinguished from their back-scattered ultrasonic signals (Bystrov et al., 2016), so it is not hard to imagine that the snow, ice, or slurry on the road can be identified and help AV weather classification as well.
The following extract makes clear that the solutions known to these researchers are still to be found if ADAS or AV is to manage predictable
extreme light and weather conditions are to be managed within an acceptable power envelope in real time.
8. Conclusion
In this work, we surveyed the influence of adverse weather conditions on 5 major ADS sensors. Sensor fusion solutions were listed. The core solution to adverse weather problems is perception enhancement and various machine learning and image processing methods such as de-noising were thoroughly analyzed. Additional sensing enhancement methods including classification and localization were also among the discussions. A research tendency towards robust sensor fusions, sophisticated networks and computer vision models is concluded. Candidates for future ADS sensors such as FMCW LiDAR, HDR camera and hyperspectral camera were introduced. The limitations brought by the lack of relevant datasets and the difficulty of 1550 nm LiDAR were thoroughly explained. Finally, we believe that V2X and IoT have a brighter prospect in future weather research. This survey covered almost all types of common weather that pose negative effects on sensors’ perception and sensing abilities including rain, snow, fog, haze, strong light, and contamination, and listed out datasets, simulators, and experimental facilities that have weather support.
With the development of advanced test instruments and new technologies in LiDAR architectures, signs of progress have been largely made in the performance of perception and sensing in common wet weather. Rain and fog conditions seem to be getting better with the advanced development in computer vision in recent years, but still have some space for improvement on LiDAR. Snow, on the other hand, is still at the stage of dataset expansion and perception enhancement against snow has some more to dig in. Hence, point cloud processing under extreme snowy conditions, preferably with interaction scenarios either under controlled environments or on open roads is part of our future work. Two major sources of influence, strong light and contamination are still not rich in research and solutions. Hopefully, efforts made towards the robustness and reliability of sensors can carry adverse weather conditions research to the next level.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Funding
The author (Y.Z) would like to take this opportunity to thank the “Nagoya University Interdisciplinary Frontier Fellowship” supported by Nagoya University and JST, Japan, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2120, and JSPS KAKENHI, Japan Grant Number JP21H04892 and JP21K12073.
The authors thank Prof. Ming Ding from Nagoya University for his help. We would also like to extend our gratitude to Sensible4, the University of Michigan, Tier IV Inc., Ouster Inc., Perception Engine Inc., and Mr. Kang Yang for their support. In addition, our deepest thanks to VTT Technical Research Center of Finland, the University of Waterloo, Pan Asia Technical Automotive Center Co., Ltd, and the Civil Engineering Research Institute for Cold Region of Japan.
Similar threads
