US20220067531 - EFFICIENT IDENTIFICATION OF CRITICAL FAULTS IN NEUROMORPHIC HARDWARE OF A NEURAL NETWORK

[uiux]

Interesting patent from NVIDIA @Diogenese

WIPO - Search International and National Patent Collections

patentscope.wipo.int

US20220067531 - EFFICIENT IDENTIFICATION OF CRITICAL FAULTS IN NEUROMORPHIC HARDWARE OF A NEURAL NETWORK

Abstract
The disclosure provides misclassification-driven training (MDT) that efficiently identifies critical faults in neuromorphic hardware, such as a memristor crossbar. MDT advantageously identifies whether a hardware fault is a critical fault and can be used to limit fault recovery when a hardware fault is not a critical fault. By applying fault-tolerant techniques directed to critical faults, such as only for critical faults, processing overhead of a neural network can be reduced. In one aspect, the disclosure provides a method of identifying critical faults in neuromorphic hardware of a neural network. In one example the method of identifying includes: (1) determining a significant parameter of a trained neural network that impacts classification of a sample of a dataset, (2) obtaining a location of the significant parameter in the neuromorphic hardware, and (3) identifying the location as a critical fault of the neuromorphic hardware.

 

[Phoenix14]

Hi Uiux , can you please give us some of your thoughts on this patent ? It’s intent ?
Is it significant and could it have Any affect on the industry ?

Thanks

 

[uiux]

Hi Uiux , can you please give us some of your thoughts on this patent ? It’s intent ?
Is it significant and could it have Any affect on the industry ?

Thanks

Would wait for @Diogenese opinion

My thoughts are that this is complementary tech to ours. It mentions memristors in the patent so I am wondering if this would be relevant to our tech or if this could diagnose faulty nodes in our tech.

Either way it shows Nvidia is innovating for the neuromorphic industry. This indicates they actually could be a potential licensee for neuromorphic IP - if they aren't developing their own.

 

[TechGirl]

Come on Dio we are waiting

 

[Diogenese]

Hi Uiux , can you please give us some of your thoughts on this patent ? It’s intent ?
Is it significant and could it have Any affect on the industry ?

Thanks

Would wait for @Diogenese opinion

My thoughts are that this is complementary tech to ours. It mentions memristors in the patent so I am wondering if this would be relevant to our tech or if this could diagnose faulty nodes in our tech.

Either way it shows Nvidia is innovating for the neuromorphic industry. This indicates they actually could be a potential licensee for neuromorphic IP - if they aren't developing their own.

Hi Phoenix,

This invention is specifically directed to "specialized neuromorphic hardware", and in particular to the "stress testing" of the circuit design to identify points in the circuit layout which are vulnerable to fault errors.

Deep Learning (DL) applications are becoming increasingly ubiquitous. However, recent research has highlighted a number of reliability concerns associated with deep neural networks (DNNs) used for DL. In particular, hardware-level reliability of DNNs is of particular concern when DL models are mapped to specialized neuromorphic hardware such as memristor-based crossbars. However, DNN architectures are inherently fault-tolerant and many faults do not have any significant impact on inferencing accuracy.
...

Deep Learning (DL) applications, e.g., self-driving cars, image recognition, and medical diagnosis, are becoming an increasingly important part of life. However, recent research has highlighted a number of reliability concerns associated with DL architectures. Some of the reliability concerns include the vulnerability of DL models to perturbations of the input dataset of an application and adversarial attacks. For example, when a small amount of noise or backdoor triggers are added to input data, the victim models provide significantly low prediction accuracy. Defenses against such malicious attacks at an abstract functional level have been proposed.

Memristor NNs rely on fairly precise measurement of the currents and voltages in the circuit, but memristors are prone to inaccuracies due to manufacturing variations because they rely on analog elements like capacitors and resistors.

However, neuromorphic hardware can have defects due to the immature fabrication process and high density. For example, memristor devices can be subject to various types of manufacturing defects and process variations. In addition, both training and inferencing associated with a memristor crossbar can be significantly degraded by defects, variations, and non-idealities. These defects, variations, and non-idealities can be viewed as faults, which is defined as any deviation from the nominal expected behavior of a neuromorphic hardware cell, such as memristor cells in a memristor crossbar. These faults can adversely affect the classification accuracy when such crossbars are used for inferencing.

We have Anil for that sort of thing in our binary digital CMOS circuit (BDCC). But BDCCs are ON/OFF devices which have a large tolerance for voltage variations.

This patent is not relevant to the manufacture of BDCCs.

 

[TechGirl]

Hi Phoenix,

This invention is specifically directed to "specialized neuromorphic hardware", and in particular to the "stress testing" of the circuit design to identify points in the circuit layout which are vulnerable to fault errors.

Deep Learning (DL) applications are becoming increasingly ubiquitous. However, recent research has highlighted a number of reliability concerns associated with deep neural networks (DNNs) used for DL. In particular, hardware-level reliability of DNNs is of particular concern when DL models are mapped to specialized neuromorphic hardware such as memristor-based crossbars. However, DNN architectures are inherently fault-tolerant and many faults do not have any significant impact on inferencing accuracy.
...

Deep Learning (DL) applications, e.g., self-driving cars, image recognition, and medical diagnosis, are becoming an increasingly important part of life. However, recent research has highlighted a number of reliability concerns associated with DL architectures. Some of the reliability concerns include the vulnerability of DL models to perturbations of the input dataset of an application and adversarial attacks. For example, when a small amount of noise or backdoor triggers are added to input data, the victim models provide significantly low prediction accuracy. Defenses against such malicious attacks at an abstract functional level have been proposed.

Memristor NNs rely on fairly precise measurement of the currents and voltages in the circuit, but memristors are prone to inaccuracies due to manufacturing variations because they rely on analog elements like capacitors and resistors.

However, neuromorphic hardware can have defects due to the immature fabrication process and high density. For example, memristor devices can be subject to various types of manufacturing defects and process variations. In addition, both training and inferencing associated with a memristor crossbar can be significantly degraded by defects, variations, and non-idealities. These defects, variations, and non-idealities can be viewed as faults, which is defined as any deviation from the nominal expected behavior of a neuromorphic hardware cell, such as memristor cells in a memristor crossbar. These faults can adversely affect the classification accuracy when such crossbars are used for inferencing.

We have Anil for that sort of thing in our binary digital CMOS circuit (BDCC). But BDCCs are ON/OFF devices which have a large tolerance for voltage variations.

This patent is not relevant to the manufacture of BDCCs.

View attachment 2630

Thanks Dio

 

[Phoenix14]

Hi Phoenix,

This invention is specifically directed to "specialized neuromorphic hardware", and in particular to the "stress testing" of the circuit design to identify points in the circuit layout which are vulnerable to fault errors.

Deep Learning (DL) applications are becoming increasingly ubiquitous. However, recent research has highlighted a number of reliability concerns associated with deep neural networks (DNNs) used for DL. In particular, hardware-level reliability of DNNs is of particular concern when DL models are mapped to specialized neuromorphic hardware such as memristor-based crossbars. However, DNN architectures are inherently fault-tolerant and many faults do not have any significant impact on inferencing accuracy.
...

Deep Learning (DL) applications, e.g., self-driving cars, image recognition, and medical diagnosis, are becoming an increasingly important part of life. However, recent research has highlighted a number of reliability concerns associated with DL architectures. Some of the reliability concerns include the vulnerability of DL models to perturbations of the input dataset of an application and adversarial attacks. For example, when a small amount of noise or backdoor triggers are added to input data, the victim models provide significantly low prediction accuracy. Defenses against such malicious attacks at an abstract functional level have been proposed.

Memristor NNs rely on fairly precise measurement of the currents and voltages in the circuit, but memristors are prone to inaccuracies due to manufacturing variations because they rely on analog elements like capacitors and resistors.

However, neuromorphic hardware can have defects due to the immature fabrication process and high density. For example, memristor devices can be subject to various types of manufacturing defects and process variations. In addition, both training and inferencing associated with a memristor crossbar can be significantly degraded by defects, variations, and non-idealities. These defects, variations, and non-idealities can be viewed as faults, which is defined as any deviation from the nominal expected behavior of a neuromorphic hardware cell, such as memristor cells in a memristor crossbar. These faults can adversely affect the classification accuracy when such crossbars are used for inferencing.

We have Anil for that sort of thing in our binary digital CMOS circuit (BDCC). But BDCCs are ON/OFF devices which have a large tolerance for voltage variations.

This patent is not relevant to the manufacture of BDCCs.

View attachment 2630

Thanks Dio
I guess my next question is …
Is this a stepping stone for them or a piece of a jigsaw puzzle to try and get around our patents ?
Or are they simply just filing any discovery they make in the event that it becomes useful in the future ? Or is it anyones guess !

 

[Diogenese]

Thanks Dio
I guess my next question is …
Is this a stepping stone for them or a piece of a jigsaw puzzle to try and get around our patents ?
Or are they simply just filing any discovery they make in the event that it becomes useful in the future ? Or is it anyones guess !

Hi Phoenix,

Analog spiking neural networks (MemRistors/ReRAM) (see, eg, WeeBit, ) have been around for a while and there are several companies working on the technology.

A Survey of ReRAM-Based Architectures for Processing-In-Memory and Neural Networks (April 2018)
https://res.mdpi.com/make/make-01-00005/article_deploy/make-01-00005.pdf [see page 106]

ASNNs are a different type of technology attempting to do what Akida's digital SNN (DSNN) can do. I think Nvidia are having 5 bob each way.

 

[Diogenese]

Hi Phoenix,

This invention is specifically directed to "specialized neuromorphic hardware", and in particular to the "stress testing" of the circuit design to identify points in the circuit layout which are vulnerable to fault errors.

Deep Learning (DL) applications are becoming increasingly ubiquitous. However, recent research has highlighted a number of reliability concerns associated with deep neural networks (DNNs) used for DL. In particular, hardware-level reliability of DNNs is of particular concern when DL models are mapped to specialized neuromorphic hardware such as memristor-based crossbars. However, DNN architectures are inherently fault-tolerant and many faults do not have any significant impact on inferencing accuracy.
...

Deep Learning (DL) applications, e.g., self-driving cars, image recognition, and medical diagnosis, are becoming an increasingly important part of life. However, recent research has highlighted a number of reliability concerns associated with DL architectures. Some of the reliability concerns include the vulnerability of DL models to perturbations of the input dataset of an application and adversarial attacks. For example, when a small amount of noise or backdoor triggers are added to input data, the victim models provide significantly low prediction accuracy. Defenses against such malicious attacks at an abstract functional level have been proposed.

Memristor NNs rely on fairly precise measurement of the currents and voltages in the circuit, but memristors are prone to inaccuracies due to manufacturing variations because they rely on analog elements like capacitors and resistors.

However, neuromorphic hardware can have defects due to the immature fabrication process and high density. For example, memristor devices can be subject to various types of manufacturing defects and process variations. In addition, both training and inferencing associated with a memristor crossbar can be significantly degraded by defects, variations, and non-idealities. These defects, variations, and non-idealities can be viewed as faults, which is defined as any deviation from the nominal expected behavior of a neuromorphic hardware cell, such as memristor cells in a memristor crossbar. These faults can adversely affect the classification accuracy when such crossbars are used for inferencing.

We have Anil for that sort of thing in our binary digital CMOS circuit (BDCC). But BDCCs are ON/OFF devices which have a large tolerance for voltage variations.

This patent is not relevant to the manufacture of BDCCs.

View attachment 2630

PS: A similar approach to that described by Nvidia may be useful in designing radiation hardened circuits where the points of vulnerability could be identified.

 

[Phoenix14]

Hi Phoenix,

Analog spiking neural networks (MemRistors/ReRAM) (see, eg, WeeBit, ) have been around for a while and there are several companies working on the technology.

A Survey of ReRAM-Based Architectures for Processing-In-Memory and Neural Networks (April 2018)
https://res.mdpi.com/make/make-01-00005/article_deploy/make-01-00005.pdf [see page 106]

ASNNs are a different type of technology attempting to do what Akida's digital SNN (DSNN) can do. I think Nvidia are having 5 bob each way.

Hi Dio
Thanks for your expertise .

 

[uiux]

Hi Phoenix,

Analog spiking neural networks (MemRistors/ReRAM) (see, eg, WeeBit, ) have been around for a while and there are several companies working on the technology.

A Survey of ReRAM-Based Architectures for Processing-In-Memory and Neural Networks (April 2018)
https://res.mdpi.com/make/make-01-00005/article_deploy/make-01-00005.pdf [see page 106]

ASNNs are a different type of technology attempting to do what Akida's digital SNN (DSNN) can do. I think Nvidia are having 5 bob each way.

Hi Dio - thanks for that


Do you think this patent could be broad enough to cover identifying faults in a single node of an Akida neural fabric?

 

[Diogenese]

Hi Dio - thanks for that


Do you think this patent could be broad enough to cover identifying faults in a single node of an Akida neural fabric?

Hi Ui,

With the proviso that I haven't read the whole patent description, it's nor really relevant to binary digital CMOS circuits. It is intended to be applied to analog MemRistor/ReRAM circuits.

The system checks a software simulation of the chip layout.

Certainly, claim1 is broad enuf to cover our system, but claim 2 is their fallback position for analog NNs during examination by the patent office.

In fact, if the method is not applicable to all NNs, the claims may be invalidated.

Claims​
1. A method of identifying critical faults in neuromorphic hardware of a neural network, comprising: determining a significant parameter of a trained neural network that impacts classification of a sample of a dataset; obtaining a location of the significant parameter in the neuromorphic hardware; and identifying the location as a critical fault of the neuromorphic hardware.
2. The method as recited in claim 1, wherein the neuromorphic hardware is a crossbar of memristors and the location is a memristor location of the crossbar.

As I said, we don't need to worry about faults in our circuit because we have Anil.

 

[thelittleshort]

Would wait for @Diogenese opinion

My thoughts are that this is complementary tech to ours. It mentions memristors in the patent so I am wondering if this would be relevant to our tech or if this could diagnose faulty nodes in our tech.

Either way it shows Nvidia is innovating for the neuromorphic industry. This indicates they actually could be a potential licensee for neuromorphic IP - if they aren't developing their own.

@uiux - further to your statement from your earlier post regarding Nvidia:

1. I found some interesting humanised terminology (screen shots below) on the Nio website that made me think of BrainChip. (Alternatively, they are simply terms associated with AI in general). For those who are not aware of Nio, they are a Chinese automaker specialising in EVs.

Either Nvidia are further along with their own tech or they are indeed adopting some BrainChip IP in my opinion

Thoughts?



2. The Nio intelligent fragrance and air quality system also piqued my interest (screen shots below). It reminded me instantly of your post in 'Ford smells'

Ford smells

https://www.caranddriver.com/news/a32010208/ford-smell-detecting-car-patent/ Ford Patents App to Check Ride-Share Cars for Bad Smells before You Get In How exactly does Ford intend to detect these smells? The patent doesn't go into detail, only describing a multitude of different sensors...

thestockexchange.com.au

US20110089255 - IN-VEHICLE SMELL NOTIFICATION SYSTEM

Abstract
An in-vehicle smell notification system including a vehicle operating computer. A detector is operably connected with the vehicle operating computer and adapted to identify multiple external stimuli. A scent emitter includes multiple scent reservoirs. A controller is operably connected with the detector and the scent emitter and adapted to identify each of the multiple external stimuli and release a predetermined scent from the scent emitter upon identification of a predetermined stimulus.