It does seem like a very clever move on the part of Brainchip to almost a year ago to bring Krishnamurthy V. Venuru on board and allow him to complete his Canny Edge Detector Using a Spiking Neural Network algorithm research funded by Riverside Research’s Open Innovation Center which will accelerate bringing to AKIDA technology the ability to identify boundaries for medical imaging one of the many large addressable markets Brainchip is targeting. There are other vision applications such as UAVs where this has application however medical imaging is a standout application in my opinion. Being able to define the exact outline of a tumour or foreign body is invaluable to surgeons preparing to operate.
My opinion only DYOR
FF
AKIDA BALLISTA - (This paper has been previously referenced here on TSEx - this is an extract as the full paper exceeds the site limits.) :
Implementation of the Canny Edge Detector Using a Spiking Neural Network
by
Krishnamurthy V. Vemuru
†
Riverside Research, 2900 Crystal Dr., Arlington, VA 22202, USA
†
Current Address: BrainChip Inc., 23041 Avenida de la Carlota, Laguna Hills, CA 92653, USA.
Future Internet 2022,
14(12), 371;
https://doi.org/10.3390/fi14120371
Received: 6 June 2022 / Revised: 5 December 2022 / Accepted: 7 December 2022 / Published: 11 December 2022
(This article belongs to the Collection
Computer Vision, Deep Learning and Machine Learning with Applications)
Abstract
Edge detectors are widely used in computer vision applications to locate sharp intensity changes and find object boundaries in an image. The Canny edge detector is the most popular edge detector, and it uses a multi-step process, including the first step of noise reduction using a Gaussian kernel and a final step to remove the weak edges by the hysteresis threshold. In this work, a spike-based computing algorithm is presented as a neuromorphic analogue of the Canny edge detector, where the five steps of the conventional algorithm are processed using spikes. A spiking neural network layer consisting of a simplified version of a conductance-based Hodgkin–Huxley neuron as a building block is used to calculate the gradients. The effectiveness of the spiking neural-network-based algorithm is demonstrated on a variety of images, showing its successful adaptation of the principle of the Canny edge detector. These results demonstrate that the proposed algorithm performs as a complete spike domain implementation of the Canny edge detector.
Keywords:
edge detection;
segmentation;
spiking neural networks;
bio-inspired neurons
Graphical Abstract
1. Introduction
Artificial neural networks (ANNs) have become an indispensable tool for implementing machine learning and computer vision algorithms in a variety of pattern recognition and knowledge discovery tasks for both commercial and defense interests. Recent progress in neural networks is driven by the increase in computing power in data centers, cloud computing platforms, and edge computing boards. In size, weight, and power (SWaP)–constrained applications, such as unmanned aerial vehicles (UAVs), augmented reality headsets, and smart phones, more novel computing architectures are desirable. The state-of-the-art deep learning hardware platforms are often based on graphics processing units (GPUs), tensor processing units (TPUs) and field programmable gate arrays (FPGAs). The human brain is capable of performing more general and complex tasks at a minute fraction of the power required by deep learning hardware platforms. Spiking neurons are regarded as the building blocks of the neural networks in the brain. Moreover, research in neuroscience indicates the spatiotemporal computing capabilities of spiking neurons play a role in the energy efficiency of the brain. In addition, spiking neurons leverage sparse time-based information encoding, event-triggered plasticity, and low-power inter-neuron signaling. In this context, neuromorphic computing hardware architecture and spike domain machine learning algorithms offer a low-power alternative to ANNs on von Neumann computing architectures. The availability of neuromorphic processors, such as IBM’s TrueNorth [
1], Intel’s Loihi [
2], and event-domain neural processors, for example, BrainChip’s Akida [
3,
4], which offers the flexibility to define both artificial neural network layers and spiking neuron layers, are motivating the research and development of new algorithms for edge computing. In the present work, we have investigated how one can program an algorithm for Canny type edge detection using a spiking neural network and spike-based computing.
2. Background
An edge detection algorithm is widely used in computer vision to locate object boundaries in images. An edge in an image shows a sharp change in image brightness, which is a result of a sharp change in pixel intensity data. An edge detector computes and identifies the pixels with sharp changes in intensity with respect to the intensity of neighboring pixels. There are several edge detection image processing algorithms.
The three stages in edge detection are image smoothing, detection, edge localization. There are mainly three types of operators in edge detection. These are (i) gradient-based, (ii) Laplacian-based and (iii) Gaussian-based. The gradient-based edge detection method detects the edges by finding the maximum and the minimum in the first derivative of the image using a threshold. The Roberts edge detector [
5], Sobel edge detector [
6], and Prewitt edge detector [
7] are some of the examples of gradient-based edge detectors. These detectors use a 3 × 3 pattern grid. A detailed discussion on these edge detectors and a comparison of their advantages and disadvantages can be found in [
8]. The Roberts edge detection method is built on the idea that a difference on any pair of mutually perpendicular directions can be used to calculate the gradient. The Sobel operator uses the convolution of the images with a small, separable, and integer-valued filter in horizontal and vertical directions for edge detection. The Prewitt edge detector uses two masks, each computing the derivate of the image in the x-direction and the y-direction. This detector is suitable to estimate the magnitude and orientation of the edge. Laplacian-based edge detectors find the edges by searching for zero crossings in the second derivative of the image. The Laplacian of the Gaussian algorithm uses a pre-smoothing step with a Gaussian low-pass filter on an image followed by a second-order differential, i.e., Laplacian, which finds the image edge. This method needs a discrete convolutional kernel that can approximate the second derivative for the image which consists of discrete pixels. The Marr–Hildreth edge detector is also based on the Laplacian of the Gaussian operator [
9]. The Gabor filter edge detector [
10] and Canny edge detector [
11] are Gaussian-based edge detectors. The Gabor filter is a linear filter with its impulse response function defined by the product of a harmonic function with a Gaussian function and is similar to the human perception system.
The Canny edge detector provides excellent edge detection, as it meets the three criteria for edge detection [
12]: (i) detection with low error rate, (ii) the edge point should localize in the center of the edge, and (iii) an edge should only be marked once and image noise should not create edges. Canny edge detection uses the calculus of variations to optimize a functional which is a sum of four exponential terms, which approximates the first derivative of a Gaussian. A Canny edge detector is a multi-step algorithm designed to detect the edges of any analyzed image. The steps of this process are: (1) removal of noise in the image using a Gaussian filter, (2) calculation of the gradient of the image pixels along x- and y-directions, (3) non-maximum suppression to thin out edges, (4) double-threshold filtering to detect strong, weak and non-relevant pixels, and (5) edge tracking by hysteresis to transform weaker pixels into stronger pixels if at least one of their neighbors is a stronger pixel. The Canny edge detection algorithm is highly cited (∼36,000 citations) and the most commonly used edge detection algorithm [
11].
Edge detection is a primary step in identifying an object and further research is strongly desirable to expand these methods to event-domain applications. The Canny edge detector has a better performance as an edge detector compared to Roberts, Sobel and Prewitt edge detectors, but at a higher computational cost [
8]. An alternate implementation of the Canny edge detection algorithm is for edge computing event-domain applications, where low-power and real-time solutions can be attractive to target applications, in view of its tunable performance using the standard deviation of the Gaussian filter.
3. Related Work
In the human vision system, the photoreceptors in the retina convert the light intensity into nerve signals. These signals are further processed and converted into spike trains by the ganglion cells in the retina. The spike trains travel along the optic nerve for further processing in the visual cortex. Neural networks that are inspired by the human vision system have been introduced to improve image processing techniques, such as edge detection [
13]. Spiking neural networks, which are built on the concepts of spike encoding techniques [
14], spiking neuron models [
15] and spike-based learning rules [
16], are biologically inspired in their mechanism of image processing. SNNs are gaining attraction for biologically inspired computing and learning applications [
17,
18]. Wu et al. simulated a three-layer spiking neural network (SNN), consisting of a receptor layer, an intermediate layer with four filters, respectively, for up, down, left, and right directions, and an oputput layer with Hogkin–Huxley-type neurons as the building blocks for edge detection [
19].
Clogenson et al. demonstrated how a SNN with scalable, hexagonally shaped receptive fields performs edge detection with computational improvements over rectangular shaped pixel-based SNN approaches [
20]. The digital images are converted into a hexagonal pixel representation before being processed by the SNN. A spiking neuron integrates the spikes from a group of afferent neurons in a receptive field. The network model used by the authors consists of an intermediate layer with four types of neurons corresponding to four different receptive fields, corresponding to up, down, right and left orientations. Yedjour et al. [
21] demonstrated the basic task of contour detection using a spiking neural network based on the Hodgkin–Huxley neuron model. In this approach, the synaptic weights are determined by the Gabor function to describe the receptive field’s behaviors of simple cells in the visual cortex. Vemuru [
22] reported the design of a SNN edge detector with biologically inspired neurons and demonstrated that the edge detector detects edges in simulated low-contrast images. These studies focused on defining SNNs using an array of Gabor filter receptive fields in the edge detector. In view of the success of SNNs in edge detection, it is desirable to develop a spike domain implementation of an edge detector with the Canny edge detector algorithm because it has the potential to offer a high performance alternative for edge detection.
5. Results and Discussion
Figure 2 displays the original image (No. 61060) from BSD500 dataset [
32,
33] (license: Not found) and the intermediate output images of the spiking detector network after the Gaussian kernel, gradient calculation, non-maximum suppression, double-threshold filtering, and edge-tracking by hysteresis. The intermediate features show the spike domain implementation is effective in performing all five steps involved in the conventional Canny edge detection algorithm. The edge map resulting from the spike-based computation, i.e., the output of the network after the edge-tracking hysteresis step, can be referred to as the neuromorphic Canny edge detector or SNN-based Canny edge detector. We find that the resolution of the images from BSD500 dataset is sufficient to evaluate the qualitative and quantitative performance of edge detection. In
Figure 3, we compare the neuromorphic Canny detector with the Sobel detector and the conventional Canny detector for a set of four images and their ground truth edges from the BSD500 dataset [
32,
33]. We find the neuromorphic Canny detector detects edges similarly to the edges generated by both the conventional detectors. The spike-based detection reported here results in relatively more highlighted edges in the objects while retaining some of the structural features that seem to have been lost in the conventional edge detectors. This can be attributed to the difference in the range of thresholds used in the spike domain implementation compared to the conventional Canny edge detector. With an implementation on a low size, weight and power (SWaP) hardware, for example, by emulating spikes on a field programmable gate array (FPGA) [
34] or edge computing devices, such as NVIDIA Jetson TX2, together with a neuromorphic camera, the spike domain image processing algorithm will be attractive for new applications, for example, in handheld computer vision systems and for low-Earth satellite-based object detection.
Figure 2. The original and the output images from all 5 stages of neuromorphic Canny edge detection.
Figure 3. A comparison of the ground truth with the results from Canny edge detector and the SNN analogue of Canny edge detector for selected images from BSD500 dataset.
In image processing, it is common to select a few images and demonstrate how the method works. Statistics on training sets and test sets usually applies to machine learning experiments, which is the not the case in the present study. All the images that are pictures of natural scenes with edges and none of the examples are synthesized. From visual inspection, the SNN edge detector appears to render wider edges and detect more background information. It is possible that the difference in the SNN Canny edge map compared to the conventional Canny edge map is also related to the choice of the threshold used or the smoothing parameters. The SNN edge maps are realized in a narrower threshold range, and this makes it difficult to perform an ablation study, which is typically done in machine learning experiments. We would like to note that the primary goal of the present work is to demonstrate a spike-based implementation of Canny edge detection, not the superior computational efficiency. Currently, there is no hardware that can implement the exact version of the neuron model used in the present work to evaluate the computational time for the targeted neuromorphic domain. The context of this work is that it addresses the question of whether it is possible to compute the steps of the Canny edge algorithm by exclusively using spikes.
Figure 4. A comparison of the ground truth with the results from Canny edge detector and the SNN analogue of Canny edge detector for selected images from BSD500 dataset. For these images, the metrics for SNN Canny edge detector come out lower compared to the Canny edge detector.
Table 1. Comparison of performance ratio, PR, and F1�1-score for selected images.
Object detection in infrared images is another key application for edge detectors, for example in night vision gadgets or for autonomous driving. Novel edge detectors, especially those that can extract edges in low-resolution infrared images with low-SWaP requirements for hardware implementation, are sought across commercial and defense applications. In this context, we evaluated our SNN edge detector with a few infrared images from the Thermal Road Dataset [
36] (license: Not found).
Figure 5 shows a comparison of edge detection with SNN-based Canny detector, conventional Canny detector and Sobel detector. Ground truth edge maps are not available for this dataset to perform a quantitative comparison similar to the one presented for the RGB images in
Table 1. A visual comparison of the results from the three edge detectors in
Figure 5 indicates that the SNN-based Canny edge detector is able to generate edge maps very similar to the ones generated by the conventional Canny edge detector.
Figure 5. A comparison of edge detection by Sobel, Canny, and SNN Canny edge detectors for images from thermal road dataset.
Biomedical image processing is a field with an increasing demand for advanced algorithms that automate and accelerate the process of segmentation. In a recent review, Wallner et al. reported multi-platofrm performance evaluation of open-source segmentation algorithms, including the Canny-edge-based segmentation method, for cranio-maxillofacial surgery [
37]. In this context, it is desirable to test newer algorithms such as the one developed here using spiking neurons on medical images. To this end, we performed an edge detection experiment with a few representative medical images from the computed tomography (CT) emphysema dataset [
38] (this database can be used free of charge for research and educational purposes). This dataset consists of 115 high-resolution CT slices as well as 168 square patches that are manually annotated in a subset of the slices.
Figure 6 illustrates a comparison of the three edge detectors, Sobel, Canny and SNN-based Canny detectors, for a few example images from the CT emphysema dataset. Ground truth for edge maps is not available for this dataset to perform a quantitative comparison. A visual comparison of the edges generated from the three edge detectors, presented in
Figure 6, shows that the SNN-based Canny edge detector is competitive with the other two edge detectors and offer an algorithmically neuromorphic alternative to the conventional Canny edge detector.
Figure 6. A comparison of edge detection by Sobel, Canny, and SNN Canny edge detectors for medical images from CT emphysema dataset.
6. Conclusions
In conclusion, we present a spiking neural network (SNN) implementation of the Canny edge detector as its neuromorphic analogue by introducing algorithms for spike based computation in the five steps of the conventional algorithm with the conductance-based Hodgkin–Huxley neuron as the building block. Edge detection examples are presented for RGB and infrared images with a variety of objects. A quantitative comparison of the edge maps from the SNN-based Canny detector, conventional Canny detector and a Sobel detector using the F1�1-score as a metric, shows that the neuromorphic implementation of the Canny edge detector achieves better performance. The SNN architecture of the Canny edge detector also offers promise for image processing and object recognition applications in the infrared domain. The SNN Canny edge detector is also evaluated with medical images, and the edge maps compare well with the edges generated with the conventional Canny detector. Future work will focus on the implementation of the algorithm on an FPGA or on a neuromorphic chip for hardware acceleration and testing in an infrared object detection task potentially with edge maps as features together with a pre-processing layer to remove any distortions, enhance contrast, remove blur, etc., and a spiking neuron layer as a final layer to introduce a machine learning component. An extension of the SNN architecture of the Canny edge detector with additional processing layers for object detection in LiDAR point clouds would be another interesting new direction of research [
39].
Funding
This work was internally funded by Riverside Research’s Open Innovation Center.
brainchip.com
www.electronics-lab.com
news.yahoo.com
www.linkedin.com
