Interesting recent interview with a team member from Sony Semicon who is working with Prophesee on EVS.
Nothing on us however my takeaway is an understanding of the "time and effort" that has to go into the joint dev, testing and production in merging tech to create one final product.
Food for thought & appreciation maybe in some TSE discussions re how long for consumer / commercial / enterprise product releases...and this by one of the industry giants
Introducing special contents of Sony Semiconductor Solutions Group.
www.sony-semicon.com
Sony Semiconductor Solutions Group
Pursuing the approach to social value in developing new technology
March 22, 2023
There has been a growing trend in recent years to combine AI with diverse sensors, and against this backdrop, the Event-based Vision Sensor (EVS) is garnering much interest.
This is a sensor that only extracts changes within a frame, enabling to significantly reduce the power consumption and the volume of data to be handled compared to the conventional frame-based sensing data.
It has great potential for extensive applications. We interviewed the engineer who developed the world’s smallest*1 pixel for EVS, Atsumi Niwa of Sony Semiconductor Solutions Corporation (SSS) Research Division 1, and asked him about his experience in the development project and the secrets of his success in creating the technology the world has never seen before.
*1) According to Sony research (as of September 2021)
Niwa Atsumi
Sony Semiconductor Solutions Corporation
Research Division 1
Profile:Niwa joined the Sony Corporation’s Semiconductor Business Group (present SSS) in 2008.
He initially worked on the development of analog circuits for TV tuners, where he mainly designed the base band analog signal processors and AD converters. Subsequently, he contributed to enhance the performance of the CMOS image sensor for mobile applications. He joined the project to develop EVS in 2017, which led to the launch of IMX636 released in 2021. In recognition of this achievement, he received 2021 Sony Outstanding Engineer Award. Today, he pursues the internal rollout of the EVS technology and pursues to expand the scope of application for the technology through his development projects.
The desire to find out what contributions EVS might offer to society prompted him to participate in the project to commercialize the technology
- What kind of sensor is an EVS?
EVS is a fundamentally different sensor from general image sensors which have been evolving so far to reproduce images as truthfully to our visions as possible. It measures differences in luminance at the pixel level and outputs the data with their coordinates and time stamps. There are three major advantages of EVS.
Firstly, EVS can detect the luminance changes without complex setting depended on light condition.
Thus user can easily use sensor compared to image sensor which requires lots of setting to produce optimal data.Secondly, it consumes far less power because the output data only contains the detected differences.
And thirdly, thanks to the pixel-parallel detection, EVS only outputs change unlike conventional image sensors, which read out approximately 30 to 60 frames of whole images per second. This allows EVS to respond quickly and capture changes at less than a millisecond.
Suppose that we are capturing a welding process in a factory, the conventional image sensors would produce an image that part is white out because the welded part is extremely bright. Similarly, the sparks must be observed to discern whether the welding is applied correctly, but the conventional image sensors are not capable of precisely capturing the fast-moving sparks. The EVS, on the other hand, can easily capture individual sparks even under this high-contrast condition.
Metal process monitoring
Frame-based sensor image
The sparks are overexposed due to high luminance,
making linear trails in the image.
EVS image
Each fast-moving spark is captured individually [high frame rate]
Data other than the sparks (such as the machinery) are not output [high efficiency: minimal data output]
Application output
Each spark is tagged with ID and tracked
⇒ analyzable in terms of the number, size, speed, etc
Alternatively, the sensor’s capability to measure motion can be applied to machine inspection. Engines, for example, normally run constantly, and their movements in a normal condition become disturbed if there are some abnormalities. The EVS can be used to detect such abnormal movements to catch early signs of malfunctions.
Vibration monitoring
Frame-based sensor image
It is impossible to discern the vibration
in the model car on the platform with naked eye.
EVS image
Only the vibrating areas are processed and
output so that the vibration is clearly visualized.
Application output
The frequency is analyzed per pixel and
can be mapped out in two dimensions.
- Has EVS existed for a long time?
The technology emerged during the first decade of the 2000s, inspired by the ways in which the human eye recognized images. The sensor only catches moving targets and outputs the data at high speed. This simply represents the high efficiency in only capturing changes, but the vision sensor has evolved through attempts to apply it to various contexts by combining with a neural network.*2
With reference to the point of efficiency I just mentioned, extracting change information from images captured using the conventional image sensors significantly increases power consumption and the volume of data to be handled. To further evolve AI, sensors will be expected to only provide the information required by the neural network, and EVS represents one of the viable options for this purpose.
*2) A series of algorithms for pattern recognition modeled on the ways in which the human brain operates.
- When you first came across this technology, did you immediately want to get involved in developing it?
When I heard of EVS, I found it interesting, and the idea of making one appealed to me. Also, I was curious to know how this technology could contribute to society.
My background was CMOS image sensors, developing them for taking clear pictures.
EVS with its principles totally different from what I knew excited me for discovering new potential. Soon afterwards, a French venture of sensing devices, Prophesee, approached us to propose a joint project to commercialize EVS. This project was the first attempt in the world to turn EVS into a commercial product. I was keen to witness firsthand how the technology would contribute to society and decided to join the project.
Unprecedented external joint development, having to deal with differences in everything from workflows to technical terms
- What was the most difficult part in developing this EVS with the smallest pixels in the world?
Each pixel in an EVS has its event detection circuit. If the circuit specifications become demanding, the pixel size increases to accommodate all of it, and this was a problem. The possibilities of implementing it on wearable devices and smartphones make it absolutely necessary that the pixels are smaller and more power-efficient.
So, we decided to design a circuit from scratch, drawing on SSS’s expertise, stacking technology and circuit design know-how.
It was hard-going at the stage where we combined this new circuit design with the process technology. We had many trials and errors before achieving the pixel size reduction.
Meanwhile, another difficulty was to deal with so many unknown factors. Usually, we pursue a development by supposing some use contexts for the product, but the technology concerning EVS had not been established within SSS. We could imagine some use cases but could not verify whether desired data could be obtained without actually testing.
Moreover, the technology being new to us, we had no adequate equipment for verification, data acquisition and, therefore, evaluation. It was necessary to develop the environment for technological verification in parallel with the product development, which made the project time-consuming and difficult to organize.
- This was the first collaboration with other companies in the EVS domain. How did you find it?
There have been many collaborative projects where responsibilities were clearly divided. This time, it was the first ever experience for me as well as for SSS in that two companies joined forces to develop and design one stacked sensor.
SSS has predetermined design development workflows, and these are, obviously, unique to the company. The language we use is also different from theirs.
Communication was often riddled with difficulties due to the differences in the technical background knowledge between Prophesee and SSS. We made efforts to maintain close communication with them and ensure mutual understanding because many little misunderstandings would eventually slow down the development schedules.
While Prophesee and SSS were in agreement to leverage our expertise to create a highly sophisticated EVS product, there were some discrepancies as to what level to be aimed at and how the product should be promoted. We needed to spend quite a lot of time to ensure that we had the same understanding.
The collaboration entailed the establishment of a new development flow in tandem with the product development, and also a new evaluation environment was to be developed. All in all, this was a whole new experience for us.
On top of this, the new concept EVS presented necessitated enduring efforts to explain to people within the Company so that they understood the sensor’s characteristics and were interested in this project. These efforts were necessary to build up the project team.
- How did you face the unfamiliar development workflow?
Despite the difficulties experienced in the development processes and communication, I enjoyed the job very much. I also learned much from the project.
For Prophesee, this technology determines their corporate success or failure. Seeing their uncompromising attitude to details, I thought we could learn from it.
Finding semiconductors interesting at university lab
- When did you become interested in semiconductors?
I was studying circuit design at university, and though our lab was not dealing with the latest processes, we the students were making prototypes and studying the discrepancies between theories and implementations. It was the process of probing why and how the reality defied theories that appealed to me, and I found semiconductors interesting in this way.
The theory-oriented lab taught me the pleasure of in-depth thinking. Joining this lab was a pivotal experience for my future course of life.
- What was the deciding factor for your joining SSS?
At the end of my master’s course, I was in two minds about staying on to pursue a Doctor of Philosophy. Meanwhile, I saw recruiters of various corporations, of whom the members of SSS left a good impression in me. They said that, at SSS, it was possible to propose and pursue projects based on what you found interesting. It seemed to be a great place to be if I wanted to pursue what interested me.
At the time, I was enjoying the process of theorizing and testing the theory at the university lab, and I was also drawn to the idea of creating products and seeing how they benefited people in society. These were the factors that made me want to test my abilities at SSS.
- Have you ever experienced a failure?
I did when I was engaged in the development of CMOS image sensors for cell phones. I was working on a very demanding project in terms of development requirements, so much so that some people in the Company found it unjustifiable.
I was determined to succeed and managed to make it to the prototyping phase. However, a new specification was added before the commercialization. This latest addition proved to be totally incompatible with the structure of the prototype, and eventually my development had to be suspended.
The technology was modified and eventually adopted in the client’s later model. Even so, the taste of failure in the first attempt was so bitter.
Think deeply about the technology, what value it offers and how it can contribute to society
- What do you pay attention to when you work?
I take time to consider the research and development project at hand in terms of its future significance and diffusion of value.
Any research and development would need to envision how the technology under development might be put to use. However, thinking about it within a framework of today’s conventions and values may lead to an impasse.
If you are confined in the values of conventional image sensors, for example, you might fail to see the worth of EVS by thinking that the sensor “only captures changes,” which is not a feature required for producing clear images. I think this is a trap that we should avoid. I would like to be someone who really think through “what value can be found” in only capturing changes and “what it will make possible.”
Today, I am fortunate to be at the research division, where we give a thorough consideration on the contributions to society which could be made by the technology we develop and study, and the value it offers. I always learn from this approach.
- What would you like to work on in the future?
SSS has so far been focused on image sensors that are capable of capturing the world as we see it as truthfully as possible, and the world has been demanding such technology and products.
However, I believe that the future opens up to approaches to changing our ways of life from the viewpoint of “social richness.” A sensor may not be able to capture a beautiful photo for viewing, but it is designed to extract some information, which will enable a system to function in such a way that it liberates us humans from our heavy-load tasks.
I would like to be part of the efforts to expand the imaging technology for “photography” into the domain of sensing for “obtaining necessary information.” It would be exciting to keep exploring the possible value that sensing technology can create.
I imagine that the future will have richer combinations of new image sensors and their applications, making our lives better and turning society into a more interesting place. With the technology concerning image sensors at the core, how should we leverage it and what value can be derived from it? We would like to adopt this application-oriented approach to prove the value we create so that our work and products will lead to positive changes in society.
- Image sensors that enrich society sounds like a wonderful idea.
An ambition like this cannot be fulfilled by a solo effort. This is why I always discuss desirable image sensors with many colleagues from various fields of expertise.
And discussing is not enough to move things forward. I think it is important to always keep myself open to new information and keep an extensive network of people as potential source.
- Interviewer’s postscript
It is clear that Niwa has a constant attitude of pursuing something, such as theories at university, the project which everyone else had given up on at the development division, and value for society which he currently pursues at the research division. It seemed to me that this attitude and persistence to challenge were a crucial part of his competence that enabled him to create something the world had never known. His journey to explore the potential of image sensors is far from over. It will be exciting to see what kind of technology he comes up with next to surprise us.
Related links
wand and poof it’s there. Dream on guys, if it were that easy everyone would be doing it. 
) I just happened to read somewhere that Bayern München are planning a tour through Asia in July. Let’s assume they will be playing each of their three planned matches in Bangkok, Singapore and Tokyo at 8 pm local time. For fans in Germany wanting to watch a live TV broadcast, however, the kick-off will be at different times each match day (3 pm German time for the match in Bangkok, 2 pm for the one in Singapore and 1 pm for the match in Tokyo), as there is no one single Asian time zone. Neither is there one time zone for all of Australia.
