Well....isn't this new revelation dandy for a Monday ASX200 entry
Recent interview (Mar 22) with Sony Team about their EVS & Prophesee relationship.
Interesting read in regards to speed to commercialisation efforts and also the way Prophesee work with sense of urgency and exploring all avenues.
Introducing special contents of Sony Semiconductor Solutions Group.
www.sony-semicon.co.jp
Event-based Vision Sensors that can function like human optic nerves and monitor only changes in subjects will open up vast new possibilities for AI and robots
March 25, 2022
Event-based Vision Sensors (“EVS”) are said to have been modeled after human optic nerves. Completely different from previous frame-based image sensors, EVS can perceive only changes that have occurred in subjects, in a form like a path or trajectory, so they do not use the concept of “frames” in which sensing targets are captured as images, and have the characteristic of being able to monitor even objects moving at high speed in real-time. Also, in keeping with their origin of being modeled after optic nerves, EVS are anticipated to function as the eyes of AI or robots, and it is believed that if they can be commercialized, they will allow robots to make judgments and decisions even more quickly. These are the EVS that have been developed by the Sony Semiconductor Solutions Group (“SSS Group”), which are expected to breathe new life into various fields including the industrial equipment fields. Their development involved difficulties and discoveries that were only possible precisely because they are entirely new types of vision sensors.
Profile
Sakai.SSony Semiconductor Solutions Corporation
Imaging System Business Division
Main job: coordination of product development, determination of specifications
Responsibilities: product leader
Furukawa.MSony Semiconductor Solutions Corporation
Imaging System Business Division
Main job: coordination of specifications for testing and reliability
Responsibilities: product technology engineer
Nitta.SSony Semiconductor Solutions Corporation
System Solution Business Division
Main job: EVS business representative, EVS promotion, development of applications
Responsibility: business leader
Ihara.SSony Semiconductor Solutions Corporation
System Solution Business Division
Main job: proof of concept activities promotion, response to customer issues and questions
Responsibilities: solution business development engineer
Vision sensors with a completely new concept, born from collaboration with a startup company
What kind of vision sensors are EVS?
Sakai：When people see objects, they are not actually seeing everything that is reflected in their eyes, but perceive areas that undergo movement such as changes in brightness. Like human optic nerves, EVS are vision sensors that only output pixels which have undergone changes in luminance. While conventional image sensors output all image information that they have captured, EVS only output pixels that have changed, so they are capable of high-speed, high-efficiency sensing while also reducing data volume. They are anticipated to be applied in recognition technology that goes a step beyond conventional frame-based image sensors. Meanwhile, since they are sensors that still have low familiarity in the market, they have strong expectations for the future which include the possible ways they will be used by end users, namely what kinds of applications they can be utilized in to receive significant benefits.
What was the trigger for the start of your development of EVS in collaboration with French company Prophesee?
Sakai：It began with an exchange of technology related to EVS with Prophesee by the department I was previously affiliated with. Prophesee had originally intended to commercialize this technology on their own, but their desire to make it even better led them to start collaboration with SSS Group’s Research Division on prototype development.
At that time, I was responsible for research and development of a different project, but when I consulted with my supervisor on my desire to be involved not only with research and development but also in the processes up until commercialization, it turned out that efforts toward EVS as a collaborative project were moving toward commercialization in my same department. Therefore, I transferred business departments and participated as Development Project Leader together with Prophesee.
Two weeks after being assigned to this position I took a business trip to Paris, where I participated in discussions on specifications with Prophesee toward the goal of commercialization. I remember being overwhelmed that the meetings were held in English, which I had difficulty getting used to, and by the speed of actions from prototyping to commercialization.
Nitta：SSS Group is very active in incorporating new technology, and constantly researches startup companies that possess promising technology both domestically and overseas. At this time, the EVS technology being developed by Prophesee apparently had caught the eye of our Technical Supervisor, which led to the start of a technology exchange. After this we realized the potential of EVS to be deployed to a huge variety of applications, so we built a collaborative relationship with the premise of moving forward and building it into a business, and began development toward the goal of commercialization.
What were your concepts during development and any points that you were strict to adhere to?
Sakai：We are constantly thinking of ways to differentiate ourselves in the market, such as by asking “Why are these things being made by SSS Group?” and “What will happen if they are made by SSS Group?” EVS are a type of vision sensor whose market has not yet been widely cultivated, so we considered what kinds of characteristics we could incorporate into them that were different from the technology made first by Prophesee.
Nitta：At the time, multiple companies were making technical announcements related to EVS or holding demonstrations using samples, but they were not yet being deployed to the market. SSS Group had the goal of developing products from this technology as quickly as possible, and establishing a first-mover advantage by releasing them to the market.
Furukawa：Since these were sensors to be created in collaboration with Prophesee, we thought that we would need to thoroughly evaluate matters including their contents.
Also, although conventional frame-based image sensors react to changes in brightness, EVS are different since they react to modulation of luminance. Therefore, we proceeded with development while being very careful about how to ensure their characteristics and quality to prevent malfunctions or problems from occurring after they are shipped to customers.
As a collaborative effort with an overseas company, how did you move forward with development?
Sakai：From the stage of prototype design, we held video conferences once or twice a week, and around two times a month we held face-to-face meetings as we alternated between gathering in Paris and Atsugi. We moved forward with discussions regarding the project progress and issues related to their characteristics and quality.
At the beginning, we learned about the development methods of conventional image sensors and then raised issues and carried out design, but as we actually continued, we found many issues that we had not seen initially, so each time we investigated them to search for solutions. Prophesee had ample knowledge related to EVS, so we were able to proceed with development while complementing each other well with the methods and technology for mass production and quality assurance independently cultivated by SSS Group.
Nitta：SSS Group developed the vision sensors, and Prophesee developed applications to operate them and kits for their evaluation. With EVS it is somewhat difficult for customers to form an image of situations where they can be applied if only products alone are proposed, so we had high expectations for the ideas of Prophesee’s applications. Under these divisions of roles, we established communication groups between SSS Group’s sales companies in each region and Prophesee’s regional business representatives, which function to share information on customer acquisition and conduct joint promotional activities. When necessary, we also go together to customer locations to provide proposals. On the other hand, we had some difficulties regarding the matter of contracts in finding common ground between Prophesee, a startup company, and SSS Group, which had created a brand up to the present.
Furukawa：We determined evaluation methods and criteria at regular meetings, but since this was basically a technology that had been developed by Prophesee first, we arranged opportunities to exchange opinions where Prophesee would bring up points that they wanted to be evaluated, while we would also suggest, for example, that it would be better to conduct measurements from certain perspectives. Additionally, we prepared an environment for evaluation at our company. It was slightly different from the environment at Prophesee, but it was customized for EVS based on the environment we had developed for CMOS image sensors. Looking from the perspective of mass production, we found issues with how to effectively reduce defects, so we proceeded with development using a stance of incorporating the expertise on CMOS image sensors we had developed in the past, and then fine-tuning it.
The most difficult point was the challenge of developing something that did not yet exist
Tell us about some difficulties you faced in development.
Sakai：The first difficulty we encountered was deciding on specifications. The specification items for EVS do not exist with conventional image sensors, so we had nothing to use as reference. Since there were almost no examples of this technology released to the world, the only references we could find were works of research literature from universities. Also, if there had already been customers who were using this technology, it would have been possible to receive requests from them and coordinate specifications, but since even customers did not have a clear image of the best ways to use EVS, we had to determine their specifications while having Nitta-san and Ihara-san envision possible situations for their use. Determining their specifications on our own meant that we ourselves could influence the goal we were trying to achieve. Although changing specifications can improve the yield during mass production, in this case we finalized the specifications while making adjustments in anticipation of the overall product, and what degree of impact those specification changes would have on business.
Nitta：The business-related point we struggled the most with was that when we introduced EVS to customers, at first everyone was very interested and curious about this unique type of sensor, but when it came to considering their applications, in many cases they did not know how to proceed, and had difficulties in moving onward to the next step. Under these conditions, we realized that using the evaluation kits and application software provided by Prophesee was very important to carry out promotions as a solution to such issues. We also discovered the importance of having these sensors be used to their fullest capability in accordance with the usage situations of each customer.
Ihara：The ways in which EVS are being used by customers who have introduced them are extremely diverse. That is why there is a need to bring out aspects of performance that align with the expectations of each individual customer. We are moving forward while carefully confirming what kinds of environments customers are thinking of using them in, and where the present issues exist. Since there are currently no uniform methods to adjust parameters to determine the properties of EVS, we are working steadily and gradually on repeated experiments and adjustments, so that they can approach the performance required by each usage situation. Rather than only supporting sensors, we are also working in cooperation with Prophesee to resolve the issues faced by customers in terms of both software and hardware. We are still in the process of development, but we have attempted to aim for the best possible solutions that we can currently find.
Also, since the structure of these sensors uses an arrangement of photodiodes, just as with frame-based image sensors, they share the same optical effects. However, there are many properties unique to EVS as well. In cooperation with design and evaluation teams who have knowledge of both sensor types, we are proceeding to examine the causes of new phenomena discovered through experiments and investigate improvement measures.
Sakai：In the course of development with the members of Prophesee, they often suggested many ideas with the goal of creating even better products, all the way up until just before deadlines. We felt that we should learn from their stance toward improvement, but also must think about the extent of the resulting impacts since there are so many people involved in development. We held discussions with them many times on how to realize their ideas while also explaining the obstacles that existed. Other points we were careful of during collaboration were to have trust in the Prophesee members, be polite and careful when explaining things to them, and build stronger relationships of trust. To Prophesee, EVS technology is the lifeline of their own company, so if they are placed in a defensive position of preventing all of their technology from being absorbed by Sony, we will not be able to hold productive discussions. We feel that having a relationship where both sides can raise their own opinions and hold discussions regardless of the scope of issues was a factor that contributed greatly to the smooth progress of this development project.
Furukawa：In terms of evaluations, the materials and documents presented by Prophesee, as well as the file formats used for evaluation, were very different from those of SSS Group. Therefore, when receiving submissions and passing them on to our evaluation team, there was a need to convert them into the formats that SSS Group used. Also, since the Prophesee members occasionally held investigations right up to the last minute, and the specifications given to us sometimes included content that needed checking, we verified each specification individually and asked for confirmation if corrections were necessary. This type of work required large amounts of time in order to increase quality.
Even the methods of evaluating these sensors contained many items that did not exist with conventional image sensors, from light sources to calculation methods, and evaluation and selection during mass production. That meant it was necessary to create new evaluation methods tailored to EVS.
When conducting tests and quality evaluations, in the past we often compared characteristics with conventional devices. With EVS, however, we needed to evaluate the response characteristics of devices to modulation of luminance, which had not been done for past sensors, so we introduced new light sources corresponding to modulated light. Furthermore, because their circuit configurations were also different from conventional image sensors, we held repeated discussions with Sakai-san and other in-house testing and evaluation experts, on how to formulate measurement and evaluation specifications related to response characteristics, and on investigating causes and countermeasures to obtain stable test results in testing environments.
Sakai：It was very difficult because since there were no previous examples, we had to start by looking at the results of tests conducted in the same way as conventional image sensors, and decide whether it was good or bad for there to be no changes in the characteristics.
EVS are vision sensors that will drive the neuromorphic field
Tell us about the solutions that can be provided by SSS’s EVS.
Sakai：Our EVS incorporate a wealth of technology for device and circuit design cultivated by SSS Group through its experience in CMOS image sensors, including photodiodes and Cu-Cu (copper-copper) connections*1. We are extremely careful about differentiating the characteristics of our technology from others, to the point that without such technology, our EVS could not be completed.
Aside from being used in devices, EVS can also produce value by building total solutions from post processing to camera and system applications. By including all of the technology SSS Group is involved in from semiconductors to sensing technology, we believe we will be able to provide the best possible devices to customers.
*1. Technology that provides electrical continuity via connected Cu (copper) pads when stacking the pixel section (top chip) and logic circuits (bottom chip). Compared with through-silicon via (TSV) wiring, where the connection is achieved by electrodes intruded around the circumference of the pixel section, this method gives more freedom in design, improves productivity, allows for a more compact size, and increases performance.
Nitta：As mentioned up until now, EVS are capable of capturing objects moving at extremely high speeds, and can output data with low latency (delay time). Their characteristics also include wide dynamic range performance allowing them to capture images ranging from dark to bright locations, and extremely small output data sizes. We believe that these properties of EVS can allow them to be anticipated for use in a wide variety of areas.
One of these is the industrial machinery area, where they can be used to monitor, inspect, and measure equipment or products moving at extremely high speeds. These types of uses are difficult with conventional industrial cameras, but EVS can be applied to them.
Because of their exceptionally wide dynamic range, they can also be used to accurately capture human movement in locations where dark areas and extremely bright areas like those exposed to sunlight would both exist on the same screen. Furthermore, although EVS can perceive human movement they will only output the outline of a human body and will not capture facial or other features, so we believe they can be applied as security cameras which take privacy into consideration.
Tell us about the future potential of EVS.
Nitta：In addition to the industrial machinery and security areas mentioned earlier, the application of EVS is anticipated in a wide variety of areas that include medical devices, consumer equipment, robotics, game peripheral equipment, and sports technology. In addition to our cooperation with Prophesee, we will fully apply all of the strengths of our company in areas from semiconductors to sensing technology, and hope to provide various applications to the market.
The System Solutions Business Division also announced the “
AITRIOS™”*2 edge AI sensing platform in October 2021. By converting EVS to smart cameras as devices to connect to this platform, and linking them to the ideas of many developers, we hope to expand the range of solutions provided and spread this technology extensively throughout the market.
*2 AITRIOS and AITRIOS logos are the registered trademarks or trademarks of Sony Group Corporation or its affiliated companies.
Sakai：The source of EVS technology is the simulation of optic nerve signals. The human brain contains roughly 100 billion neurons which respectively connect to each other to form a network. Their connecting areas are known as synapses, and roughly 1 quadrillion synapses operate the brain’s mechanisms using energy that is less than that of a single light bulb. We believe that EVS will become a core presence in the neuromorphic field, which involves systems such as AI that apply electronic circuits patterned after these neurons. Robots and other industrial machines cannot even begin to operate without carrying out sensing by means of sensors which act as their eyes. We believe that EVS will be the sensors to first be used for such purposes.
Of course, robots even up until now have been equipped with cameras for this, but much of the data they obtained was redundant since they also accounted for the work of humans performing subsequent confirmation. If systems such as automated control can be developed, control by humans will become unnecessary, so EVS, which can make even quicker judgements and output only necessary data, is anticipated to take a mainstream role.
Tell us what you have gained from this collaboration with a startup company.
Sakai：With a startup company that has a single technology as its core, it is possible that failing to develop that technology into products could simply lead to the company’s collapse. That is why when we look at Prophesee, we feel its determination and aggressive stance to develop its technology into products and release them to the world, as well as its sense of urgency and earnestness, at every turn.
In SSS Group, I have previously worked on projects where I performed research and development that unfortunately did not lead to commercialization, but at those times I was able to think about the next project and search for what could be done by SSS Group. I had of course dedicated myself diligently to each of those projects, but the stance taken by Prophesee made me feel that the attitude of always having a “next project” is a somewhat spoiled viewpoint. This led me to think in ways like “Is there something further I can do now?” and “Is there anything I can do to make things better?” which gives me great personal incentive.
Nitta：The Prophesee members continually take their own application software and evaluation kits and make approaches to customers, conclude NDAs (non-disclosure agreements) quickly, and enter phases of joint technical investigations. I believe we can learn much from their speed and adaptability in finding new customers.
I also feel that they have tremendous skill in providing information through promotions. They hold promotional activities with extremely high visual appeal, and I am always impressed by their highly-varied methods of sharing information including SNS, as well as the speed of their actions such as presenting displays in many exhibitions.
Furukawa：This project for collaboration with a startup company has actually led to more opportunities to consult and speak with each other within our own company, so I have come to find out a great deal about various technologies and other members.
Although we struggled with matters such as communicating information, we were also able to make new discoveries to proceed with our own future operations, including what methods of transmitting information we can use to thoroughly convey what we want to communicate to companies outside of SSS Group.
Ihara：Prophesee is a pioneer in event-based information processing, so they provide software and documents that reflect their abundant knowledge and expertise. They update their software quickly while identifying the functions that will be required in the future and including them in those updates, and the quality of their documents becomes even higher every time we see them. We can learn much from their development speed and their stance of wanting to always do better.
Tell us about the challenges you want to undertake from here on.
Sakai：First of all, I want to devote myself fully to this project until we can achieve mass production and shipment of this product, the IMX636.
In the future, I want to give all of my effort to connecting research and development to commercialization in the same way as with EVS. I think it would be highly beneficial for other employees in SSS Group to be able to have such experiences, so I would like to contribute to creating mechanisms allowing people to actively attempt new challenges.
Nitta：Generally speaking, I want to focus on having EVS become widespread in the market.
EVS have the characteristics of high speed, high dynamic range, and energy conservation. By fully applying these characteristics, I want to become able to contribute to the creation of a society and infrastructure that strongly consider the environment, such as by reducing carbon dioxide emissions, and I believe EVS have the potential to do so.
Furukawa：To begin with, I believe we must first properly bring EVS to mass production.
Thinking about what comes after that, I would like to become involved with the development of new technology that, like EVS, is not widely known in society, and contribute to spreading its products throughout the world.
Ihara：In addition to the manufacturing and inspection industries, I would like to expand the application range of EVS to various areas where frame-based image sensors are currently in use. If we can be selected by customers, I want to become able to provide them with solutions including EVS that use the best possible image sensors.
www.edgeimpulse.com
stocksdownunder.com
" 
what where did you find this out ??
=> https://brainchip.com/brainchip-ann...ed-ai-intellectual-property-brainchip-280519/ => May 28, 2019
