In the late 1980s, one of the inventors of Very large-scale integration (VLSI), Professor Carver Mead of the California Institute of Technology in the United States first proposed the concept of neuromorphism, imitating the use of CMOS analog circuits. Biological retina, build a system with biological computing characteristics, and carry out research in this area with students.
Later, this technology spread from the United States to Europe and spread around the world. The research of scientists and entrepreneurs all over the world pointed to the same problem: how to make the retina based on CMOS analog circuits closer to the human eye ?
Christoph Posch from Austria is one of the scholars who pay attention to neuromorphic technology. He has been engaged in neuromorphic related work since around 2000, and co-founded Prophesee, a company that develops neuromorphic vision chips, with Luca Verre. In the forefront of the world.
Not long ago, the author had an exchange with Luca. Luca talked about the latest development of neuromorphic vision chip technology: there have been successful cases in the field of artificial retina, and he is currently looking for a broader development space in consumer electronics such as mobile phones.
Event Cameras for the Visually Impaired
Like the neural reflexes of the human body, neuromorphic circuits are mainly divided into two parts, the sensor that receives the signal and the nerve center that senses the signal. Therefore, neuromorphic chips such as human eyes and brains are derived. The work of simulating the human eye is mainly undertaken by the event camera.
Different from traditional vision sensors, event cameras are a new type of vision sensors inspired by biotechnology, sometimes called dynamic vision sensors. In biological vision, some visual cells are sensitive to dynamic things. Following this feature of the human eye, an event camera for sensitively capturing moving objects was born.
The core of the event camera is that each pixel has an independent light source. When the brightness of a certain pixel changes beyond the set threshold, pulse data will be generated and output, so it can complete some frame-based traditional cameras. Tasks such as high-speed motion, high dynamic range mapping, etc., have the advantages of high dynamic range and low latency.
Christoph, who has more than two decades of research experience in neuromorphic vision technology, saw the potential of event cameras and decided it was time to start a company that would take this technology from the lab to the market.
Prophesee CEO Luca said that about eight years ago, Luca was studying for an MBA in Paris and wanted to seek some new breakthroughs in his career. It was during this period that Luca met Christoph. After listening to Christoph’s introduction and analysis of neuromorphic vision technology, Luca was greatly inspired and knew that this technology would bring great value to the market, so he decided to start a business together with Christoph without hesitation. A few days later, And so Prophesee was born.
Initially, both Christoph and Luca believed that the essence of this technology is to develop an artificial model that simulates the human eye, so it can be used to simulate the working principle of the human retina and help blind people with impaired retinal functions restore their vision. Technology applied in the medical field.
“In a way, that was one of the inspirations for our decision to start this company,” says Luca.
Therefore, in the early days of Prophesee’s establishment, the first generation of products released was mainly used to help the visually impaired and blind restore their vision. Luca said that Prophesee’s first-generation products are mainly used in the applications of bioelectronics company Pixium Vision and biopharmaceutical company GenSight, and have achieved good experimental results. “Now the two companies are trying to find ways to reduce the complexity of the surgery as much as possible, hoping to apply this technology to more patients as soon as possible,” Luca explained.
As the world‘s leading company in neuromorphic vision technology, Prophesee has confirmed the potential of event cameras in simulating the human eye. In order to further promote the development of neuromorphic technology, Prophesee decided to continue to focus on medical applications while focusing on business. Many places have turned to the mobile phone terminal and electronic fields, as well as the industrial field.
From the human eye to the mobile phone, a powerful tool for image improvement
In fact, compared with the progress made in simulating human eyes, neuromorphic vision technology is developing faster in the fields of industry and consumer electronics. In the case of difficult CMOS pain points and severe homogenization of smartphones, it will gradually evolve into the future competition of mobile phone manufacturers. part of the game.
Tracing back the development of image sensors, CCD devices with high pixels and high manufacturing difficulty were the mainstream at first, and were gradually replaced by CMOS with lower power consumption and cost.
However, CMOS also has its own problems. Because it mainly uses the time-based technology of photocurrent to achieve exposure, it takes a long time from exposure to digital-to-analog conversion and then to data transmission, and the power consumption is high, which affects the frame rate of the image, frame to frame. The long interval between images causes signal loss, which increases the machine’s judgment of the image.
At the same time, the CMOS image sensor outputs 8-12bit signals, which has a large amount of data transmission, high energy consumption, slow reading, storage space requirements, and high energy consumption for large-scale image processing.
During the development of CMOS image sensors, many manufacturers have tried to solve these pain points. However, based on the working principle of using the shutter to control all pixels to perform integral exposure at the same time, the frame rate, data volume and The three parameters of dynamic range are difficult to balance.
For example, in high dynamic range shooting, multiple exposure algorithms are often used, that is, multiple frame images with different exposure times are fused to form a high dynamic range image, which will inevitably lead to a decrease in frame rate, more data calculations and larger The system delay is not applicable in scenarios that require short response time and capture fast changes.
“When we usually use mobile phones to shoot moving objects, especially in low light conditions, we will find that the edges of the subjects are not clear. This is because the current mobile phone lens will obtain information at a fixed time node. When we press the camera Shutter, the sensor starts to enter light, and the shutter closes. During this exposure time, if the subject moves, the resulting photo will be blurred.” Luca said this is a common problem with traditional lenses.
Event cameras with dynamic capture capabilities can better solve these problems. Therefore, in recent years, Sony and Samsung have entered event cameras one after another, rushing to make a layout on the eve of the CMOS revolution and improve the imaging capabilities of mobile phones.
As one of the few event camera startups and a partner of Sony, Prophesee has also made new progress in the field of smartphones in view of its experience in medical applications, launching photo deblurring functions and has already cooperated with Qualcomm.
Luca showed the comparison between the photos taken by the Prophesee event camera chip and the project of the traditional lens, and it can be observed that the rackets and balls in motion captured by the former are clearer.
(Source: Video Screenshot)
“We have also cooperated with some major mobile phone manufacturers, and the goal is to achieve mass production in 2024,” Luca said.
The landing of AR, VR and smart driving applications will follow closely behind
In addition to the application in the field of mobile phone lenses, event cameras can also play a role in some hot tracks. Luca said that AR, VR and smart driving are all target markets for event cameras.
In the field of AR and VR, event cameras are expected to solve the problem of eye tracking and help users reduce the feeling of vertigo when using AR and VR devices. A year ago, Meta released an applied research on solving the eye movement problem at the WACV conference, which is based on the Prophesee event camera to detect changes in the scene.
“The eye moves very fast, so a very high-speed lens is required to track the movement of the eye, but traditional lenses are based on frames, and fast eye movement requires a very high frame rate for shooting, which will generate a lot of data, increase energy consumption and data processing. Time. But the event camera can only track the moving object itself in response to changes in information, and achieve high-speed tracking while maintaining low power consumption and low latency.” Luca explained.
In terms of smart driving, Luca said that Prophesee’s current application research mainly focuses on two aspects, one is to monitor the driver’s behavior in the cockpit, and to monitor the driver’s attention while driving, for example, by monitoring the speed and frequency of blinking, Determine whether the driver has a decrease in concentration.
Second, in terms of assisted driving, it can help detect obstacles in front of the car, especially in low-light or strong-light environments, providing more accurate obstacle detection through a higher dynamic range and improving driving safety.
At present, Prophesee has also cooperated with Bosch, Intel and Renault Nissan to carry out more experiments in the automotive field.
Will such an advanced neuromorphic vision technology be more expensive than CMOS? Luca said that Prophesee’s sensor is also based on traditional CMOS technology, which is comparable in terms of process cost. However, as a brand new technology, it still needs Only by further expanding R&D investment and expanding market demand can we truly achieve cost reduction and efficiency increase.
(This article is reproduced with the authorization of Leifeng.com; the source of the first picture: Prophesee)