Event-oriented CMOS image sensor will be used for lidar

According to James Consulting, in the rapidly growing market of factory automation, IoT and autonomous vehicles, CMOS image sensors seem to play a role no longer as human consumer goods, but as sensors that allow machines to acquire data and make it understand the world.

Pierre Cambou, head of the MEMS & Imaging Division of Yole, a well-known French market research and technical analysis agency, commented: "The sensing function of CMOS image sensors will gradually exceed its own imaging capabilities," and boldly predicts that "by 2030, half of CMOS image sensors will serve the realm of perception."

Prophesee SA (formerly Chronocam), headquartered in Paris, France, is a revolutionary leader. As an advanced neuro-visual system design company, an event-based approach to sensing and processing has been proposed. Prophesee's bionic vision technology has always been considered to be substantially different from traditional machine vision, in a dangerous "pre-stage". But Prophesee co-founder and CEO Luca Verre told us: "In the near future, this idea will prove to be wrong."

Prophesee co-founder and CEO Luca Verre

In a face-to-face interview with Luca Verre, we learned that Prophesee has received B+ round financing (Prophesee has raised $40 million in the past three years). Prophesee has signed a cooperation agreement with a major consumer electronics company (without a specific name). Most importantly, Prophesee is actively moving the neuro-vision system out of the usual technical concept into a reference system for developers.

Prophesee's first reference design is used in VGA resolution applications, and the Prophesee provides an asynchronous time-based image sensor (Asynchro nous Time-ba sed Image Sensor, hereinafter referred to as ATIS) chip and software algorithms. The ASIC will be built by an Israeli foundry partner (which we suspect is likely to be Tower Jazz).

Prophesee is temporarily reluctant to provide detailed instructions for ASIC and reference design specifications. The product is scheduled to be officially released in the coming weeks. Nonetheless, reference designs provide system designers with the opportunity to witness and experience ATIS's role in data sensing, and it has been proven that the startup has completed a milestone. ATIS features high instantaneous resolution, low data rate, high dynamic range, and low power consumption.

Camera = bottleneck

Manufacturers of machine vision systems, whether smart factories, the Internet of Things, or autonomous vehicles, have focused on event-based approaches, which were promoted by Prophesee co-founders Ryad Benosman and Christoph Posch.

It captures all the detailed visual information captured by traditional cameras, and Verre believes that "the camera becomes a technical bottleneck." There is no doubt that the camera is the most powerful sensing device. However, for visual data from automated systems, surveillance cameras or highly automated vehicles, the processing speed of the camera may be slowed down.

But when it comes to autonomous driving, Verre believes that the central processing system inside the vehicle is “bombed” by data from cameras, laser radar , radar and other visual sources. The key to managing this type of overload is how to best reduce the amount of raw data from the sensor. Sensors can only capture data and bring extra useless visual information.

Prophesee explained to interviewers that Prophesee's event-driven vision sensor was inspired by biology. This perception stems from the co-founder's research on human eyes and brain work. Ryad Benosman, one of Prophesee's founders, tells us that human eyes and brain "do not record visual information based on a series of frames." Creatures are more complex than you think. "Humans capture the things of interest in time and space and send this information to the brain effectively." This is also the main role of Prophesee's ATIS.

In summary, Prophesee's ATIS provides everything that frame-based image sensing can't. In the eyes of another co-founder, Christoph Posch, "the frame-based approach leads to redundant recording of data, which leads to high power consumption," he said. "The consequences are inefficient data rates and expanded storage. Frame-based video, running at 30 frames per second or 60 frames or higher, can cause distortion of the captured image."

A method for event based laser radar (the LiDAR) of

Verre revealed to us that Prophesee is exploring the possibility of using event-driven methods for other sensors such as lidar and radar. Verre assumes: “If we can guide the lidar to capture only the relevant and necessary data, would it be attractive?” If you can do this, you can not only speed up data collection, but also reduce the amount of data that needs to be processed.

Phrophesee is currently evaluating the idea, and Verre says the company will need months to evaluate before it can reach a conclusion. But he added: "We are very confident that we can succeed." When asked about new ways to expand event-oriented approaches to other sensors, Yole analyst Cambou believes that "the combination of event-based cameras and lidar (providing 'Z' information) is a very interesting thing. ”

It should be noted that the problem with conventional lidars is lower resolution than typical high-end industrial cameras, limited by resolution. Cambou believes that event-oriented approaches can improve lidar performance, especially when events are approaching quickly, such as when pedestrians suddenly appear in front of autonomous vehicles. The downside is that the lidar hardware must be changed accordingly. Obviously, for Phrophesee, a strong support from a laser radar company is needed to facilitate this new approach.

Cambou said, "Of course, this is always a problem for technology start-ups." He pointed out that Mobileye needs leading automakers like Volvo and Tesla (the technology will become mainstream and needs to be more and more widely accepted. ); Movidius (acquired by Intel) requires the Dow Jones Index to pave the way for success. “Prophesee will need a strong partner to facilitate the adoption of the solution,” Cambou said.

Cambou added: "Considering the market drivers in the robotics sector (safety first, technology-driven, non-cost first), this should be possible." Although Cambou expressed his dependence on giants (such as Google) relying on technological start-ups The company's concerns, he also said that such a small market consumption will not bring much trouble.