Waabi, a start-up that specializes in autonomous driving, wants to use a novel AI model to predict the next movements of other road users. Not only machine learning will be used, but also generative AI – a novelty in this segment. The system, called Copilot4D, was trained based on data from LIDAR sensors, which use light to measure the distance to objects. If you give the model a situation – e.g. For example, if a driver recklessly enters a highway at high speed, it predicts how other vehicles in the area will move. The result is a LIDAR representation that looks 5 to 10 seconds into the future, in this case, for example, a pile-up.
Advertisement
A first version of Copilot4D is currently available. According to boss Raquel Urtasun, Waabi is already working on more precise systems that will be used in a test fleet of autonomous trucks in Texas, in which the driving software then decides how it has to react to certain situations – an integrated interpreter, so to speak.
Autonomous driving has long relied on machine learning to plan routes and recognize objects. Generative AI models that use data from the environment and then make predictions are a new level. Optimists hope it could take autonomy to a whole new level. Wayve, a competitor to Waabi, already released a comparable model last year that was trained based on driving videos collected from its vehicles. Waabi works similarly to image or video generators such as OpenAI’s DALL-E and Sora, but does not use camera data: it takes point clouds from LIDAR sensors, which visualize a 3D map of the vehicle’s surroundings, and breaks them down into parts, similar to image generators Break photos into pixels. Based on its training data, Copilot4D then predicts how the point cloud will move from the LIDAR data.
Waabi is one of the few autonomous driving companies – including competitors Wayve and Ghost – to describe its approach as “AI-first.” For Urtasun, that means developing a system that learns from data rather than having to be taught how to respond to specific situations. The start-ups are betting that their methods require fewer hours of road testing with self-driving cars. This is not without controversy – there have been several accidents in the past, for example in October 2023, when a cruise robot taxi swept away a pedestrian in San Francisco.
LIDAR instead of Camera
As mentioned, Waabi differs from its competitors in that it develops a generative model for LIDAR (“light detection and ranging”) radar technology and not for cameras. “If you want to achieve Level 4 autonomy, LIDAR is a must,” says Urtasun, meaning the level of automation at which the car should no longer need the attention of a human to drive safely. Cameras are good at showing what the car sees, but they aren’t good enough at measuring distances or understanding the geometry of the car’s surroundings, she says.
Although Waabi’s model can produce videos showing what a car sees through its LIDAR sensors, those videos are not used for training in the company’s driving simulator, which it uses to develop and test its driving model. This is to ensure that hallucinations that may still arise from Copilot4D are not transferred to the simulator operation. The underlying technology is not new, says Bernard Adam Lange, a graduate student at Stanford who has created and researched similar models. However, it is the first time that a generative LIDAR model has left the laboratory and is being prepared for commercial use.
Such a model would, Lange and others hope, enable the “brain” of an autonomous vehicle to “think” faster and more accurately. “It is a benchmark that is transformative,” he believes. “The hope is that these models can be used for downstream tasks such as detecting objects plus predicting where people or things will move next.”
Open source or not?
So far, Copilot4D can only look into the future to a limited extent. In addition, models for motion prediction generally deteriorate the more extensive they are supposed to be. According to Urtasun, what happens 5 to 10 seconds in the future is enough for most driving decisions. The current Waabi benchmark tests are based on 3 second predictions. Chris Gerdes, co-director of the Stanford Center for Automotive Research, believes this benchmark will be critical to how useful the model is in decision-making. “If the 5-second predictions are solid, but the 10-second predictions would be just about usable, there are a number of situations where the model on the road is not sufficient,” he says.
The new model also raises a question that keeps coming up in the world of generative AI: Should it be open source? The release of Copilot4D would allow researchers at universities who struggle to access large data sets to look under the hood, independently assess the security of such systems and potentially advance the field. The same applies to Waabi’s competitors. So far there are only papers here, but they don’t delve deeply enough into the subject matter, so it’s not possible to replicate it.
“We want science to also have a say in the future of self-driving cars,” says Urtasun, adding that open source models are more trustworthy. “But we also have to be a little careful when we develop our technology so that we don’t give everything away to our competitors.”
(jl)
To home page