Digital sculptor: Neuralangelo transforms 2D videos into 3D sculptures

The art of the 16th century meets the technology of the 21st century: With these words, the AI ​​department of the chip manufacturer Nvidia is promoting a new software called Neuralangelo, which uses artificial intelligence to create detailed 3D models of iconic sculptures, ordinary objects or even complete buildings can. All she needs is a video filmed with her smartphone or drone.

Neural surface reconstruction is the name of the process in which 3D models are created on the basis of 2D content such as photos and videos: Appropriate software analyzes the input and uses different perspectives to calculate how the object – such as a sculpture or a vehicle – is in 3D look.

Like a sculptor looking at his object from different angles, Neuraangelo analyzes the frames of a video to get a clear understanding of an object’s depth, size and shape. An initial 3D representation of this is then constructed and further refined to reflect the intricate details. As the researchers from Nvidia write in their paper, Neuraangelo is able to capture and display in detail even complex surfaces such as roof shingles or marble slabs, which previous models often fail to do.

More detail thanks to NeRFs

This is made possible, among other things, by a development in the field of artificial intelligence called Neural Radiance Fields, or NeRFs for short. These are 3D models that can be created with just a few dozen 2D photos. The technology is being tested by Google, among others, to render products in 3D in the search engine. With Instant NGP, Nvidia presented its own NeRF model for beginners last year, which is now also implemented in Neuralangelo and ensures that even the finest details are recognized.

But what is that good for? “Neuralangelo’s 3D reconstruction will help recreate the real world in the digital world,” says Ming-Yu Liu, co-author of the paper. Areas of application for the technology could be virtual worlds such as video games or the metaverse, in which digital twins of real objects and buildings could be created with comparatively little effort, without first having to model them in painstaking detail work. But interior architects, industrial designers and robotics companies can also use the technology to simulate the events of a plant or building on the computer.

At the Conference on Computer Vision and Pattern Recognition (CVPR), taking place June 18-22 in Vancouver, Nvidia plans to present more details of Neuraangelo – along with 30 other projects revolving around artificial intelligence. With this, the company underscores its claim that it is no longer just a graphics chip manufacturer, but an “AI Computing Company”.

(bsc)