Nvidia's new AI converts photos into a 3D scene within seconds

Nvidia may have come up with a critical tool just at a time when the technology world is going big on immersive experiences.

Thanks to the technology we use in our everyday lives, we are able to capture moments in the world around us in the form of photographs. The process, however, can only create a 2D recollection of the 3D world we see. Scientists have now been trying to recreate 3D instances from these 2D captures and Nvidia may have come up with groundbreaking technology in this regard.

Called Instant NeRF, the company showcased the new ability at Nvidia GTC, a global conference for AI developers. The technology is an improvement to NeRF or neural radiance field, which is used to develop realistic 3D scenes from a collection of 2D images. For this, a process called inverse rendering is used wherein artificial intelligence is employed to estimate how light will behave on or around an object in the real world.

Nvidia says that its research team has now developed a superfast version of NeRF. As per a blog detailing the development, Nvidia says that its new technology "accomplishes this task [inverse rendering] almost instantly." Hence the title "Instant NeRF."

With this, Nvidia says that its technology is "one of the first models of its kind to combine ultra-fast neural network training and rapid rendering." As mentioned in its blog, Instant NeRF is able to learn a high-resolution 3D scene in seconds, and "can render images of that scene in a few milliseconds." This is touted to be "more than 1,000x speedups" than regular NeRF processes seen to date.

So how does it work?

Nvidia says that the AI model simply requires seconds to train on still photos in order to come up with a 3D rendition. For this, it feeds on "a few dozen still photos" of the object, as well as the data on the camera angles these shots were taken from. Within "tens of milliseconds", Instant NeRF can then render a 3D scene based on the 2D photos.

The technology seems impressive right off the bat. Nvidia knows this and has not shied away from stressing its importance during the presentation and in the blog. David Luebke, vice president for graphics research at NVIDIA, says “Instant NeRF could be as important to 3D as digital cameras and JPEG compression have been to 2D photography." The comment reiterates the level of speed the new AI model brings to 3D renders generated from still photos.