US based GPU manufacturer Nvidia has published a research paper outlining a new artificial intelligence (AI) model capable of converting 2D video clips into detailed 3D models.

The research found that current generative AI models struggle to capture fine details of objects, such as repetitive texture patterns, homogenous colors, and strong color variations. Nvidia’s new AI technology, called Neuralangelo, is said to surpass these problems. Neuralangelo offers high fidelity models and the ability to accurately translate the textures of complex materials from 2D videos.

 

Digital Renaissance: Neuralangelo by NVIDIA Research Reconstructs 3D Scenes from 2D Video Clips

 

How might Nvidia AI as a 3D printing tool develop?

Nvidia highlights the potential benefits for creative professionals in particular, who can quickly and easily create virtual objects using footage captured from a smartphone. Art, virtual reality, video game development, robotics, and the production of industrial digital twins have been outlined as potential applications of Neuralangelo.

The company has already demonstrated the wide ranging capabilities of their new AI technology, recreating Michelangelo’s David sculpture and a flatbed truck. Neuralangelo can also reconstruct large scale environments. Nvidia has successfully created a 3D model of the park at their Bay Area campus using birdseye video footage captured by a drone.    

Neuralangelo reconstruction from video to 3D model. Image via NVIDIA.

The 3D reconstruction capabilities Neuralangelo offers will be a high benefit to creators, helping them recreate the real world in the digital world,” commented Ming-Yu Liu, Senior Director of Research at Nvidia, and co-author of the research paper. This tool will eventually enable developers to import detailed objects – whether small statues or massive buildings – into virtual environments for video games or industrial digital twins.

 

While this new AI tool is yet to be combined with 3D printing applications, an Nvidia spokesperson told this could be a future direction. Currently, Neuralangelo’s models cannot be 3D printed.

It’s easy to imagine the potential for additive manufacturing integration with Neuralangelo.

Whether through exporting the AI generated mesh as a printable file in the future, or as part of a digital twin. Presently, digital twins are leveraged for the design and prototyping stages of product development, and further into the product life cycle.

 

Converting 2D videos into 3D models with AI

Neuralangelo employs Instant Neural Graphics Primitives (NGP) in combination with signal distance functions (SDF) to achieve high quality video conversion. NGP provides hybrid 3D grid structures with a multi resolution hash encoding. This hybrid representation significantly increases the representation power of neural fields, and can capture more fine-grained details.

Effectively, Neuralangelo takes a 2D video of an object or environment, and selects several frames that highlight different viewpoints. Once the camera position of each frame is determined, the AI creates a rough 3D model. This rough rendering is then optimized, sharpening up the details and producing a finished product with high surface quality and accuracy.    

NVIDIA HQ Park converted into a 3D model using Neuralangelo. Image via NVIDIA.

The researchers highlight two factors that are key to this approach. Firstly, using numerical gradients to compute high-order derivatives. This is said to be critical for stabilizing the optimization process. Secondly, a progressive optimization schedule is vital for recovering structures at different levels of detail. This allows for comprehensive and accurate 3D model reconstruction. 

  • TCM International - Tool Consulting and Management Group Worcon official partner
  • Feal perfection in aluminium Worcon partner
  • Iscar
  • Schunk - competence leader for gripping systems and clamping technology signed partnership contract with Worcon B2B manufacturing platform
  • Đuro Đaković Energy official Worcon partner
  • Promming d.o.o. Worcon official partner
  • Tim teh d.o.o. Worcon official partner
  • Kemperle Worcon official partner
  • CRYO-RT Worcon official partner
  • Nieros Worcon official partner
  • Tija Industry Worcon official partner
  • Pib Extra d.o.o. Worcon official partner
  • Kowe CNC Worcon official partner
  • Biam i zavarivanje
  • Celje International Industry Fair
  • Mediteran Inox d,o.o. VAŠ EKSPERT ZA METAL!
  • Kovinoplastika Worcon official partner
  • Ing Lukac Worcon official partner
  • Tehnovar Worcon official partner
  • Marex Elektrostroj official Worcon partner
  • Development and Educational Centre for the Metal Industry Cakovec Croatia is Worcon platform official partner
  • Alas info Worcon partner in CNC technology
  • Hstec
  • Strojotehnika
  • Nejci
  • Mastel CNC Technik Worcon Partner