Physically Based Simulation for Real-world Scenes
Session Number
CMPS 37
Advisor(s)
Zhi-Hao Lin, UIUC
Shenlong Wang, University of Illinois Urbana Champaign
Discipline
Computer Science
Start Date
17-4-2024 10:25 AM
End Date
17-4-2024 10:40 AM
Abstract
3D reconstruction for images has several applications including mixed reality, game development, and film production. Recently, several 3D reconstruction algorithms have been proposed such as Neural Radiance Fields (NERF) and 3d Gaussian Splatting, which can synthesize realistic images from novel viewpoints. However, these algorithms do not model the physical interactions of objects in scenes, which are crucial for an immersive experience in real-world applications. In the virtual content creation pipeline, the physical interactions are usually simulated on synthetic scenes and objects, and such pipelines cannot easily generalize real-world scenes. The goal of the project is to perform physically based simulations on real-world scenes. For instance, we intend to add fluid dynamics (e.g. flood), particle systems (e.g. snow), and object collisions in scenes. The key idea is to leverage 3D gaussian splatting for 3D reconstruction from real-world images, and perform physically based simulations with simulators such as Nvidia Warp and Taichi.
Physically Based Simulation for Real-world Scenes
3D reconstruction for images has several applications including mixed reality, game development, and film production. Recently, several 3D reconstruction algorithms have been proposed such as Neural Radiance Fields (NERF) and 3d Gaussian Splatting, which can synthesize realistic images from novel viewpoints. However, these algorithms do not model the physical interactions of objects in scenes, which are crucial for an immersive experience in real-world applications. In the virtual content creation pipeline, the physical interactions are usually simulated on synthetic scenes and objects, and such pipelines cannot easily generalize real-world scenes. The goal of the project is to perform physically based simulations on real-world scenes. For instance, we intend to add fluid dynamics (e.g. flood), particle systems (e.g. snow), and object collisions in scenes. The key idea is to leverage 3D gaussian splatting for 3D reconstruction from real-world images, and perform physically based simulations with simulators such as Nvidia Warp and Taichi.