How can missing portions of a 3D capture be effectively completed? This research paper from Google Research and UC Berkeley introduces “NeRFiller,” a novel approach for 3D inpainting, which addresses the challenge of reconstructing incomplete 3D scenes or objects often missing due to reconstruction failures or lack of observations. This approach allows precise and customizable scene completions by controlling the inpainting process through reference examples. NeRFiller is a 3D generative inpainting approach that enhances scenes or objects in 3D captures, making it an effective solution for improving 3D reconstructions.
The study explores diverse methodologies for completing missing sections in 3D scenes, ranging from traditional 2D inpainting to advanced techniques like LaMa for large-scale inpainting. It delves into probabilistic and latent diffusion models, considering 3D generation approaches involving text or images as inputs. The relevance of object removal settings is emphasized, and various baselines and datasets for 3D inpainting are evaluated. While touching on related works in video and scene editing, it primarily focuses on scene completion within the context of existing 3D scenes.
The research addresses the challenge of 3D scene completion and inpainting, emphasizing the importance of a 3D-aware and multi-view consistent approach. Distinguishing between scene completion and object removal, the focus is on generating new content within 3D scenes. The limitations of 2D generative inpainting models for 3D-consistent images are discussed. The proposed NeRFiller approach leverages the grid prior phenomenon from text-to-image diffusion models to enhance multi-view consistency in inpaints. Related works in generating 3D scenes and object removal methods are also discussed.
NeRFiller is a method utilizing a generative 2D diffusion model as inpainting before completing missing regions in 3D scenes. It tackles the challenges of diverse inpainted estimates and the lack of 3D consistency in 2D models. NeRFiller incorporates consolidation mechanisms for salient inpainted results and encourages 3D character. It utilizes iterative 3D scene optimization, extending grid inpainting to a large image collection. Baselines like Masked NeRF and LaMask are compared, demonstrating NeRFiller’s effectiveness. Evaluation includes comparisons, novel-view metrics, image quality, and geometry metrics.
NeRFiller excels in 3D scene completion, filling missing regions and removing unwanted occluders, demonstrating 3D consistency and plausibility. Compared to object-removal baselines, NeRFiller outperforms in completing missing areas. Evaluation metrics encompass NeRF, novel-view, MUSIQ image quality, and geometry metrics, showcasing its effectiveness in generating coherent and realistic 3D scenes.
In conclusion, NeRFiller is a powerful 3D inpainting tool that can accurately complete missing parts in 3D scenes. Its ability to fill gaps and remove unwanted elements outperforms object-removal baselines. The introduction of Joint Multi-View Inpainting further enhances its consistency by averaging noise predictions across multiple images. NeRFiller has demonstrated its effectiveness in completing user-specified 3D scenes by comparing them with state-of-the-art baselines. It provides a valuable framework for inpainting missing regions in 3D captures with user-defined specifications.
Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 33k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.
If you like our work, you will love our newsletter..
Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
