3D Snapshot : Invertible Embedding of 3D Neural Representations in a Single Image

3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this...

Ausführliche Beschreibung

Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on pattern analysis and machine intelligence. - 1979. - PP(2024) vom: 07. Juni
1. Verfasser: Lu, Yuqin (VerfasserIn)
Weitere Verfasser: Deng, Bailin, Zhong, Zhixuan, Zhang, Tianle, Quan, Yuhui, Cai, Hongmin, He, Shengfeng
Format: Online-Aufsatz
Sprache:English
Veröffentlicht: 2024
Zugriff auf das übergeordnete Werk:IEEE transactions on pattern analysis and machine intelligence
Schlagworte:Journal Article
LEADER 01000caa a22002652 4500
001 NLM373363990
003 DE-627
005 20240626232514.0
007 cr uuu---uuuuu
008 240608s2024 xx |||||o 00| ||eng c
024 7 |a 10.1109/TPAMI.2024.3411051  |2 doi 
028 5 2 |a pubmed24n1452.xml 
035 |a (DE-627)NLM373363990 
035 |a (NLM)38848236 
040 |a DE-627  |b ger  |c DE-627  |e rakwb 
041 |a eng 
100 1 |a Lu, Yuqin  |e verfasserin  |4 aut 
245 1 0 |a 3D Snapshot  |b Invertible Embedding of 3D Neural Representations in a Single Image 
264 1 |c 2024 
336 |a Text  |b txt  |2 rdacontent 
337 |a ƒaComputermedien  |b c  |2 rdamedia 
338 |a ƒa Online-Ressource  |b cr  |2 rdacarrier 
500 |a Date Revised 25.06.2024 
500 |a published: Print-Electronic 
500 |a Citation Status Publisher 
520 |a 3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5MB), and embedding each 3D scene takes up only 160KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse 
650 4 |a Journal Article 
700 1 |a Deng, Bailin  |e verfasserin  |4 aut 
700 1 |a Zhong, Zhixuan  |e verfasserin  |4 aut 
700 1 |a Zhang, Tianle  |e verfasserin  |4 aut 
700 1 |a Quan, Yuhui  |e verfasserin  |4 aut 
700 1 |a Cai, Hongmin  |e verfasserin  |4 aut 
700 1 |a He, Shengfeng  |e verfasserin  |4 aut 
773 0 8 |i Enthalten in  |t IEEE transactions on pattern analysis and machine intelligence  |d 1979  |g PP(2024) vom: 07. Juni  |w (DE-627)NLM098212257  |x 1939-3539  |7 nnns 
773 1 8 |g volume:PP  |g year:2024  |g day:07  |g month:06 
856 4 0 |u http://dx.doi.org/10.1109/TPAMI.2024.3411051  |3 Volltext 
912 |a GBV_USEFLAG_A 
912 |a SYSFLAG_A 
912 |a GBV_NLM 
912 |a GBV_ILN_350 
951 |a AR 
952 |d PP  |j 2024  |b 07  |c 06