Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding

Boyu Cai^1,2 Li Yang^2* Yan Xu³ Wei Liu² Nian Liu² Sikui Zhang² Yan Wang⁴ Chunfeng Yuan² Weiming Hu^1,2

¹ShanghaiTech University ²Institute of Automation, Chinese Academy of Sciences ³The Chinese University of Hong Kong ⁴Deepeleph Intelligent Technology

^* Corresponding author

ECCV 2026

Overview of dynamic-robust photometric-semantic reconstruction

TL;DR: We jointly reconstruct clean novel views and open-vocabulary semantics from sparse, unposed observations of dynamic scenes.

Abstract

The integration of novel view synthesis (NVS) and open-vocabulary segmentation (OVS) has recently yielded powerful feed-forward 3D foundation models. However, their inherent reliance on static-scene assumptions leads to severe misalignment of spatial features in unconstrained dynamic environments. To bridge this critical gap, we propose SPAR, a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation. Furthermore, we introduce a dynamic-region-aware end-to-end training paradigm that structurally couples motion estimation with multi-view visual and semantic learning. This unified approach enables the network to inherently resolve motion conflicts and distill multi-view consistent, temporally stable scene representations from dynamic inputs.

Extensive experiments on the challenging D-RE10K benchmark demonstrate that SPAR achieves state-of-the-art performance. Our end-to-end approach yields a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views, respectively. Despite being trained in a self-supervised manner, our model achieves an mIoU of 88.5% for motion mask prediction. Semantic synthesis learning also consistently enhances photometric fidelity in novel-view rendering.

Method

SPAR framework for photometric-semantic reconstruction — SPAR couples a Cross-View Dynamic Region Predictor (CV-DRP) with joint photometric-semantic scene encoding. Dynamic regions are suppressed before static multi-view evidence is aggregated and rendered into target-view RGB images and semantic features.

Results

Dynamic Novel-View Synthesis on D-RE10K

Input Views	PSNR ↑	SSIM ↑	LPIPS ↓
2	19.97	0.627	0.339
3	22.15	0.702	0.283
4	23.33	0.739	0.263

Novel-view synthesis and semantic segmentation comparisons — Qualitative comparison of novel-view synthesis and semantic segmentation on dynamic scenes.

Dynamic region prediction comparisons — Dynamic region prediction on diverse multi-view sequences. The public evaluator uses the raw CV-DRP output shown as Ours w/o Refine; SAM2 refinement is disabled.

Takeaways

Semantics Helps Reconstruction

Semantic learning supplies structural cues that improve photometric scene reconstruction and novel-view rendering.

Reconstruction Reveals Dynamics

Novel View Synthesis

Semantic Segmentation

BibTeX

@inproceedings{cai2026spar,
  title     = {Dynamic-Robust Photometric-Semantic Reconstruction for
               Open-Vocabulary 3D Scene Understanding},
  author    = {Cai, Boyu and Yang, Li and Xu, Yan and Liu, Wei and
               Liu, Nian and Zhang, Sikui and Wang, Yan and
               Yuan, Chunfeng and Hu, Weiming},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgements

This project builds upon LVSM, RayZer, and WildRayZer. We thank their authors for releasing the code.