arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Computer VisionEnglishPublished

LucidLyra Turned 3‑D Reconstruction Into Queryable Semantic Space

November 1, 2026arXiv: 2610.40495v1

LucidLyra is a new framework that lets one input image turn into a full 3‑dimensional scene where geometry, appearance, language‑aligned labels, and object identities are all encoded in the same structure. The method fuses a frozen video‑generation model that produces multi‑view latent codes with pretrained 2‑D vision models to supply semantic and instance supervision. A feed‑forward Gaussian decoder maps these latents, camera rays, and image features into a jointly‑learned 3‑D Gaussian splatting scene. Evaluation on synthetic and real‑image benchmarks shows that LucidLyra outperforms all two‑stage baselines that first reconstruct and then segment. Its semantic mean intersection‑over‑union reaches 64.1 % on synthetic data and 49.8 % on real images, while instance tracking scores rise to 21.5 % and 18.5 %, respectively. Reconstruction quality (PSNR, LPIPS) stays on par with state‑of‑the‑art 3‑D reconstruction. The dynamic, monocular‑video extension retains good visual fidelity, though PSNR drops slightly compared to a reference 3‑D reconstruction, indicating a small reconstruction trade‑off. LucidLyra therefore offers a fast, annotation‑free pipeline to build queryable 3‑D worlds that can be rendered, segmented, or edited on demand.

Diagram of the LucidLyra pipeline: a frozen video model produces multi‑view latent codes, 2‑D vision models generate semantic and instance targets, and a Gaussian decoder constructs a 3‑D splatting scene encoding geometry, appearance, language‑aligned labels, and instance identities.
Figure 2. Figure 2 shows the LucidLyra framework, illustrating that a frozen video generation model supplies multi‑view latent codes, 2‑D vision foundation models provide semantic and instance supervision, and a feed‑forward Gaussian decoder maps these inputs into a joint 3‑D Gaussian splatting scene.Hahn, Oliver, Marcuzzi, Rodrigo, Srinivasan, Sangeetha Grama, Burzio, Alessandro, Bahmani, Sherwin, Huang, Jiahui et al. · arXiv:2610.40495v1 · Figure 2: · CC BY 4.0
Table comparing reconstruction, semantics, and instance metrics for LucidLyra and various baselines on SDG and ADE20K datasets; LucidLyra achieves the best semantic mIoU and instance tracking scores without degrading PSNR or LPIPS.
LucidLyra-SDG StaticLucidLyra-ADE20K
ReconstructionSemanticsInstanceReconstructionSemanticsInstance
MethodPSNR ↑LPIPS ↓mIoU ↑mAcc ↑mTrack PR ↑mTrackPR ↑Track PR25 ↑TrackPR25 ↑PSNR ↑LPIPS ↓mIoU ↑mAcc ↑mTrack PR ↑mTrackPR ↑Track PR25 ↑TrackPR25 ↑
mTrack
PR ↑
Track
PR25 ↑
mTrack
PR ↑
Track
PR25 ↑
Lyra + Lifted Teachers24.490.21160.575.322.833.224.590.21750.164.517.326.8
Lyra24.490.211––––24.590.217––––
Lyra + LangSplatV224.490.21132.452.7––24.590.21737.153.7––
Lyra + SFS24.490.21131.949.5––24.590.21735.251.3––
Lyra + Unified-Lift24.490.211––11.717.024.590.217––7.511.5
Lyra + Chorus24.490.21111.519.3––24.590.2179.316.1––
Lyra†+ IGGT––29.238.35.29.3––32.741.29.516.6
LucidLyra (Ours)25.020.18664.179.221.529.324.720.19949.865.618.529.7
LucidLyra+ (Ours)25.020.18667.981.722.031.124.720.19953.771.618.528.5
Table 1. Table 1 reports quantitative results on synthetic (SDG) and real‑image (ADE20K) benchmarks, showing LucidLyra’s higher semantic mIoU (64.1 %/49.8 %) and instance mTrackPR (21.5 %/18.5 %) compared with all two‑stage baselines while maintaining reconstruction quality.Hahn, Oliver, Marcuzzi, Rodrigo, Srinivasan, Sangeetha Grama, Burzio, Alessandro, Bahmani, Sherwin, Huang, Jiahui et al. · arXiv:2610.40495v1 · Table 1: · CC BY 4.0

Source paper

LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Computer Vision
View on arXiv
LucidLyra Turned 3‑D Reconstruction Into Queryable Semantic Space | arXiv News