First-hand information for everyone
LucidLyra is a new framework that lets one input image turn into a full 3‑dimensional scene where geometry, appearance, language‑aligned labels, and object identities are all encoded in the same structure. The method fuses a frozen video‑generation model that produces multi‑view latent codes with pretrained 2‑D vision models to supply semantic and instance supervision. A feed‑forward Gaussian decoder maps these latents, camera rays, and image features into a jointly‑learned 3‑D Gaussian splatting scene. Evaluation on synthetic and real‑image benchmarks shows that LucidLyra outperforms all two‑stage baselines that first reconstruct and then segment. Its semantic mean intersection‑over‑union reaches 64.1 % on synthetic data and 49.8 % on real images, while instance tracking scores rise to 21.5 % and 18.5 %, respectively. Reconstruction quality (PSNR, LPIPS) stays on par with state‑of‑the‑art 3‑D reconstruction. The dynamic, monocular‑video extension retains good visual fidelity, though PSNR drops slightly compared to a reference 3‑D reconstruction, indicating a small reconstruction trade‑off. LucidLyra therefore offers a fast, annotation‑free pipeline to build queryable 3‑D worlds that can be rendered, segmented, or edited on demand.

| LucidLyra-SDG Static | LucidLyra-ADE20K | |||||||||||||||||||
| Reconstruction | Semantics | Instance | Reconstruction | Semantics | Instance | |||||||||||||||
| Method | PSNR ↑ | LPIPS ↓ | mIoU ↑ | mAcc ↑ | mTrack PR ↑ | mTrack | PR ↑ | Track PR25 ↑ | Track | PR25 ↑ | PSNR ↑ | LPIPS ↓ | mIoU ↑ | mAcc ↑ | mTrack PR ↑ | mTrack | PR ↑ | Track PR25 ↑ | Track | PR25 ↑ |
| mTrack | ||||||||||||||||||||
| PR ↑ | ||||||||||||||||||||
| Track | ||||||||||||||||||||
| PR25 ↑ | ||||||||||||||||||||
| mTrack | ||||||||||||||||||||
| PR ↑ | ||||||||||||||||||||
| Track | ||||||||||||||||||||
| PR25 ↑ | ||||||||||||||||||||
| Lyra + Lifted Teachers | 24.49 | 0.211 | 60.5 | 75.3 | 22.8 | 33.2 | 24.59 | 0.217 | 50.1 | 64.5 | 17.3 | 26.8 | ||||||||
| Lyra | 24.49 | 0.211 | – | – | – | – | 24.59 | 0.217 | – | – | – | – | ||||||||
| Lyra + LangSplatV2 | 24.49 | 0.211 | 32.4 | 52.7 | – | – | 24.59 | 0.217 | 37.1 | 53.7 | – | – | ||||||||
| Lyra + SFS | 24.49 | 0.211 | 31.9 | 49.5 | – | – | 24.59 | 0.217 | 35.2 | 51.3 | – | – | ||||||||
| Lyra + Unified-Lift | 24.49 | 0.211 | – | – | 11.7 | 17.0 | 24.59 | 0.217 | – | – | 7.5 | 11.5 | ||||||||
| Lyra + Chorus | 24.49 | 0.211 | 11.5 | 19.3 | – | – | 24.59 | 0.217 | 9.3 | 16.1 | – | – | ||||||||
| Lyra†+ IGGT | – | – | 29.2 | 38.3 | 5.2 | 9.3 | – | – | 32.7 | 41.2 | 9.5 | 16.6 | ||||||||
| LucidLyra (Ours) | 25.02 | 0.186 | 64.1 | 79.2 | 21.5 | 29.3 | 24.72 | 0.199 | 49.8 | 65.6 | 18.5 | 29.7 | ||||||||
| LucidLyra+ (Ours) | 25.02 | 0.186 | 67.9 | 81.7 | 22.0 | 31.1 | 24.72 | 0.199 | 53.7 | 71.6 | 18.5 | 28.5 | ||||||||