First-hand information for everyone
Main discovery FlexPart is a unified image-to-3D part generation framework. It integrates variable-granularity geometric prompts — sparse points, 2D boxes, and masks — into a single Diffusion Transformer via gated adaptive modulation. It also uses an asymmetric geometric guidance step during training to align representations across prompt granularities.
How it works (short) Point prompts are encoded with Gaussian Fourier features and an MLP. Box prompts encode top-left and bottom-right corners with a learnable corner-type embedding. Mask prompts are processed by a small CNN with global average pooling. Missing inputs are replaced by learnable null embeddings and the modality embeddings are fused by a learnable MLP to produce a fixed part-conditioning vector. The fused geometric condition modulates the Diffusion Transformer through a Gated Adaptive Layer Normalization (Gated AdaLN) module that predicts adaptive scale, shift, and gating parameters. Image features (from DINOv2) are injected by cross-attention. A part-aware attention design treats each part as a distinct sample. Asymmetric Geometric Guidance projects point prompts (student) and box/mask prompts (teacher) into a shared visual space using a Visual Bridge. Teacher embeddings are stop-gradiented. Student and teacher features are aligned via similarity-based partial optimal transport with a learnable dustbin score and Sinkhorn optimization. "Optimal transport" here is used as a differentiable alignment objective during training.
Training and inference Training is two-stage and initialized from pretrained TripoSG weights and a pretrained 3D VAE. Stage 1 trains on objects with 1–8 parts and 512 tokens per part. Stage 2 scales to 2–16 parts and 1024 tokens per part. Training used a composite dataset derived from Objaverse and PartVerseXL and ran on 8 NVIDIA H20 GPUs with a batch size of 256. At inference, FlexPart supports flexible geometric conditions from sparse points (the mandatory minimal input) to optional boxes and masks. Higher-level cues can be incorporated hierarchically without retraining.
Concrete results (selected) Under mask conditioning on the PartVerseXL benchmark, FlexPart attains Part CD = 0.2938. That is a 16.6% reduction relative to TripoSG + SAM3D (Part CD = 0.3523) and a 10.5% reduction relative to OmniPart (Part CD = 0.3283). On whole-mesh metrics for PartVerseXL, FlexPart reports Mean CD = 0.1418, F-Score = 0.7877, and ULIP = 0.2206. This Mean CD improves on OmniPart (Mean CD = 0.1547). Joint multi-granularity training yields cross-modal synergy. Point-only inference improves by 7.9% in IoU V and 6.1% in CD compared to single-modality baselines. Under mask conditioning, joint Point+Box+Mask training reduces Part CD from 0.3224 (mask-only training) to 0.2938. Ablating the asymmetric guidance shows gains under sparse point prompts: Part CD drops from 0.3406 (no guidance) to 0.3334 (with guidance). Point IoU V rises from 0.1811 to 0.1902 and IoU B from 0.2323 to 0.2506. Mask-conditioned results remain comparable with small gains. A FlexPart variant retrained for 3D box conditioning on PartObjaverse-Tiny outperforms FullPart on most reported metrics. Whole-mesh Mean CD is 0.1385 (FlexPart) vs 0.1625 (FullPart). Part-level CD is 0.2356 (FlexPart) vs 0.2488 (FullPart). FlexPart is robust to noisy and heterogeneous prompts. Perturbing 2D box position/size by up to 20% increases Part CD from 0.3042 to 0.3171. Providing boxes for half the parts and points for the remainder yields Part CD = 0.3287. Mixing masks for half the parts and boxes for the rest yields Part CD = 0.3190, despite some combinations being unseen during training.
Jargon note Chamfer Distance (CD) is reported throughout as a measure of geometric deviation between generated and ground-truth shapes.
Significance and caveats FlexPart demonstrates that a single, end-to-end model can accept heterogeneous geometric prompts and improve part-level 3D generation while preserving whole-mesh quality. The asymmetric guidance helps transfer structure from dense annotations to sparse inputs during training. Caveats supported by the evaluation: occlusion handling remains challenging with single-view 2D inputs; the model can place occluded parts incorrectly because single views provide limited depth cues. The part-level evaluation protocol computes metrics only over CD-based Hungarian-matched part pairs and excludes unmatched ground-truth and generated parts without additional penalties. This affects how completeness and unmatched errors appear in the reported part-level numbers.