First-hand information for everyone
This paper studies how foundation models—large pre-trained AI models used for image segmentation—perform when asked to outline lung nodules in CT scans. Segmentation means marking which pixels belong to a nodule so doctors can measure size and shape. The authors show that these models are sensitive to the form and placement of input “prompts,” such as a point inside the nodule or a bounding box around it. Small changes to those prompts can change the result a lot.
The team ran experiments with several recent foundation models, including SAM and MedSAM and their 3D variants (SAM2 and MedSAM2). They tested three kinds of prompts: a single point, a bounding box, and a point plus box. They used the LUNA16 dataset, which comes from the LIDC-IDRI collection and contains 888 chest CT scans with 1,186 annotated nodules larger than 3 mm. To measure mask quality they report the Dice coefficient, a common overlap score where higher means a better match to the radiologists’ masks.
They also tested robustness by perturbing prompts. For point prompts they shifted the point by a random radius. For boxes they changed box size by adding or subtracting pixels. These tests confirmed that bounding boxes generally give better results than single points, and that some models are more robust to such changes than others. For example, SAM2 achieved a Dice of about 0.89 with bounding boxes in one configuration, and several 3D model configurations reached about 0.88. Point-only prompts performed poorly in many cases (Dice below 0.1) except for MedSAM2 which reached above 0.7 in the experiments.
To reduce reliance on manual prompts, the authors built a lightweight synthetic prompt generator. It reuses the foundation model’s image encoder to produce dense features, then trains a small decoder (based on a feature pyramid network) to predict a coarse nodule mask and to output a suggested point and bounding box. The synthetic prompts are fed back into the foundation model and the final mask is compared to the ground truth during training. Loss terms include Dice and binary cross-entropy for masks, L1 losses for point and box coordinates, and a generalized Intersection-over-Union loss for boxes. This pipeline reached a Dice of 0.85, suggesting automatic prompt generation can be a promising way to run prompt-based models without manual input.
Why this matters: accurate nodule segmentation is important for lung cancer assessment and treatment planning. If foundation models can be made less dependent on perfectly placed manual prompts, they could be more practical in clinical workflows and reduce the need for time-consuming human guidance. The synthetic prompt approach also keeps the pipeline compact by learning prompts directly from the model’s own features.
Important caveats: the work focuses on prompt sensitivity and on experiments using a single public dataset (LUNA16). Performance varied by model and by the model’s configuration, so results are not uniform across all foundation models. The paper also notes that many foundation models were pre-trained on natural images, which can limit their zero-shot performance on medical CT images with different contrast and 3D context. Finally, the excerpt is limited and may not include all experimental details or external validations. These factors mean more testing, especially on diverse clinical data, is needed before clinical use.