First-hand information for everyone
Diffuse‑model creators often train on vast image collections that include private or copyrighted pictures. A membership‑inference attack (MIA) tells whether a certain image was part of a model’s training set. Current MIAs score images by how well the model can reconstruct them. But images vary in how difficult they are to reconstruct, so an image that was never in training can sometimes score lower than a complex, memorised image.
To cleanly separate true memorisation from image difficulty, researchers introduced Reference‑Model Calibration (RMC). RMC compares a fine‑tuned model’s loss to that of an earlier pretrained checkpoint. By subtracting the reference loss, RMC isolates newly memorised data, enabling near‑perfect auditing even when no text captions are available.
When applied to realistic Stable Diffusion fine‑tunes, RMC achieved over 99 % true‑positive rates with only two model queries per image – a drastic improvement over traditional attacks that need 15 queries and captions. This demonstrates that simple subtraction of an earlier checkpoint’s loss can erase the confounding effect of image complexity.
| Pokémon | MS COCO | Flickr | ||||||||
| Method | AUC | ASR | TPR@1%FPR | AUC | ASR | TPR@1%FPR | AUC | ASR | TPR@1%FPR | |
| CLiD | (q=15) | 96.98 | 91.83 | 64.75 | 95.55 | 88.72 | 44.04 | 95.22 | 88.16 | 47.14 |
| RMC | (q=2) | 99.44 | 96.51 | 90.63 | 99.88 | 98.52 | 96.60 | 99.80 | 98.14 | 96.86 |
| (q=20) | 99.73 | 97.84 | 96.64 | 99.94 | 99.04 | 98.52 | 99.96 | 99.50 | 99.82 | |
A second technique, Temporal Self‑Calibration (TSC), tackles difficulty along the diffusion trajectory. TSC does not require a reference checkpoint; instead, it compares losses at different noise levels using a single model. On the standard CIFAR‑10 benchmark, TSC reached a true‑positive rate of 62.3 % at 1 % false‑positive rate – more than double the 30.9 % achieved by the best existing baseline.
| CIFAR-10 | Tiny ImageNet | STL-10-U | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | AUC | ASR | TPR@1%FPR | AUC | ASR | TPR@1%FPR | AUC | ASR | TPR@1%FPR | |
| Naive | (q=20) | 85.39 | 78.21 | 7.35 | 64.37 | 60.52 | 2.93 | 64.15 | 60.50 | 2.51 |
| SecMI | (q=12) | 88.24 | 80.91 | 10.20 | 66.37 | 61.88 | 2.91 | 68.12 | 63.33 | 3.03 |
| PIA | (q=2) | 91.90 | 84.88 | 25.80 | 68.99 | 64.25 | 4.43 | 69.46 | 64.51 | 4.08 |
| PIAN | (q=2) | 89.26 | 83.32 | 30.90 | 67.04 | 63.44 | 6.10 | 66.72 | 63.36 | 4.32 |
| TSC | (q=15) | 95.91 | 90.15 | 62.31 | 78.63 | 72.18 | 12.27 | 77.84 | 71.29 | 9.57 |
These findings highlight that difficulty calibration, both across training time and diffusion time, is crucial for accurate membership inference. However, RMC’s reliance on a pre‑trained checkpoint limits its use when that checkpoint is unavailable. Future work will need to extend TSC‑style calibration to broader settings where only the final generated images are exposed.
The table below illustrates RMC’s near‑perfect performance on real‑world Stable Diffusion fine‑tunes, showing its superiority over caption‑based attacks and its robustness with minimal query budgets. The second table summarizes TSC’s advantage on standard diffusion‑model benchmarks, reporting the two‑time higher true‑positive rate at 1 % false‑positive rate on CIFAR‑10.