A new framework automatically turns brain scans into audio by conditioning a diffusion model directly on fMRI signals
arXiv: 2611.12603v1
Scientists have created MAAS (Multi‑Area Auditory Synthesis), a method that lets a pretrained audio diffusion transformer generate sounds from brain activity without first decoding the signals into a separate audio representation. The technique projects region‑of‑interest (ROI) fMRI responses into learned brain tokens and injects them into a cross‑attention adapter while keeping the main diffusion backbone frozen.
Figure 1. Figure 1 shows how MAAS maps ROI‑wise fMRI responses to learned brain tokens and injects them into a dedicated cross‑attention adapter while the pretrained diffusion backbone remains frozen.Ciferri, Matteo, Weidler, Tonio, Ferrante, Matteo, Toschi, Nicola, Formisano, Elia · arXiv:2611.12603v1 · Figure 1: ·
The adapter combines the brain tokens with the text‑conditioning pathway. In each cross‑attention layer, the model forms a weighted sum of attention over the text prompt (an empty string) and attention over the brain tokens; a learnable scalar adjusts how strongly the brain pathway influences the generation.
On a 7‑Tesla fMRI dataset of five participants and 288 one‑second sounds, MAAS outperformed classical two‑stage fMRI‑to‑latent baselines. For the CLAP similarity metric the MAAS adapter achieved 0.614 ± 0.13 versus 0.509 ± 0.16 for the baseline. Pairwise identification accuracy rose to 85.4 ± 4.2 % compared with 70.1 ± 5.9 % for the baseline. The reconstructed audio also scored higher on Mel‑SSIM (0.297 ± 0.09) and lower on log‑mel distance (0.152 ± 0.04).
Table of quantitative results showing MAAS outperforms a two‑stage baseline across CLAP similarity, pairwise identification accuracy, Mel‑SSIM, and log‑mel distance.
Condition
CLAP sim. ↑
ID acc. ↑
Mel-SSIM ↑
Log-mel dist. ↓
Brain-conditioning baselines
MAAS (our adapter)
0.614±0.13
85.4±4.2%
0.297±0.09
0.152±0.04
MAAS (no backbone fine-tuning)
0.548±0.17
73.1±4.8%
0.283±0.08
0.171±0.04
Brain-plus-empty
0.436±0.14
63.1±5.7%
0.273±0.10
0.185±0.05
Brain-only
0.403±0.15
58.3±5.9%
0.226±0.08
0.197±0.06
Two-stage fMRI-to-latent
0.509±0.16
70.1±5.9%
0.270±0.10
0.177±0.08
Caption-based conditions
Caption-only
0.572±0.12
87.9±5.4%
0.166±0.08
0.249±0.07
Caption-plus-brain (zero-shot)
0.610±0.14
84.6±5.7%
0.275±0.09
0.167±0.06
Caption-plus-brain (retrained)
0.584±0.15
82.4±5.1%
0.311±0.11
0.143±0.04
Table 1. Table 1 lists MAAS reconstruction performance on a 7‑Tesla fMRI dataset: higher CLAP similarity, pairwise ID accuracy, and Mel‑SSIM, and lower log‑mel distance compared with a two‑stage fMRI‑to‑latent baseline.Ciferri, Matteo, Weidler, Tonio, Ferrante, Matteo, Toschi, Nicola, Formisano, Elia · arXiv:2611.12603v1 · Table 1: · CC BY 4.0
MAAS also shows that the details of its architecture matter. Removing the backbone fine‑tuning or disabling the brain‑only branch reduced performance, confirming the need for the dedicated adapter.
Because the fMRI signals are temporally smoothed by the hemodynamic response, the model operates on an integrated snapshot of auditory cortical activity. The study’s sample size—five subjects and 288 sounds—limits the generalizability of the results and future work will need larger datasets to confirm the findings.
The technique demonstrates that audio diffusion models can be conditioned directly on brain scans, opening a new pathway for neural synthesis that bypasses intermediate decoding.
A new framework automatically turns brain scans into audio by conditioning a diffusion model directly on fMRI signals | arXiv News
