First-hand information for everyone
This paper looks at how speech deepfake detectors learn from different types of fake-speech attacks. In common training setups, many spoofing methods are lumped together as one “fake” class and final performance is reported as a single number called Equal Error Rate (EER). The authors show that this hides important differences: some attack types steer the model’s learning much more than others and cause uneven performance across attacks.
To study this, the researchers used benchmark datasets that contain several spoofing systems. They focus in detail on ASVspoof 2019 Logical Access, whose training split has six attacks: four text-to-speech (TTS) systems and two voice conversion (VC) systems, each with the same number of samples but more TTS types overall. They created eight training setups — one with no omissions, one that reduces samples to balance TTS vs VC, and six that omit particular attack types — and trained representative detectors (RawNet2, AASIST, and an SSL-based model where SSL stands for self-supervised learning) under each setting.
Instead of only reporting the aggregate EER, the team measured EER per attack at test time and tracked two signals during training: the model’s loss (the optimization objective) and predictive entropy (a measure of how uncertain the model is about a sample). They used how much an attack changes EER when it is removed (omission sensitivity) together with the loss and entropy patterns to decide which attacks drive learning. Some attacks had low loss and low entropy but high omission sensitivity; these they call high-impact attacks because they tightly shape the learned boundary between real and fake speech.
Finding that a few attacks can dominate learning, the authors propose a simple curriculum to reduce that dominance. The method first trains on low-impact attacks and then gradually introduces high-impact attacks. To avoid forgetting what was learned earlier, the curriculum replays a held-out set of low-impact samples while using a teacher–student regularization step (the teacher’s outputs help keep the student model aligned with earlier knowledge). The paper reports experiments on ASVspoof 2019, ASVspoof 2021, ASVspoof 5, and a Fake-or-Real set, showing improved overall robustness and a more balanced per-attack performance compared with standard multi-attack training.
Important caveats are noted. The analysis is tied to benchmark datasets and to the particular attacks included there; balanced sample counts alone did not fix the imbalance, so attack identity matters beyond mere quantity. The curriculum reduces but does not claim to eliminate all uneven learning, and the TeX excerpt here may be incomplete. The authors say they release code, models, and the balanced training configurations to help reproduce and extend their work.