arXiv: Ethical and Technical Limits of Deepfake Speech Datasets

This publication from arXiv, dated June 2026, presents a critical analysis of the ethical and technical limitations inherent in current deepfake speech datasets used to train AI systems. While not a regulatory mandate itself, the paper highlights a significant gap in data governance and model transparency, directly impacting compliance with the EU AI Act’s requirements for high-risk AI systems, particularly around bias, accuracy, and transparency. The authors demonstrate that many widely-used speech datasets contain synthetic voices that are not properly labeled, lack demographic diversity, and can produce outputs that are indistinguishable from real human speech, raising serious concerns about deception, fraud, and non-consensual use.

Organizations developing or deploying AI systems that generate or analyze human speech are most affected. This includes sectors such as financial services (voice authentication), healthcare (telemedicine), customer service (chatbots and voice assistants), and media (content creation). Compliance teams in these sectors must now reassess their training data provenance and model documentation to ensure they can demonstrate compliance with the AI Act’s requirements for data quality, transparency, and risk management.

Compliance teams should immediately audit any deepfake or synthetic speech datasets used in their AI pipelines, verifying that all synthetic data is clearly labeled and that the dataset’s demographic composition is documented. They should also update their technical documentation and risk assessments to address the specific risks of speech spoofing and impersonation. Finally, teams should monitor the EU’s upcoming implementing acts on synthetic data and consider engaging with standardisation bodies to develop sector-specific guidelines for ethical speech dataset creation.