Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs

Abstract

Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count. We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm. Single-pool per-layer methods suffer \emph{cross-component budget collapse}, dragging word error rate (WER) far from flat global clipping or collapsing

Categories

@misc{luque2026componentaware,
  title = {{Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs}},
  author = {Jordi Luque and Fernando López and Aleix Sant},
  year = {2026},
  month = sep,
  eprint = {2609.11762},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.11762}
}