Spire-LM: Extending LLMs to Speech

• Paper
• arXiv
• Code

In this work, we extended TOWER, an open-source multilingual LLM, to process speech using Discrete Speech Units (DSU). Speech integration was done in two steps, continued pre-training on a mixture of automatic speech recognition (ASR) and machine translation data, starting from the TowerBase model. Following this, the model was instruction-tuned jointly on translation-related data, ASR and Speech Translation. Our results show that DSU integration is feasible in the TowerLLM training setup, moreover, using this training recipe, we recover all the strongholds of TowerLLM while giving the model ASR and ST capabilities.