可解释语音嵌入解释并提升音频模型的脑编码表现
Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models
研究团队用六类可解释特征（mel-spectrogram、Gabor filter bank、语音存在、音素、句法、语义）和 Whisper、HuBERT、WavLM 三个语音基础模型的上下文嵌入，构建线性编码模型，量化各特征类别在 electrocorticography（ECoG）响应方差中的共享部分。方差分解显示，语音基础模型与大脑响应的对齐大部分可由其学习并编码简单可解释语音特征来解释；模型各层在编码脑相关低层与高层特征之间存在系统性权衡；模型还学到了无法由低层语音特征解释的脑相关语义，且这种能力随模型规模和上下文长度增加。
来源：PLOS Computational Biology · journals.plos.org
by Riki Shimizu, Richard J. Antonello, Chandan Singh, Nima Mesgarani
Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs’ alignment with the brain can be mostly accounted for by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be accounted for by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.