1
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
利用注意力汇合现象，在注意力层内实现原生MoE训练，有效解决头部坍塌问题。
arXiv:2602.01203v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) often assign disproportionate attention to the first token, a p…