1
In-Place Tokenizer Expansion for Pre-trained LLMs
提出一种原地扩展分词器的方案,解决预训练大模型因后期语言加入导致的 token 效率低下问题,降低延迟与能耗。
arXiv:2607.15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-trainin…