LLM预训练中领域数据重复的缩放规律，揭示数据重复如何影响模型性能与训练效率的边界效应。
arXiv:2608.14071v1 Announce Type: new Abstract: As large language models scale, their training-token budgets must also increase to maintain an appropr…
探讨SOAP、Muon等新型优化器如何突破LLM预训练规模瓶颈，13位作者揭示前沿缩放规律
arXiv:2607.20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computa…
突破带宽限制，DiLoCo类方法如何平衡通信与性能，提升LLM预训练效率
arXiv:2508.15706v3 Announce Type: replace Abstract: Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable …
低秩预训练大模型遭遇不稳定困境？这项ICML 2026研究提出原生低秩LLM预训练的稳定化方法，兼顾效率与质量。
arXiv:2602.12429v2 Announce Type: replace Abstract: Foundation models have achieved remarkable success, yet their growing parameter counts pose signif…
提出GIFT方法，利用梯度几何信息实现低精度通信，在不牺牲模型精度的前提下显著降低LLM预训练通信开销。
arXiv:2607.07494v1 Announce Type: cross Abstract: Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Co…
首次实现全栈FP4量化预训练，突破LLM训练速度和显存瓶颈
arXiv:2607.04422v1 Announce Type: cross Abstract: Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, …
打破异步训练瓶颈，一步延迟不再是障碍，大模型预训练提速新思路。
arXiv:2606.30634v1 Announce Type: new Abstract: Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous …
多目标强化学习调度数据，让大模型预训练更高效。
arXiv:2606.24133v1 Announce Type: new Abstract: The composition of training data, governed by the diversity of sources and their mixing strategy, is a…
史上最大伦理数据集合Common Corpus发布，为LLM预训练提供合规开源新选择
Article URL: https://openreview.net/pdf?id=0wSlFpMsGb Comments URL: https://news.ycombinator.com/item?id=48567122 Points: 2 # Comments: 0
用强化学习动态优化预训练数据配比，Actor-Critic框架让大模型学得更高效。
arXiv:2505.23878v2 Announce Type: replace-cross Abstract: Optimizing pretraining data composition is pivotal for LLM generalization. While dynamic mix…
当标准探测准确率饱和时，引入“脆弱性”度量作为互补指标，为LLM预训练分析提供新视角。
arXiv:2606.11375v1 Announce Type: cross Abstract: Standard linear probing declares a property "encoded" when a classifier on hidden states achieves hi…
被ICML接收的自适应梯度裁剪方法，有效提升LLM预训练稳定性，AI训练优化的新突破
arXiv:2502.11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining. While previous…
这篇论文提出统一本地通信与更新策略，旨在提升大模型预训练的通信效率，分布式训练的新视角。
arXiv:2606.11081v1 Announce Type: cross Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute …
多项式预条件层PC Layer通过重塑权重矩阵奇异值谱，稳定大模型预训练过程，提升训练质量与收敛效率。
arXiv:2606.06470v1 Announce Type: cross Abstract: We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner tha…
真实生产环境下的LLM预训练运维经验，504块GPU集群从故障检测到恢复的实证分析。
arXiv:2605.09370v2 Announce Type: replace-cross Abstract: Large-scale AI training is now fundamentally a distributed systems problem, and hardware fai…
提出NITP隐式Token预测新方法，革新LLM预训练范式，已被ICML 2026接收。
arXiv:2605.24956v1 Announce Type: new Abstract: Standard next-token prediction (NTP) supervises language models solely through discrete labels in the …
新论文提出ReCoVer系统，用容错集合和灵活工作负载增强LLM预训练弹性，减少训练中断损失。
arXiv:2605.11215v2 Announce Type: replace-cross Abstract: Pre-training large language models on massive GPU clusters has made hardware faults routine …
颠覆认知？弱教师模型也能有效蒸馏LLM，预训练阶段教师强度并非关键。
arXiv:2605.23857v1 Announce Type: new Abstract: Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield b…
提出极简优化器设计，大幅降低大模型预训练内存占用，已被ICML 2026接收。
arXiv:2506.16659v3 Announce Type: replace-cross Abstract: Training large language models (LLMs) relies on adaptive optimizers such as Adam, which intr…
揭秘SGD在LLM预训练中不如Adam的根源：大有效学习率的关键作用。
arXiv:2605.17787v1 Announce Type: new Abstract: It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptiv…