拆解Adam优化器背后的脆弱根基，用原理推导新设计，深度学习调参必备的理论参考。
arXiv:2608.16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for …
MoE训练新方案：Sinkhorn梯度下降替代AdamW，减少优化器状态内存，让大模型训练更省显存。
arXiv:2608.04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer sta…
用大模型自动设计优化器特征函数，突破人工设计瓶颈，学习优化领域的新范式。
arXiv:2607.27389v1 Announce Type: new Abstract: Learning-to-optimize (L2O) methods accelerate repeated optimization by training models to predict solu…
探讨SOAP、Muon等新型优化器如何突破LLM预训练规模瓶颈，13位作者揭示前沿缩放规律
arXiv:2607.20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computa…
首次实现全栈FP4量化预训练，突破LLM训练速度和显存瓶颈
arXiv:2607.04422v1 Announce Type: cross Abstract: Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, …
优化器竟成AI行为“开关”：可放大或压制涌现性失配，安全研究必读。
arXiv:2606.31591v1 Announce Type: cross Abstract: Emergent misalignment (EM) is a recently discovered phenomenon in LLMs where fine-tuning on a narrow…
AutoML管道如何挑选损失函数与优化器最佳搭档？这项研究为NNGPT系统找到了稳定训练的配对规律
arXiv:2606.20933v1 Announce Type: new Abstract: The choice of loss function and optimizer is an important decision, that shapes further model training…
AdamW在重尾噪声下是否有效？这篇论文提出了一个亟待解决的开放问题。
arXiv:2606.23676v1 Announce Type: cross Abstract: AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it …
新型FOGO优化器通过遗忘感知正交化机制，有效缓解持续学习中的灾难性遗忘问题
arXiv:2606.10406v1 Announce Type: cross Abstract: We argue that forgetting is not confined to continual learning but is a general optimization phenome…
Muon优化器比Adam学习到更鲁棒和可迁移的特征，为深度学习训练提供新选择。
arXiv:2606.09658v1 Announce Type: new Abstract: Muon has recently emerged as a state-of-the-art optimizer for pretraining Large Language Models (LLMs)…
系统性揭示LLM在生物与经济安全基准上的类失控优化器失败模式，视角新颖，观察格式简化。
arXiv:2509.02655v3 Announce Type: replace-cross Abstract: Many AI alignment discussions of "runaway optimisation" focus on RL agents: unbounded utilit…
Muon优化器正交化程度对训练效果的影响研究，给出放松的立方Newton-Schulz调度方案
arXiv:2606.00371v1 Announce Type: new Abstract: Muon optimizers improve neural-network training by replacing ill-conditioned momentum updates with app…
提出可学习的零阶优化器，无需梯度即可高效微调大模型，大幅降低内存开销。
arXiv:2510.00419v2 Announce Type: replace Abstract: Zeroth-order optimizers have recently emerged as an attractive approach for fine-tuning large lang…
MuCon通过裁剪奇异值改进Muon优化器，为LLM训练提供更稳定的更新策略
arXiv:2605.26459v1 Announce Type: new Abstract: Muon-style optimizers take a matrix-valued momentum or preconditioned update $B = U \operatorname{diag…
Muon优化器新升级MONA，融合Nesterov加速突破局部极小，提升大模型训练效率
arXiv:2605.26842v1 Announce Type: new Abstract: The Muon optimizer has recently offered a promising alternative to AdamW for large language model trai…
Muon优化器新突破，基于重尾谱校正解决噪声方向过量问题，助力大模型高效训练
arXiv:2603.10067v2 Announce Type: replace-cross Abstract: Muon has recently shown promising results in LLM training. In this work, we study how to fur…
提出极简优化器设计，大幅降低大模型预训练内存占用，已被ICML 2026接收。
arXiv:2506.16659v3 Announce Type: replace-cross Abstract: Training large language models (LLMs) relies on adaptive optimizers such as Adam, which intr…
Muon优化器新变体AMUSE，实现任意时刻稳定梯度评估，41页论文含25图详解算法改进。
arXiv:2605.22432v1 Announce Type: new Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent work…
深入解读从SGD到Muon的优化器演进，以Schatten-p范数统一矩阵几何约束，为AI研究者提供理论新视角
arXiv:2605.19781v1 Announce Type: new Abstract: Modern optimizers, like Muon, impose matrix-wise geometry constraints on their updates. These matrix-w…
揭秘SGD在LLM预训练中不如Adam的根源：大有效学习率的关键作用。
arXiv:2605.17787v1 Announce Type: new Abstract: It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptiv…