1
A simple fix for LLM tail latency
对话遇长时间沉默？根因在LLM尾部延迟。一个简单修复方案，直击响应不稳痛点，告别通话卡壳。
Article URL: https://engineering.myhoai.com/posts/a-simple-fix-for-llm-tail-latency/ Comments URL: https://news.ycombinator.com/item?id=49278732 Point…
2
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
针对大模型推理的尾部延迟问题，提出超越预测的尾感知调度方法，兼顾效率与公平。
arXiv:2606.18431v1 Announce Type: new Abstract: LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. R…