1
EP223: Ollama vs vLLM vs SGLang
对比三大 LLM 推理框架 Ollama、vLLM 与 SGLang，理清性能与适用场景，帮你选对部署方案。
To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently.
2
Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with Llama.cpp
苹果芯片上跑macOS虚拟机，GPU直通让llama.cpp大模型推理性能暴涨11-16倍，实测细节满满。
Article URL: https://github.com/trycua/cua/blob/main/blog/gpu-passthrough-macos-vms.md Comments URL: https://news.ycombinator.com/item?id=49259339 Poi…
3
Should You Self-Host LLM Inference? Cost and Risk Guide
自托管大模型推理到底值不值？一文拆解成本与风险关键点。
Article URL: https://theaiengineer.substack.com/p/should-you-self-host-inference Comments URL: https://news.ycombinator.com/item?id=49258074 Points: 1…
4
Statistically-Lossless Quantization of Large Language Models
突破量化 trade-off：统计无损新方法，兼顾模型保真与推理加速，大模型部署必读。
arXiv:2605.02404v2 Announce Type: replace Abstract: Model quantization has become essential for efficient large language model deployment, yet existin…
5
Kimi K3 on AWS Bedrock?
Kimi K3能否像Claude一样在AWS Bedrock按需使用？社区提问引关注，但暂无官方回应，追踪模型上云动态可留意此帖。
Will it become available? Would love to get some insights. Not Sagemaker or EC2 custom deployments, but on-demand like Claude etc. Comments URL: https…
6
DeepSeek V4 Flash on a Single AMD MI300X
单卡AMD MI300X跑起DeepSeek V4 Flash，低成本部署大模型的最新实践。
Article URL: https://github.com/ryanzhou/deepseek-v4-flash-mi300x Comments URL: https://news.ycombinator.com/item?id=49166386 Points: 272 # Comments: …
7
AirLLM 70B inference with single 4GB GPU
单张4GB显卡就能跑70B大模型推理，AirLLM把内存优化玩到了极致，开源即用
Article URL: https://github.com/lyogavin/airllm Comments URL: https://news.ycombinator.com/item?id=49154228 Points: 204 # Comments: 76
8
ESP32-AI 亮相：8MB PSRAM 运行 2890 万参数模型，设想离线调配出最佳咖啡
8MB内存跑2890万参数AI模型，ESP32离线调配咖啡的硬核尝试来了！
IT之家 7 月 28 日消息，科技媒体 Tom's Hardware 昨日（7 月 27 日）发布博文，报道称乌克兰开发者 Slava S 发布 ESP32-AI 项目， 针对乐鑫（Espressif）的 ESP32-S3 开发板，成功部署 2890 万个参数的本地 AI 模型。 IT之家注：ES…
9
OpenAI’s Hugging Face breach has reignited the debate over alignment and control
OpenAI Hugging Face安全事故暴露AI对齐难题，长期任务评估盲区或引发严重后果
OpenAI's Hugging Face breach has reignited debate over AI alignment and control, exposing competing views on whether increasingly capable AI should be…
10
FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs
移动GPU上微调大模型耗内存？FBLayout通过优化内存布局实现高效微调，Mobisys 2026论文。
arXiv:2607.21624v1 Announce Type: new Abstract: Transformer-based models have enabled unprecedented capabilities across language, vision, and multimod…
11
Build a Local LLM Chatbot with Ollama and Python
手把手教你用Ollama和Python搭建本地大模型聊天机器人，零门槛运行AI
Build a Local LLM Chatbot with Ollama and Python Build a Local LLM Chatbot with Ollama and Python Imagine typing a question into your chatbot and gett…
12
Bootcamp Grad Explores Open-Source AI APIs: What I Learned
从训练营毕业生的视角，揭秘开源AI API的成本与体验，告诉你为何自建模型可能比API更贵。
Here's the thing: bootcamp Grad Explores Open-Source AI APIs: What I Learned I graduated from a coding bootcamp about six months ago, and honestly, I …
13
Linux of AI open-source tools for reducing AI vendor lock-in
汇集开源AI工具，从模型到部署全方位避免依赖单一厂商，值得开发者收藏。
Article URL: https://github.com/sekacorn/Linux-of-Ai Comments URL: https://news.ycombinator.com/item?id=48881655 Points: 1 # Comments: 0
14
Show HN: Using Wake-on-LAN for an AI Project
用Wake-on-LAN按需唤醒RTX 5080，自建本地大模型省钱省电，实战经验全记录
I finally have an excuse to use Wake-on-LAN and save on my AI and energy bill. Comments URL: https://news.ycombinator.com/item?id=48789027 Points: 1 #…
15
The Secret to Blazing Fast On-Device AI: Mastering TFLite Delegates, NPUs, and the Future of Android AI
端侧AI提速关键全在TFLite Delegate与NPU调度，想告别Android模型卡顿和发热，这篇底层拆解值得细读。
If you have ever tried to run a heavy deep learning model on an Android device, you have likely encountered the "AI Lag." The device heats up, the fra…
16
Ask HN: Best Local LLM Setup for a 128GB M4 Max Mac Studio?
128GB M4 Max Mac Studio上本地跑大模型的最优配置讨论，硬件玩家必看。
Comments URL: https://news.ycombinator.com/item?id=48769635 Points: 1 # Comments: 0
17
Ahmad Osman on why local AI is catching up
本地AI无需许可即可研究和部署，正快速追赶云端模型，带来自主智能的新时代，你准备好迎接了吗？
Ahmad Osman has been advocating for local AI — running models on your own computer, workstation or dedicated hardware — long before it became a major …
18
Tiny LLM Benchmark: Jetson Orin Nano Super 8GB
边缘设备跑小模型能有多快？实测 Jetson Orin Nano Super 8GB 的完整性能数据，帮你在预算内选对部署方案。
Article URL: https://www.smolhub.com/posts/jetson-nano-super-benchmark-non-reasoning/ Comments URL: https://news.ycombinator.com/item?id=48705815 Poin…
19
How I Replaced Gemini with a Self-Hosted LLM for Two Production Apps
从Gemini迁移到自托管LLM的真实案例，揭示API服务被替代的三大隐形推力，为生产环境部署提供参考。
A while back I wrote about my terminal-inspired portfolio and the products it indexes. Two of those products lean on a language model: the portfolio t…
20
LLM APIs with built-in chatbot in 1 line of code
一行命令部署LLM API，Flama 2.0自带聊天界面，轻松搞定Gemma等模型服务。
Article URL: https://flama.dev/blog/serving_llms_with_flama_cli/ Comments URL: https://news.ycombinator.com/item?id=48671740 Points: 17 # Comments: 0