1
SLM 生成式高并发推理引擎：延迟约束下的有效吞吐优化
本文为SLM（Small Language Model， 小参数量语言模型0.5~1B）在搜推场景的大规模落地提供了一套高性能、可复用的工程实践。
2
H3-metal – Native MiniMax-H3 inference for Apple Silicon
Redis作者antirez用纯C为Apple Silicon打造原生MiniMax-H3推理，单文件实现性能惊艳，本地AI玩家不容错过。
Article URL: https://github.com/antirez/h3.c Comments URL: https://news.ycombinator.com/item?id=49252179 Points: 287 # Comments: 60
3
VKAE: VIDRAFT's Inference Engine Hits 23 GPU Speedup and ~10K Tokens/sec on a Single Nvidia B200
推理引擎VKAE实现23倍GPU加速，单卡B200跑出万级token吞吐，AI部署效率再突破。
VKAE: VIDRAFT's Inference Engine Hits 23× GPU Speedup and ~10K Tokens/sec on a Single Nvidia B200 TL;DR: VIDRAFT, a Korean Pre-AGI AI startup, has dev…
4
Millfolio – my take on local/hybrid AI
从零手写推理引擎，用Metal内核在Mac mini上跑起本地AI，硬核技术细节全公开。
Article URL: https://millfolio.app/blog/send-the-program-to-your-data/ Comments URL: https://news.ycombinator.com/item?id=48812775 Points: 1 # Comment…
5
Show HN: Morph Reflexes – Multi-head classifiers for agent traces
用多分类器从代理轨迹中提取语义信号，比前沿模型评判更快更便宜，直击生产代理行为故障痛点。
The most common failures for production agents are behavioral: looping, reasoning leakage, user frustration, and more. Using a frontier model like GPT…
6
Quantifying LLM Cost Savings from Cache-Aware Inference Routing
一文看懂如何通过缓存感知推理路由为LLM调用省下大笔开销，实测8万次请求数据说话。
Article URL: https://www.auriko.ai/reports/llm-cost-arbitrage Comments URL: https://news.ycombinator.com/item?id=48588557 Points: 4 # Comments: 1
7
Run Agent Skills with mistral.rs v0.8.10: /v1/skills support and more
mistral.rs新版本支持本地模型运行OpenAI兼容的Agent Skills，打破封闭限制，实现私有智能。
Hey all! I'm the maintainer of mistral.rs. I just landed support for OpenAI-compatible Agent Skills via a /v1/skills endpoint, and it works with local…
8
A Guide to AI Inference Engineering
从基础设施到模型优化，全面解析AI推理工程的关键技术与最佳实践。
In this article, we will walk through how inference works and why the field’s optimization techniques exist.
9
Show HN: FlashQwen – A from-scratch CUDA inference engine for Qwen3
FlashQwen 从零构建的 CUDA 推理引擎，专为 Qwen3-8B 优化，开源可部署。
Article URL: https://github.com/frankkk96 Comments URL: https://news.ycombinator.com/item?id=48551028 Points: 2 # Comments: 0
10
Show HN: We built an LLM inference engine in pure Python – no PyTorch, no Triton
纯Python实现的LLM推理引擎，无需PyTorch或Triton，极简依赖且开源可玩！
Article URL: https://github.com/Zyora-Dev/zse/releases/tag/v2.0.0 Comments URL: https://news.ycombinator.com/item?id=48370827 Points: 2 # Comments: 0
11
LLM, give me a JSON. Make no mistakes
深入解析LLM如何高效、可靠地输出无差错JSON，揭秘推理引擎的底层技术。
Article URL: https://nobodywho.ooo/posts/llm-give-me-a-json/ Comments URL: https://news.ycombinator.com/item?id=48369379 Points: 8 # Comments: 0
12
Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
用C++和CUDA打造的高性能轻量LLM推理引擎，代码开源，适合自部署优化。
Article URL: https://github.com/jmaczan/tiny-vllm Comments URL: https://news.ycombinator.com/item?id=48328184 Points: 6 # Comments: 0
13
LLM-Manager: Orchestrating Ollama and Llama.cpp with Pure Bash
纯 Bash 打造的 LLM 管理工具，轻量级替代 Python、Docker 等复杂方案，同时支持 JSON 和交互界面，完美驾驭 Ollama 与 Llama.cpp。
LLM-Manager is a lightweight, modular Bash suite with a dual JSON/Interactive interface designed to manage local and remote inference engines across L…
14
Show HN: GPT-2 inference in pure C#, 0 bytes allocated per token
纯C#实现GPT-2推理引擎，零内存分配无GC压力，性能媲美ONNX Runtime，对.NET开发者极具吸引力。
Article URL: https://github.com/DevOnBike/Overfit Comments URL: https://news.ycombinator.com/item?id=48172293 Points: 1 # Comments: 0
15
Introducing Anthropic’s Claude Opus 4.7 model in Amazon Bedrock
Anthropic最强Opus模型登陆Amazon Bedrock，新推理引擎提升可用性与隐私保护
AWS launches Claude Opus 4.7 in Amazon Bedrock, Anthropic's most intelligent Opus model for advancing performance across coding, long-running agents, …