41天实测曝光：JavaScript生成的链接会让AI爬虫“失明”，GPTBot、Bingbot和Google的发现差异惊人。
JavaScript-injected navigation can create a material discovery gap between search crawlers and AI-focused bots. In a 41-day field experiment, pages li…
揭秘网站为何未被AI模型收录，从Cloudflare误拦到robots.txt陷阱，一份检查清单教你让AI真正读到你的页面。
You can write the perfect page, answer-first, honest, quotable, and still get zero AI citations. Not because the content lost. Because the crawler nev…
只要分析服务器日志，就能精准区分 AI 爬虫与 AI 聊天用户的抓取行为，告别传统 JS 统计盲区，让流量来源一清二楚。
Article URL: https://canonry.ai/blog/classifying-ai-crawlers-user-fetches-server-logs Comments URL: https://news.ycombinator.com/item?id=49294267 Poin…
新型字体ShieldFont，让人正常阅读、让AI抓取变乱码，巧妙对抗数据抓取。
“ShieldFont” aims to poison AI training data without making pages unreadable for people.
面向AI爬虫的开放元数据规范，让WordPress站点在2026年更具大模型可见性
We've been working on an open specification that allows every webpage to expose machine-readable metadata specifically for LLMs and AI crawlers. https…
Patreon从劝说到动手，直接封禁AI爬虫，平台反击不再手软。
Patreon is strengthening its defenses against AI scraping by working with Cloudflare to block bots that train AI models on creators’ content without p…
Patreon联手Cloudflare推出新措施，防止AI爬虫抓取创作者付费内容，保护平台收益。
IT之家 7 月 18 日消息，创意内容分享与变现平台 Patreon 昨日宣布与技术合作伙伴 Cloudflare 合作， 在该平台上完全禁止为人工智能模型训练搜集资料的爬虫机器人 ，同时继续允许增加创作者曝光度的搜索引擎爬虫爬取内容。 Patreon 平台上的创作者通常采用付费与免费结合的内容结…
Linus Torvalds 回应 AI 公司爬取 Linux 内核邮件列表，推动 Anubis 工作量证明保护，揭示开源社区与 AI 数据采集的冲突。
Article URL: https://lore.kernel.org/linux-media/CAHk-%3Dwi4zC%2BZe8e%2Bp3tMv8TtG_80KzsZ1syL9anBtmEh5Z40vg%40mail.gmail.com/ Comments URL: https://new…
网站用Anubis工作量证明机制抵御AI爬虫，保护服务器资源，防止大规模爬取导致崩溃。
Article URL: https://lore.kernel.org/linux-media/CAHk-=wi4zC+Ze8e+p3tMv8TtG_80KzsZ1syL9anBtmEh5Z40vg@mail.gmail.com/#t Comments URL: https://news.ycom…
开源规范让AI爬虫流式解析全站内容，支持NDJSON与markdown twins，大幅提升摄入效率
Article URL: https://github.com/CKL75/semantic-manifest-specification Comments URL: https://news.ycombinator.com/item?id=48774528 Points: 1 # Comments…
Cloudflare新规要求AI公司区分搜索与AI爬虫，否则默认封杀，推动内容付费模式落地
Cloudflare is giving AI companies until September 15 to separate web crawlers used for search from those used for AI training and agents, or risk bein…
AI爬虫引发网站内容边界争议：你是否该为自己的数据设下“新国境”？
Recently I came across a discussion about AI crawlers and robots.txt files. The conversation centered on a simple question: Should website owners allo…
专为 AI 智能体设计的自托管轻量爬虫，Docker 一键部署，简洁高效。
Article URL: https://github.com/JorG18/agentcrawl Comments URL: https://news.ycombinator.com/item?id=48709512 Points: 2 # Comments: 0
9天日志拆穿AI爬虫真相：ChatGPT-User不是爬虫？小博客如何应对？
I run a small Home Assistant / self-hosting blog. On a normal day a few dozen humans show up. So when I finally grepped my nginx logs for AI crawlers,…
一个Claude Code技能帮你快速检测网站是否被AI爬虫抓取，保护内容不被滥用。
Article URL: https://github.com/MerqryLabs/ai-crawler-visibility Comments URL: https://news.ycombinator.com/item?id=48683149 Points: 2 # Comments: 0
像Lighthouse但专为AI Agent设计，几秒审计网站对AI爬虫的友好度，开源且实用。
Article URL: https://github.com/lucioduran/ax-audit Comments URL: https://news.ycombinator.com/item?id=48567130 Points: 2 # Comments: 0
AI爬虫冲击法律网站，此案揭示反爬新技术Anubis与工作量证明的巧妙博弈。
Article URL: https://www.bailii.org/ew/cases/EWHC/Ch/2026/1199.html Comments URL: https://news.ycombinator.com/item?id=48328216 Points: 1 # Comments: …
网站站长反击AI爬虫新思路：主动投喂垃圾数据来保护原创内容
Article URL: https://iczelia.net/blog/webmaster-it-s-time-to-serve-slop-to-ai-crawlers/ Comments URL: https://news.ycombinator.com/item?id=48298655 Po…
LWN 网站运营者直面 AI 爬虫泛滥的挑战，分享对抗策略与经验。
Article URL: https://lwn.net/Articles/1008897/ Comments URL: https://news.ycombinator.com/item?id=48291189 Points: 1 # Comments: 0
微软喊话出版商别屏蔽AI爬虫，否则将自绝于内容生态。
Article URL: https://www.adexchanger.com/publishers/microsoft-to-publishers-dont-block-the-ai-bots/ Comments URL: https://news.ycombinator.com/item?id…