1
谷歌搜索结果链接新增 goto 跳转参数，旨在反制爬虫与 AI 抓取
IT之家 8 月 27 日消息，谷歌今日对 Seroundtable 回应证实，其正在搜索结果链接中逐步推广一项新的技术措施 —— 为搜索结果 URL 添加“goto”跳转参数。 用户点击搜索结果时，链接将不再直接指向目标网页，而是先经过一个带有“ google.com/goto ”参数的中转地址。…
2
How I Built Smart Scraper M2M: A Fast ~30ms Scraper API for AI Agents
30毫秒极速爬虫API，专为AI代理打造，告别爬取卡顿瓶颈。
Building AI Agents with frameworks like CrewAI or LangChain often hits a bottleneck: heavy, slow web scraping that bloats context windows and increase…
3
JavaScript Links Can Hide Content From AI Crawlers, a 41-Day Test Shows
41天实测曝光：JavaScript生成的链接会让AI爬虫“失明”，GPTBot、Bingbot和Google的发现差异惊人。
JavaScript-injected navigation can create a material discovery gap between search crawlers and AI-focused bots. In a 41-day field experiment, pages li…
4
Show HN: Amz-CLI – A command line for Amazon
一条命令抓取亚马逊公开数据：排行榜、评论、QA、优惠一目了然，适合电商选品与竞品调研。
Article URL: https://github.com/tamnd/amz-cli Comments URL: https://news.ycombinator.com/item?id=49342785 Points: 3 # Comments: 0
5
Show HN: PyScrappy, self-healing web scraping selectors plus an MCP server
网页爬虫遇到改版不再翻车——自愈选择器自动重定位元素，还带MCP服务器让AI直接调用。
Article URL: https://github.com/mldsveda/PyScrappy Comments URL: https://news.ycombinator.com/item?id=49317799 Points: 3 # Comments: 0
6
The AI Crawler Checklist: Why the Model Never Even Read Your Page
揭秘网站为何未被AI模型收录，从Cloudflare误拦到robots.txt陷阱，一份检查清单教你让AI真正读到你的页面。
You can write the perfect page, answer-first, honest, quotable, and still get zero AI citations. Not because the content lost. Because the crawler nev…
7
Show HN: Embassia – LLM Search Console – Web crawlers talk to your site's agent
用AI代理谈判替代800封冷邮件的低效触达，让网站爬虫直接与你的销售agent对话，B2B获客新思路。
You want to know what visitors are actually curious about. But there is no adequate tool for this. Not even the Google Search Console. Right now: -Cra…
8
Classifying AI Crawlers and AI Chat User Fetches from Server Logs
只要分析服务器日志，就能精准区分 AI 爬虫与 AI 聊天用户的抓取行为，告别传统 JS 统计盲区，让流量来源一清二楚。
Article URL: https://canonry.ai/blog/classifying-ai-crawlers-user-fetches-server-logs Comments URL: https://news.ycombinator.com/item?id=49294267 Poin…
9
The web’s newest weapon against AI scrapers is a font
新型字体ShieldFont，让人正常阅读、让AI抓取变乱码，巧妙对抗数据抓取。
“ShieldFont” aims to poison AI training data without making pages unreadable for people.
10
Show HN: Watching browser fingerprinting calls via Chromium's Blink layer
深入 Chromium Blink 层，实时捕捉浏览器指纹调用，精准验证反检测配置，是反爬对抗调试利器。
Article URL: https://kameleo.io/blog/this-is-the-closest-well-ever-get-to-reverse-engineering-anti-bots Comments URL: https://news.ycombinator.com/ite…
11
Show HN: I audited 743 URLs across 8 free browser tools
实测8个AI工具站的743个URL，挖掘出链接、标注和内容短板，工具站运营值得一读
Article URL: https://github.com/Gavin1901/free-browser-tools-index/blob/master/daily/2026-08-08-8site-pore-level-audit-and-repair.md Comments URL: htt…
12
Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt
创建蜜罐链接，吸引爬虫或攻击者访问并实时告警，轻松识破忽略 robots.txt 的恶意外部请求。
At the beginning of the year I decided to set up a scraping and LLM honeypot on one of my personal websites which included a fake git repo with code c…
13
WordPress AI Visibility in 2026
面向AI爬虫的开放元数据规范，让WordPress站点在2026年更具大模型可见性
We've been working on an open specification that allows every webpage to expose machine-readable metadata specifically for LLMs and AI crawlers. https…
14
How to Collect Product Data from a Website with Python
一步步教你用Python爬取网站产品数据，从安装工具到绕过机器人检测。
In this guide, you will learn how to write a Python script that visits a website, grabs product names and prices, and saves everything into a spreadsh…
15
Show HN: I built a scraping engine where the LLM writes configs, not code
无需手写代码，用自然语言描述需求，LLM自动生成爬虫配置，让数据采集更简单高效。
Article URL: https://github.com/PxyUp/fitter Comments URL: https://news.ycombinator.com/item?id=49015561 Points: 1 # Comments: 0
16
Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents
LLM代理正以惊人效率攻破传统网页机器人防御，这篇论文系统揭示了当前安全机制的致命盲区。
arXiv:2607.18659v1 Announce Type: cross Abstract: LLM-based browser agents are rapidly changing the threat landscape for web security. Unlike traditio…
17
Can AI agents use ur site?
开源工具OpenIngress用Playwright自动分析你的网站对AI代理的友好程度，帮助优化可访问性与导航体验。
Article URL: https://github.com/Open-Ingress/OpenIngress Comments URL: https://news.ycombinator.com/item?id=48985431 Points: 5 # Comments: 9
18
I Built a Bot That Hunts GitHub Bounties While I Sleep
用AI机器人自动抢GitHub赏金任务，却因提交太快被识破，分享反检测经验
Nine pull requests in four hours. Not spammed garbage. A missing semicolon in Express.js, dead badges stripped from a README, redundant blank lines re…
19
Patreon stops asking AI bots not to scrape — and starts blocking them
Patreon从劝说到动手，直接封禁AI爬虫，平台反击不再手软。
Patreon is strengthening its defenses against AI scraping by working with Cloudflare to block bots that train AI models on creators’ content without p…
20
Patreon 与 Cloudflare 联手，在平台禁止 AI 训练爬虫
Patreon联手Cloudflare推出新措施，防止AI爬虫抓取创作者付费内容，保护平台收益。
IT之家 7 月 18 日消息，创意内容分享与变现平台 Patreon 昨日宣布与技术合作伙伴 Cloudflare 合作， 在该平台上完全禁止为人工智能模型训练搜集资料的爬虫机器人 ，同时继续允许增加创作者曝光度的搜索引擎爬虫爬取内容。 Patreon 平台上的创作者通常采用付费与免费结合的内容结…