AI 日报hiw3c.com

人工智能爬虫指数

原文标题 · AI Crawler Index
Hacker News Top mojodojo.io 网页快照
正文为英文,可一键机器翻译(仅首次需要等待)

AI bots made 0.93 hits for every search engine hit.

Pages an assistant like ChatGPT read on the spot because someone asked it a question.

Of hits claiming to be a bot we can verify, the share that came from the wrong IP.

Crawler hits per day, fakes removed

Who is crawling

Spoofed hits are excluded from every share on this page. 535 were removed in this period.

Every bot, ranked

The standouts of the last 30 days

Fewest hits per page: visits once and moves on

Most hits per page: keeps fetching the same few

A bot needs at least 100 hits in the period to win (10 for rarest visitor). Where an AI bot would win its own category, it is listed underneath.

How the numbers are made

Source. Every page request to mojodojo.io whose user agent looks like a crawler, logged at the web server with the visitor's IP. One site, so read the numbers as a case study rather than a census. Tracking since 6 June 2026. IPs were recorded from 7 Oct 2026; hits before that are classified by user agent only and counted as unverified.

Categories. AI training bots collect pages for model training (GPTBot, ClaudeBot, CCBot). AI search bots build the index behind AI answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). AI user fetches are live reads on behalf of a person (ChatGPT-User, Claude-User, Perplexity-User).

Verification. A hit counts as verified when its IP sits inside the ranges the operator publishes (OpenAI, Anthropic, Perplexity, Google, Bing, Apple, Common Crawl, DuckDuckGo, Ahrefs), or passes forward-confirmed reverse DNS. Ranges refresh daily. A hit that fails the check is marked spoofed and left out. Bots whose operator publishes no method are counted as unverifiable.

Cite this. Mojo Dojo, “AI Crawler Index”, mojodojo.io/ai-crawler-index, accessed 8 Oct 2026. Free to reuse under CC BY 4.0 with a link back.