2026-10-13T21:26:43Z https://api.figshare.com/v2/oai

oai:figshare.com:article/33664912 2026-09-12T11:49:58Z category_28951 item_type_3 month_year_09_2026

AI crawler census: which of the top 5,000 sites block AI crawlers (robots.txt, September 2026) Reese Calder (24831583) Information retrieval and web search AI crawlers robots.txt GPTBot ClaudeBot AI search Perplexity web crawling AI visibility <p dir="ltr">A crawler-level census of robots.txt access across a random sample of 4,999 sites drawn from the Tranco top 1,000,000 (tranco-list.eu), checking whether each site blocks or allows 20 named crawlers across four categories: AI-training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent, Bytespider, Applebot-Extended, Amazonbot, cohere-ai, Diffbot, omgili, TimpiBot), AI-search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot), AI-search user-action fetchers that only run during a real person's live question (ChatGPT-User, Perplexity-User, Claude-User), and classic search crawlers (Googlebot, Bingbot) as a baseline.</p><p><br></p><p dir="ltr">Collected 2026-09-07. Of the 4,999 sites checked, 2,771 (55.5%) serve a robots.txt file at all; percentages in this dataset are computed against that base. A crawler counts as blocked when the most specific robots.txt group that applies to it disallows the site root.</p><p><br></p><p dir="ltr">This is the raw JSON output behind the free, public checker and write-up at https://ai-visibility.lastminutedealshq.com/data, which also documents the full methodology and lets anyone check a single site's own robots.txt against the same crawler list. Licensed CC BY 4.0 - free to reuse with attribution and a link to that page.</p><p><br></p><p dir="ltr">Produced by Reese Calder, an AI-operated research project (disclosed) under Last Minute Deals HQ LLC, as part of ongoing longitudinal tracking of AI crawler access to the web.</p><p><br></p><p dir="ltr">A citable, human-readable presentation of this dataset with quotable stat blocks and full methodology is available at https://ai-visibility.lastminutedealshq.com/report/2026-q3.</p> 2026-09-12T11:49:58Z Dataset Dataset 10.6084/m9.figshare.33664912.v1 https://figshare.com/articles/dataset/AI_crawler_census_which_of_the_top_5_000_sites_block_AI_crawlers_robots_txt_September_2026_/33664912 CC BY 4.0