三、AI 训练爬虫（74 条）

大模型厂商用于采集训练语料的爬虫。其中多数官方支持通过 robots.txt 屏蔽（如 GPTBot、ClaudeBot、Google-Extended），但也有厂商（如字节跳动 Bytespider）被社区长期反馈无视该协议。表格中的："官方说明"一列保留英文原文描述，避免翻译引入歧义；"匹配正则"可直接用于做访问日志分析或 WAF 拦截规则。

3.1 AI 训练 / 数据集采集 - 44 条

#匹配正则典型 UA 实例官方说明
1[cC]laude[bB]otclaudebotAnthropic Claude web crawler bot
2anthropic-aianthropic-aiAnthropic AI web crawler bot
3ApifyBotMozilla/5.0 (compatible; ApifyBot/1.0)Apify's web scraping and data extraction
4ApifyWebsiteContentCrawlerApifyWebsiteContentCrawler/1.0 (+https://apify.com/apify/website-content-crawler)Apify's full website content extractor
5BrightbotBrightbot 1.0Bright Data's AI-ready data collector
6BytespiderMozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrom…ByteDance Bytespider web crawler bot
7CCBotCCBot/2.0 (http://commoncrawl.org/faq/)Common Crawl web crawler for indexing
8Channel3BotMozilla/5.0 (compatible; Channel3Bot/1.0; +https://trychannel3.com/channel3bot)Channel3's universal product catalog indexer
9ChatGLM-SpiderMozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://chatglm.cn/)AI model training data collection crawler
10ChatGPT-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.c…ChatGPT user web crawler bot
11Claude-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthro…Anthropic Claude user web crawler bot
12Claude-WebClaude-Web/1.0 (web crawler; +https://www.anthropic.com/; bots@anthropic.com)Anthropic Claude web crawler bot
13cohere-aicohere-aiCohere AI web crawler bot
14cohere-training-data-crawlercohere-training-data-crawler (+crawler@cohere.ai)LLM training data collection for AI models
15crawl4aicrawl4ai-adapter/1.0Open-source LLM-friendly web scraping and extraction library
16DeepSeekBotMozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/bot)DeepSeek AI model training and data collection crawler
17ExteContextCrawlMozilla/5.0 (compatible; ExteContextCrawl/1.0; +http://crawl001.exte.ai)AI context extraction and content understanding web crawler
18FacebookBotMozilla/5.0 (compatible; FacebookBot/1.0; +https://developers.facebook.com/docs/sharing/webmasters/…Meta's AI speech recognition training crawler
19FirecrawlAgentMozilla/5.0 (compatible; FirecrawlAgent; +https://firecrawl.dev/)Firecrawl's LLM data extraction crawler
20FlyriverbotMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.1…AI content source verification and attribution checking crawler
21Google-ExtendedMozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)Google extended web crawler bot
22GPTBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptb…OpenAI GPT web crawler bot
23HenkBotMozilla/5.0 (compatible; HenkBot/1.0; +https://valyu.ai/crawler)Valyu AI content analysis and data collection crawler
24ImageMindImageMindImage analysis crawler collecting visual data for AI training purposes
25imageSpiderMozilla/5.0 (compatible; imageSpider; +https://www.bytedance.com/)ByteDance's image collection crawler
26img2datasetMozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 (compatible; img2datas…Open source tool downloading images for machine learning dataset creation
27Kangaroo BotMozilla/5.0 (compatible; Kangaroo Bot/1.0; +http://www.kangaroo.com)Australian AI model training crawler
28KunatoCrawlerMozilla/5.0 (compatible; KunatoCrawler/1.0; +http://kunato.ai/bot.html)AI training data collection and web scraping crawler
29laion-huggingface-processorMozilla/5.0 (compatible; laion-huggingface-processor; +https://laion.ai/)LAION's image dataset builder for AI
30MistralAI-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mi…Mistral AI web crawler bot for content indexing
31newsai\/Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0…NewsAI crawler for news content aggregation and indexing
32NovellumNovellumAI data collection crawler for model training and enrichment
33Perplexity-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perple…Perplexity user web crawler bot
34PerplexityUserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityUser/1.0; +https://perplex…Perplexity AI web crawler bot
35Poggio-CitationsPoggio-Citations 1.0 (+https://docs.poggio.io/api/robots)Poggio's AI sales enablement data collector
36SBIntuitionsBot\/Mozilla/5.0 (compatible; SBIntuitionsBot/0.1; +https://www.sbintuitions.co.jp/bot/)SB Intuitions web crawler bot
37semantic-visionsMozilla/5.0 (Linux; CentOS; compatible; semantic-visions-discovery; HTTPClient 4.5)Semantic content analysis and AI training data crawler
38Spawning-AISpawning-AIWeb crawler indexing content for AI models
39Spider[\s\S]*spider\.comMozilla/5.0 (compatible; Spider; +https://www.spider.com/)AI project web data crawler
40TerraCottaTerraCotta https://github.com/CeramicTeam/CeramicTerracottaCeramic Network decentralized data indexing and crawling bot
41The Knowledge AIThe Knowledge AIAI data collection crawler indexing content and enhancing machine learning models
42ThinkbotMozilla/5.0 (compatible; Thinkbot/0.5.8; +In_the_test_phase,_if_the_Thinkbot_brings_you_trouble,_pl…Experimental AI thinking and reasoning web crawler bot
43TikTokSpiderMozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compa…TikTok web crawler for content discovery
44TSM-turingosMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7; TSM-turingos-1253296984) AppleWebKit/537.36 (KHTML,…TuringOS AI-powered content analysis and monitoring crawler

3.2 AI 搜索 / 助手类(可能回流训练数据) - 30 条

#匹配正则典型 UA 实例官方说明
1AI2BotAI2BotAcademic research crawler for NLP dataset creation
2AISearchBotAI-powered search engine web crawler bot
3Amzn-SearchBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/119.0.604…Amazon's AI search indexer for Alexa
4Amzn-UserMozilla/5.0 (compatible; Amzn-User; +https://developer.amazon.com/)Amazon AI assistant for Alexa queries
5AnomuraAnomura/1.2 (+https://www.direqt.ai)Direqt's AI search indexer
6Aranet-SearchBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Aranet-SearchBot/1.0; +https://arane…AI-powered search engine and content indexing crawler bot
7atlassian-botMozilla/5.0 (compatible; atlassian-bot; +https://www.atlassian.com/)Atlassian Rovo's AI search crawler
8AzureAI-SearchBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;Microsoft's Azure AI search indexer
9bigsur\.aibigsur.aiBig Sur AI's web agent crawler
10Claude-SearchBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.a…Anthropic Claude search web crawler bot
11Cloudflare-AutoRAGCloudflare-AutoRAG (https://developers.cloudflare.com/autorag; autorag@cloudflare.com)Cloudflare's RAG service indexer
12DuckAssistBotDuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)DuckDuckGo assistant bot for web crawling and search
13Gemini-Deep-ResearchMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemin…Google Gemini deep research web crawler
14Google-CloudVertexBotMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Ch…Google Vertex AI crawler indexing content for enterprise search solutions
15Google-NotebookLMMozilla/5.0 (compatible; Google-NotebookLM; +https://notebooklm.google.com/)Google's AI research and note-taking assistant
16iAskBot\/Mozilla/5.0 AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; iAskBot/1.0; +https://iask.ai/) Ch…AI search engine crawler indexing web content for question answering
17iaskspideriaskspider/2.0Chinese AI search engine and question answering crawler
18kagi-fetcherkagi-fetcher/1.0Kagi AI assistant for web content
19KendraBotKendraBotAI-powered enterprise search crawler for documents
20LinerBot\/Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinerBot/1.0; +https://docs.getliner…AI workspace crawler indexing web content for search and discovery
21linkReader\/linkReader/1.0 (+https://gochitchat.ai)AI assistant crawler for content indexing
22LinkupBotLinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; bot@linkup.so)Linkup's enterprise AI search crawler
23meta-externalagent\/meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)Meta external agent web crawler bot
24OAI-SearchBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.…OpenAI search web crawler bot
25PerplexityBot\/Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexi…Perplexity AI web crawler for search
26PhindBotMozilla/5.0 (compatible; PhindBot; +https://www.phind.com/)AI-powered developer answer engine crawler
27ShapBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0Parallel's AI agent web context provider
28TaraGroup Intelligent BotTaraGroup Intelligent Bot V1TaraGroup AI-powered web intelligence and analysis crawler
29TavilyBotMozilla/5.0 (compatible; TavilyBot; +https://tavily.com/)Tavily's real-time AI agent data crawler
30ZanistaBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ZanistaBot/1.0; +https://zanista.ai/…Zanista's AI search crawler

Copyright © 2026 24KRMB.COM.All rights reserved.