四、SEO 工具型爬虫（142 条）

为 SEO 工具商采集外链、关键词、排名、站点审计数据的爬虫。对目标站点几乎无流量价值，但抓取量大，是中小站点带宽的"隐形杀手"。表格中的："官方说明"一列保留英文原文描述，避免翻译引入歧义；"匹配正则"可直接用于做访问日志分析或 WAF 拦截规则。

#匹配正则典型 UA 实例官方说明
1(sistrix\|SISTRIX) [cC]rawlerMozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)SISTRIX SEO tool web crawler for analysis
2008\/008/0.83 (http://www.advancedwebranking.com)SEO monitoring bot tracks keyword rankings across regions
3AdagioBotAdagioBotSEO analysis bot for performance and optimization
4AgencyAnalyticsBotAgencyAnalyticsBotSEO audit bot for performance and backlink analysis
5Ahrefs(Bot\|SiteAudit)Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)Ahrefs SEO tool web crawler for link analysis
6alexa site auditMozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )Web crawler for SEO data aggregation and analysis
7AlphaBotMozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html)AlphaBot SEO web crawler for analysis
8AppSiteAssociationAppSiteAssociationVerifies app connection to website domains
9AttractaMozilla/5.0 (compatible; Attracta)Attracta SEO optimization crawler
10Audisto CrawlerAudisto Crawler (mobile; +https://audisto.com/bot)Audisto SEO web crawler for analysis
11AudistoBotAudistoBotTechnical SEO and website audit bot
12backlinkcrawlerBacklink Crawler web crawler for analysis
13BigUpDataBotBigUpDataBotAnalytics crawler for SEO and metric tracking
14bitlybotbitlybot/3.0 (+http://bit.ly/)Bit.ly web crawler for link tracking
15BLEXBot\/Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)WebMeUp BLEX web crawler bot
16blogmuraBotblogmuraBot (+http://www.blogmura.com)A blog ranking site which links to blogs on just about every theme possible.
17botifyMozilla/5.0 (compatible; botify; http://botify.com)Botify SEO tool web crawler for analysis
18BrightEdge CrawlerBrightEdge Crawler/1.0 (crawler@brightedge.com)BrightEdge web crawler for SEO
19brokenlinkcheck\.comMozilla/5.0 (Windows NT 5.1) brokenlinkcheck.com/1.2Broken link detection and website integrity checker
20buzzbotBuzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)Buzzstream web crawler for link research
21ClarityBot\/Mozilla/5.0 (compatible; ClarityBot/9.0; +https://www.seoclarity.net/bot.html)SEO analytics crawler providing optimization insights
22ClearscopebotUA Clearscopebot or similarContent optimization and keyword performance analysis
23CloudtrellisMozilla/5.0 (compatible; Cloudtrellis; +https://www.cloudtrellis.com/)Cloudtrellis website audit crawler
24cognitiveSEO BotcognitiveSEO BotSEO analytics and digital marketing crawler
25cognitiveseo\.comMozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCr…CognitiveSEO backlink analysis and SEO research crawler bot
26contentkingappMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.1…Real-time SEO auditing and monitoring tool
27CrawlsonMozilla/5.0 (compatible; Crawlson/1.0; +https://www.crawlson.com/domain)Web crawler for auditing and monitoring
28CXK_BotCXK/5.0 (linux; 43A1AAD0; ARMv9) assistenC5/v6.5.6 (KHTML, lcxk) CXK_Bot/2.3.1Web content indexing and SEO analysis crawler
29DataForSeoBotMozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)DataForSEO web crawler for analysis
30Dead Link CheckerMozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)Broken link detection and website maintenance bot
31deadlinkcheckerwww.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) …Dead Link Checker web crawler for link validation
32DeepCrawlMozilla/5.0 (compatible; MentalHealthLeadBot/DeepCrawl/1.0; +https://aiinstall.co)Lumar enterprise SEO platform crawler
33DeskyobotDeskyobot/1.0 (+https://www.deskyo.com/bot)Website content indexing and SEO analysis
34Dlc\/Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.…Link checking crawler auditing broken URLs across websites
35DmbotMozilla/5.0 (compatible; Dmbot/1.1)Web crawler for data aggregation and SEO
36Domain Re-Animator BotDomain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.comDomain Re-Animator web crawler for domain analysis
37domaincrawlerCipaCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/www.example.com)Domain Crawler web crawler for analysis
38Domains Project\/Mozilla/5.0 (compatible; Domains Project/1.0.3; +https://github.com/tb0hdan/domains)Domains Project web crawler bot
39DomainStatsBotDomainStatsBot/1.0 (http://domainstats.io/our-bot)DomainStats web crawler for domain analysis
40DomCopBotDomCopBot (https://www.domcop.com/bot)Domain research crawler analyzing website metrics and SEO data
41dotbotMozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)Moz DotBot web crawler for SEO analysis
42ExtLinksBotMozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)ExtLinks web crawler for link analysis
43EZIDEZID (EZID link checker; https://ezid.cdlib.org/)EZID link checker web crawler bot
44FDL Stats BotFDL Stats BotFirebase dynamic link analytics and validation
45FedicabotFedicabot / FedicaAppSocial media analytics and link preview bot
46fidget-spinner-botfidget-spinner-botContent indexing and SEO analysis crawler
47findlinkfindlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)Findlinks web crawler for link discovery
48FuseonBot\/FuseonBot/1.1 (+http://linkaffinity.io)Link affinity crawler indexing web content for SEO analysis
49GoogleAssociationService$GoogleAssociationServiceGoogle service verifying digital asset link associations between platforms
50GoogleAssociationService\/GoogleAssociationService/Google service verifying digital asset links between apps and websites
51GroupHigh\/Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/GroupHigh web crawler for influencer marketing
52harsilbot\/Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.…Web crawler indexing and collecting data for SEO analysis purposes
53HubSpotMozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.…HubSpot web crawler for marketing automation
54Huckabot\/Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Ch…E-commerce SEO crawler analyzing product pages for optimization insights
55it2media-domain-crawlerit2media-domain-crawler/1.0 on crawler-prod.it2media.deIT2Media domain web crawler bot
56kinsta-botkinsta-botHosting platform crawler for cache warming and SEO audit checks
57Klaviyo\/Klaviyo/1.0Marketing automation bot fetching product data for email campaigns
58linkapediabotlinkapediabot (+http://www.linkapedia.com)Linkapedia web crawler for link discovery
59LinkCheckerBotMozilla/5.0 (compatible; LinkCheckerBot/2.0; +https://linkchecker.pro/robot/)LinkChecker broken link bot
60linkdexMozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)Linkdex SEO tool web crawler for link analysis
61LinkisBotMozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X)…Linkis web crawler for link sharing
62LinksIndexerBot\/Mozilla/5.0 (compatible; LinksIndexerBot/1.0; +http://linksindexer.com/bot)SEO crawler indexing backlinks and submitting URLs for search engines
63LinkTigerMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/5…Broken link detection crawler auditing website link integrity
64lipperheyMozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)Lipperhey SEO tool web crawler for analysis
65MainWP\/Mozilla/5.0 (compatible; MainWP/4.2.7.1; +http://mainwp.com)WordPress management platform crawler auditing child site health
66MarketGoo\/Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.1.17 (KHTML, like Gecko) Version/7.1…SEO audit spider evaluating technical search engine optimization signals
67MarketingMinerMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Ch…Digital marketing analytics crawler gathering SEO and performance data
68MBCrawlerMBCrawler/1.0 (https://monitorbacklinks.com)Monitor Backlinks web crawler bot
69MegaIndexMozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)MegaIndex SEO tool web crawler for analysis
70MJ12botMJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)Majestic-12 search engine web crawler bot
71nbertaupete95Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.comWeb crawler for SEO and competitive analysis purposes
72netEstate NE CrawlernetEstate NE Crawler (+http://www.sengine.info/)NetEstate web crawler for domain analysis
73NetpeakCheckerBot\/Mozilla/5.0 (compatible; NetpeakCheckerBot/3.6; +https://netpeaksoftware.com/checker)SEO analysis crawler identifying technical issues and metrics
74NitroBotMozilla/5.0 (compatible; NitroBot; +https://www.nitropack.io/)NitroPack SEO optimization bot
75nomore404\.com robotMozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)Broken link detection crawler identifying HTTP 404 errors
76Omnisend\/Omnisend/1.0Ecommerce marketing automation crawler fetching campaign URLs metadata
77OnCrawl\/Mozilla/5.0 (compatible; OnCrawl/1.0; +http://www.oncrawl.com)OnCrawl SEO web crawler for analysis
78Online Domain ToolsMozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online…SEO analysis crawler identifying technical issues and metrics
79Optimizer\)Mozilla/5.0 (compatible; Optimizer)SEO analysis and website optimization crawler
80Orbbot\/Mozilla/5.0 (compatible; Orbbot/1.1;)Web crawler indexing content for SEO and analysis
81OutsellURLValidatorOutsellURLValidatorDigital marketing crawler validating campaign link health
82PandalyticsPandalytics/1.0 (https://domainsbot.com/pandalytics/)Pandalytics web crawler for domain analysis
83Plesk screenshot botPlesk screenshot bot https://support.plesk.com/hc/en-us/articles/10301006946066Hosting platform management crawler for website auditing
84postrankPostRank/2.0 (postrank.com)PostRank web crawler for content ranking
85PR-CY\.RUMozilla/5.0 (compatible; PR-CY.RU; + https://a.pr-cy.ru)PR-CY web crawler for SEO analysis
86RankActiveLinkBotMozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)RankActive web crawler for link analysis
87RavenCrawler\/Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)SEO auditing crawler analyzing website optimization signals
88RevvimGort\/Chrome/54.0.2840.71 (compatible; RevvimGort/5.0; +http://www.revvim.com; webmaster@revvim.com)SEO optimization crawler detecting performance issues
89RSiteAuditorMozilla/5.0 (compatible; RSiteAuditor)DataForSEO on-page audit crawler
90s4a\/Mozilla/5.0 AppleWebKit (compatible; s4a/1.0; +https://www.seo4ajax.com/webscraper)JavaScript rendering and SEO prerendering service
91S[eE][mM]rushBotMozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)Semrush SEO tool web crawler for analysis
92Screaming Frog SEO SpiderScreaming Frog SEO Spider/5.1Screaming Frog SEO tool web crawler
93SearchAtlasSearchAtlas.com SEO CrawlerSearchAtlas SEO web crawler for analysis
94SEBot-WASEBot-WASEO audit crawler performing technical analysis
95semaltbot\/semaltbot/0.1 (+http://semalt.net)SEO analytics crawler collecting ranking and performance data
96SendGrid Event APISendGrid Event APIEmail service crawler scanning links in messages
97SenutoBotSenutoBot/1.0 (compatible; SenutoBot/1.0; +https://www.senuto.com/)Senuto web crawler for SEO analysis
98seo-audit-check-bot\/Mozilla/5.0 (compatible; seo-audit-check-bot/1.0)SEO auditing crawler evaluating optimization signals
99SeobilityBotSeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)Seobility web crawler for SEO analysis
100SEOkicksMozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)SEOkicks web crawler for SEO analysis
101SEOlizerSEOlizer/1.1 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 (+htt…SEOlizer web crawler for SEO analysis
102seoLytMozilla/5.0 (compatible; seoLyt/1.0; +https://seolyt.com)SEO analysis and website optimization tool crawler bot
103seoscannersMozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)SEO Scanners web crawler for analysis
104SeoSiteCheckupMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/…SEO site analysis crawler evaluating search performance
105seostar\.coMozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)SEOstar web crawler for SEO analysis
106SeoulBotSeoulBotWeb crawler for SEO and competitive analysis
107SERankingBacklinksBotMozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)SEranking backlinks web crawler bot
108serpstatbot\/serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)Serpstat web crawler for backlink tracking
109SERPtimizerBotMozilla/5.0 (compatible; SERPtimizerBot; +http://serptimizer.com/serptimizer-bot)SEO analysis crawler gathering data for optimization
110SiteAuditBot\/Mozilla/5.0 (compatible; SiteAuditBot/1.1; +https://siteauditbot.com)Website SEO auditing and link checking crawler
111sitebulbMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Ch…Sitebulb SEO web crawler for analysis
112SiteCheck-sitecrawlMozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawlDigital governance platform auditing web quality
113siteexplorer\.infoMozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)Site Explorer web crawler for analysis
114SocialRankIOBotSocialRankIOBot; http://socialrank.io/aboutSocialRank web crawler for social analysis
115SonicMozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)Sonic web crawler for ranking analysis
116SparkPostSparkPostEmail service crawler validating links and tracking
117spbotMozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )SEO Profiler web crawler for analysis
118StatsDroneBotStatsDroneBot (https://statsdrone.com/statsdrone-bot-documentation/) Mozilla/5.0 (Windows NT 10.0; …Affiliate marketing link validation and monitoring bot
119StatusNestBacklinkSpiderMozilla/5.0 (compatible; StatusNestBacklinkSpider; +https://statusnest.com/)StatusNest backlink tracking crawler
120SurdotlyBotMozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iP…Surly web crawler for link shortening
121SurveyBotMozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR …DomainTools survey web crawler bot
122SwifteqLinkCheckerSwifteqLinkCheckerWeb crawler verifying link availability status redirects and basic page content validation
123tchelebi\/Mozilla/5.0 (compatible; tchelebi/1.0; +http://tchelebi.io)Web crawling tool indexing content for SEO data aggregation and competitive analysis purposes
124toplistbotTopList web crawler for ranking
125Traackr\.comTraackr.comTraackr influencer marketing web crawler
126trentwil\.esdomain research project (+https://trentwil.es/domains.html)Web crawler indexing content for data aggregation SEO analysis and competitive intelligence
127UASlinkChecker\/Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)Link checking crawler validating URL availability and HTTP status across websites
128Urlcheckr\/Urlcheckr/2.0Dead link checker scanner identifying broken hyperlinks and inaccessible web pages
129VeoozbotMozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)Veooz web crawler for marketing analysis
130WebCEO Online\/Mozilla/5.0 (compatible; WebCEO Online/1.0; +http://online.webceo.com)SEO platform crawler analyzing keywords and backlinks
131webprosbot\/webprosbot/2.0 (+mailto:abuse-6337@webpros.com)SEO crawler analyzing websites for optimization content and competitive research
132wmtips\.com\/Mozilla/5.0 (compatible; wmtips.com/2.0; +http://www.wmtips.com/tools/)SEO analysis crawler scanning websites for optimization issues and metrics
133woorankreview\/Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Ver…WooRank web crawler for SEO analysis
134WordCountBot\/Mozilla/5.0 (compatible; WordCountBot/0.1;)Content analysis tool computing word counts keyword density and readability
135wowLink Crawler\/wowLink Crawler/1.0Link management crawler indexing and analyzing web content for optimization
136WPMU DEV Broken Link CheckerWPMU DEV Broken Link Checker SpiderLink auditor identifying dead links bad anchors and HTTP errors for maintenance
137Xenu Link SleuthXenu Link Sleuth/1.3.8Xenu link checker web crawler tool
138xovibotMozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)Xovi SEO tool web crawler for analysis
139XoviOnpageCrawlerMozilla/5.0 (compatible; XoviOnpageCrawler; +http://www.xovi.de/)XOVI SEO suite on-page crawler
140YaK\/Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)Linkfluence YaK web crawler bot
141Zombiebot\/Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://w…Web crawler designed to index content for SEO analysis and competitive intelligence
142ZoomBotZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)SEOZoom web crawler for SEO analysis

Copyright © 2026 24KRMB.COM.All rights reserved.