Crawlers

Crawlers in the week ending 2026-oct-16

Crawler		Visits
ClaudeBot/1.0		3028
Bytespider		1145
YouBot/1.0		657
Safari/537.36		547
AhrefsBot/7.0		379
MJ12bot/v2.0.7		302
OAI-SearchBot/1.4		302
MJ12bot/v1.4.8		292
Gort		261
HaloBot/1.0		249
CriteoBot/0.1		243
OAI-SearchBot/1.0		233
Googlebot/2.1		215
SemrushBot/7		214
facebookexternalhit/1.1		208
DotBot/1.2		204
DuckAssistBot/1.2		144
MJ12bot/v2.0.5		140
Applebot/0.1		121
coccocbot-web/1.0		120
YandexBot/3.0		111
bingbot/2.0		95
ExaSearchBot/1.0		88
Reflectionbot/1.0		81
IAS Crawler		80
PetalBot		76
PerplexityBot/1.0		65
SERankingBacklinksBot/1.0		44
QlyzeBot/1.0		40
Safari/604.1		40
Go-http-client/1.1		37
Safari		34
KeenableBot/1.0		30
Qwantbot/1.0		27
360EE		25
Barkrowler/0.9		25
SeznamBot/4.0		23
SemrushBot-BA		22
LyonlBot/1.0		22
Safari/605.1.15		18
DataForSeoBot/1.0		18
Bravebot/1.0		18
Firefox/127.0		18
Firefox/125.0		16
RetroDocumentResearch/0.1		16
WellKnownBot/0.1		12
YisouSpider		12
Sogou web spider/4.0		11
SiteAuditBot/0.97		11
python-requests/2.31.0		11
SpiderLing		10
NVBot		10
Edg/120.0.0.0		10
Edg/101.0.1210.47		8
IronFountain-Leads/1.0		8
Safari/537.3		8
CCBot/2.0		8
wpbot/1.4		7
Yeti/1.1		7
VivoBrowser/20.2.3.0		7
IbouBot/1.0		7
Claude-User/1.0		7
AdstxtCrawler/1.0		6
360Spider		6
Turnitin		6
serpstatbot/2.1		6
BeansLLM-CorpusBot/1.1		6
AwarioSmartBot/1.0		5
Mozilla/5.0		5
Safari/537.36		5
YaK/1.0		5
webXtrendBot/1.1		5
CricketBot/1.0		5
TerraCotta 0.2		4
AionBot/1.0		4
GrokWebsiteBot/1.0		4
SourceURLDiscovery/1.0		4
Twitterbot/1.0		4
Core11LocalResearch/1.0		4
Safari/604		4
AdsNexusBot/1.0		4
AwarioBot/1.0		4
python-httpx/0.28.1		4
Firefox/126.0		3
SeekportBot		3
GoogleOther		3
SEOJuice-SearchBot/1.0		3
BitSightBot/1.0		3
AudigentAdBot		3
Firefox/131.0		3
Scrapy/2.19.0		3
Firefox/129.0		3
SEOkicks		3
serpex-index/1.0		3
curl/8.7.1		2
Dataprovider.com		2
LohiSoftBot/1.0		2
DesearchBot/1.0		2
DocSpider/0.1		2
TestUserAgent		2
Meridian		2
AgentDataBot/1.0		2
aurora-robotstxt-filter/1.0		2
AgentTrustBot/1.0		2
UrsulaBot/1.0		2
BuiltWith/1.4		2
cohere-ai		2
MailCrawler/2.0		2
LanternBot/1.0		2
BitMoversBot/1.0		2
PoweredByBot/1.0		2
ShapBot/0.1.0		2
BFB2B-StoreIndex/0.1		2
LeadGennyBot/1.0		2
FundigBot/1.0		2
VelenPublicWebCrawler/1.0		2
AIWebIndex/2.0		2
undefined		2
DomainStatsBot/1.0		2
RDTvlokipBot/7.2		2
NetAPI/1.1		2
MapTheNetBot/1.0		2
VariableRatio-PublicAssetResearch/1.0		2
AteveSearchSourceUrlDiscovery/0.1		2
RepetivBot/1.0		2
SSI-Nutch/1.23		2
XiaoMi/MiuiBrowser		2
DomynResearchBot/0.1		2
Firefox/128.0		2
Harvester/2.0		2
Firefox/136.0		2
ChakloonBot/1.0		2
AstrionBot/1.0		2
lawa/0.1		2
Jambot/0.7.2		2
br-crawler/0.5		2
CompanyMapBot/0.1		2
ev-crawler/1.0		1
VexelSearchBot-1.0		1
ASRbot/1.2		1
HypernoAILeadCollector/1.0		1
OPR/109.0.0.0		1
panscient.com		1
Jambot/1.0.0		1
Edg/154.0.0.0		1
TflowSitemapBot/1.0		1
mfree-indexer/0.1		1
search.marginalia.nu		1
SojournerBot/1.0		1
ArnorBot/0.1		1
SofyaBot/1.0		1
GPTBot/1.4		1
ClueWeb-Crawler/1.0		1
onebit-crawl/0.1		1
Firefox/153.0		1
AdkernelTopicCrawler/1.0		1
Java/21.0.7		1
DuckDuckBot-Https/1.1		1
okhttp/3.0		1
AppleWebKit/537.36		1
kAIzr-registry/0.1		1
Firefox/148.0		1
Firefox/134.0		1
Safari/312.3.1		1
Firefox/135.0		1
Safari/312.3		1
Chrome-Lighthouse		1
emetriqContextualBot		1
Unique: 168		Total: 10477