AI bots made 1,635 of your 8,445 requests (19.36%).

Training crawlers 1,150Search and answer indexers 260User-triggered fetchers 155Other AI and data bots 70Classic search (baseline) 310

Bots

BotTypeRequestsErrorsURLsBandwidthLast seenrobots.txt
ClaudeBotAnthropicTraining42025.5%899.4 MB2026-10-25partial
GPTBotOpenAITraining3206.2%899.0 MB2026-10-25partialignored robots.txt ×20unverified IP ×25
BytespiderByteDanceTraining2609.6%777.0 MB2026-10-25blockedignored robots.txt ×260
GooglebotGoogleBaseline2202.3%716.4 MB2026-10-25partial
Meta-ExternalAgentMetaTraining1505.3%564.5 MB2026-10-25partial
OAI-SearchBotOpenAISearch1103.6%383.5 MB2026-10-25partial
BingbotMicrosoftBaseline901.1%462.8 MB2026-10-25partial
PerplexityBotPerplexitySearch902.2%462.3 MB2026-10-25partial
ChatGPT-UserOpenAIUser fetch701.4%442.0 MB2026-10-25partial
AmazonbotAmazonOther AI702.9%362.0 MB2026-10-25partial
Claude-SearchBotAnthropicSearch603.3%321.8 MB2026-10-25partial
Claude-UserAnthropicUser fetch454.4%321.4 MB2026-10-25partial
Perplexity-UserPerplexityUser fetch402.5%331.1 MB2026-10-25partial

robots.txt column: blocked = "/" is disallowed, partial = some paths are disallowed, allowed = own group with no disallows, default = no rule applies.

Requests to URLs robots.txt disallows

Findings

AI requests per day

2026-10-01 to 2026-10-25, peak 81 requests a day.

Most requested URLs