AI bots made 1,635 of your 8,445 requests (19.36%).
Training crawlers 1,150Search and answer indexers 260User-triggered fetchers 155Other AI and data bots 70Classic search (baseline) 310
Bots
| Bot | Type | Requests | Errors | URLs | Bandwidth | Last seen | robots.txt |
|---|---|---|---|---|---|---|---|
| ClaudeBotAnthropic | Training | 420 | 25.5% | 89 | 9.4 MB | 2026-10-25 | partial |
| GPTBotOpenAI | Training | 320 | 6.2% | 89 | 9.0 MB | 2026-10-25 | partialignored robots.txt ×20unverified IP ×25 |
| BytespiderByteDance | Training | 260 | 9.6% | 77 | 7.0 MB | 2026-10-25 | blockedignored robots.txt ×260 |
| GooglebotGoogle | Baseline | 220 | 2.3% | 71 | 6.4 MB | 2026-10-25 | partial |
| Meta-ExternalAgentMeta | Training | 150 | 5.3% | 56 | 4.5 MB | 2026-10-25 | partial |
| OAI-SearchBotOpenAI | Search | 110 | 3.6% | 38 | 3.5 MB | 2026-10-25 | partial |
| BingbotMicrosoft | Baseline | 90 | 1.1% | 46 | 2.8 MB | 2026-10-25 | partial |
| PerplexityBotPerplexity | Search | 90 | 2.2% | 46 | 2.3 MB | 2026-10-25 | partial |
| ChatGPT-UserOpenAI | User fetch | 70 | 1.4% | 44 | 2.0 MB | 2026-10-25 | partial |
| AmazonbotAmazon | Other AI | 70 | 2.9% | 36 | 2.0 MB | 2026-10-25 | partial |
| Claude-SearchBotAnthropic | Search | 60 | 3.3% | 32 | 1.8 MB | 2026-10-25 | partial |
| Claude-UserAnthropic | User fetch | 45 | 4.4% | 32 | 1.4 MB | 2026-10-25 | partial |
| Perplexity-UserPerplexity | User fetch | 40 | 2.5% | 33 | 1.1 MB | 2026-10-25 | partial |
robots.txt column: blocked = "/" is disallowed, partial = some paths are disallowed, allowed = own group with no disallows, default = no rule applies.
Requests to URLs robots.txt disallows
/api/v1/itemsBytespider ×33/api/v1/usersBytespider ×24/private/report-5Bytespider ×14/private/report-3Bytespider ×12/private/report-4Bytespider ×10/private/report-3GPTBot ×7/private/report-4GPTBot ×3/private/report-6GPTBot ×3
Findings
- robots.txt still names 'anthropic-ai', a deprecated Anthropic token. Current crawlers are ClaudeBot (training), Claude-SearchBot and Claude-User.
- 25.5% of ClaudeBot requests ended in 4xx/5xx errors. Broken links or stale sitemap entries are wasting its crawl.
- ClaudeBot is a training crawler and your robots.txt does not block it. Add a group for it if you want to opt out.
- GPTBot made 20 request(s) to URLs your robots.txt disallows. robots.txt is voluntary: enforce it at the server or CDN (return 403) if it matters.
- 25 request(s) claiming to be GPTBot came from outside the IP ranges you supplied and are probably spoofed.
- GPTBot is a training crawler and your robots.txt does not block it. Add a group for it if you want to opt out.
- Bytespider made 260 request(s) to URLs your robots.txt disallows. robots.txt is voluntary: enforce it at the server or CDN (return 403) if it matters.
- Meta-ExternalAgent is a training crawler and your robots.txt does not block it. Add a group for it if you want to opt out.
AI requests per day
2026-10-01 to 2026-10-25, peak 81 requests a day.
Most requested URLs
/about54/pricing53/contact51/changelog51/46/api/v1/items34/docs/guide-1026/docs/guide-1225/api/v1/users25/docs/guide-124