What AI crawlers fetch from a small open-source site: 31 days of logs
We checked every request that claimed to be GPTBot, ClaudeBot, PerplexityBot and the rest against the operators' published IP lists. Then we looked at what the real ones read. The data is downloadable.