The internet is loud with opinions about AI crawlers, but your website’s access logs hold the only data that actually matters. A new tutorial from seotto provides a practical Python script that parses combined log formats to classify requests into four distinct categories: search crawlers, AI training bots, AI search bots, and user-triggered fetches. This approach empowers developers to make data-driven decisions about blocking or allowing bots, rather than reacting to viral headlines.

The Classification System

The script relies on a curated list of user agent strings to sort traffic. Search crawlers include familiar names like Googlebot, bingbot, and DuckDuckBot. AI training agents are identified by strings such as GPTBot, ClaudeBot, CCBot, and Bytespider. The tutorial distinguishes these from AI search bots like OAI-SearchBot and PerplexityBot, as well as user-triggered fetches like ChatGPT-User. Crucially, the author notes that tokens like Google-Extended do not appear in logs because Google’s AI products use the standard Googlebot user agent.

Verifying Genuine Traffic

User agent strings are trivial to fake, so the script’s output must be cross-referenced with published IP ranges. The guide recommends using Python’s ipaddress module to check if requests claiming to be from OpenAI or Anthropic originate from their official CIDR blocks. Any agent mimicking a major crawler from a residential IP is likely a scraper and should be categorized as 'other-bot.' This step is vital because unverified scrapers often account for the load that site owners mistakenly attribute to legitimate AI training.

Interpreting the Metrics

Running the script against a month of logs reveals more than just request counts. The output breaks down bandwidth usage in megabytes, helping you identify if a crawler is downloading large assets or just hitting small, cached pages. Status codes provide further insight: a high volume of 4xx errors suggests the bot is chasing stale links or guessed paths, while 3xx codes indicate it is following redirects. Additionally, the script highlights which site sections, such as /blog or /docs, are being targeted, revealing whether the crawlers are collecting your published content or getting stuck in crawl traps.

Key Takeaways

  • Use your own access logs for ground-truth data on AI bot traffic.
  • Always verify user agents against published IP ranges to filter out fakes.
  • Monitor bandwidth and status codes to understand the real cost of crawler activity.
  • Re-run the script after any robots.txt changes to confirm the impact.

The Bottom Line

Don't let panic drive your robots.txt decisions; let your logs guide your rate limits.