University libraries are facing a crisis of infrastructure stability as AI companies aggressively scrape public archives to train large language models. The volume of automated traffic is so high that entire collections are being knocked offline, denying students and researchers the digital resources they rely on. This isn't just a bandwidth issue; it is a fundamental clash between the open-access philosophy of cultural heritage institutions and the extractive data needs of commercial AI firms.

The Scale of the Scraping Problem

The traffic isn't merely annoying; it is destructive. Jill Strykowski, a cataloguing and government documents lead at San Jose State University, described the bots as "bombarding" servers to grab as much material as possible. The situation has deteriorated since reports first appeared in late 2024, with Dean Seeman of University of Victoria Libraries noting that bot traffic sometimes causes sites to go down completely. The University of North Carolina at Chapel Hill recently reported over 11,000 searches for Finnish music from thousands of distinct IP addresses in a single day, periodically locking out human users. This scale of demand effectively puts regular people in competition with infinite bot traffic.

Why Traditional Defenses Fail

System administrators are finding that legacy tools are obsolete against modern scraper farms. Eric Hellman of Project Gutenberg characterizes these bots as poorly programmed "monstrosities" that ignore robots.txt files and request thousands of documents per second in every available format. Because these bot farms operate from massive pools of rotating IP addresses, simple IP blocking is no longer effective. While some libraries have implemented CAPTCHA challenges or login requirements to filter traffic, these measures risk blocking legitimate users and helpful bots, creating a barrier to the very open access these institutions strive to provide.

The Ethical and Technical Disconnect

There is a glaring irony in how AI companies consume public data. Hellman notes that Project Gutenberg offers a one-click download of its entire public domain library, yet the bots are "too stupid to use it," preferring to overwhelm the web interface instead. While major AI firms like Anthropic, OpenAI, and Google have expressed support for honoring standardized opt-out signals in robots.txt, this agreement is voluntary and excludes third-party scrapers and smaller startups. Michael Weinberg of GLAM-E argues that this dynamic creates a disincentive for libraries to digitize and share their collections, potentially reversing decades of progress in making information accessible to the public.

Key Takeaways

  • AI scraping bots are causing site outages at universities like UNC Chapel Hill and San Jose State.
  • Traditional defenses like IP blocking fail because bot farms use thousands of rotating addresses.
  • Major AI firms support voluntary robots.txt compliance, but third-party scrapers remain unregulated.
  • Libraries face a dilemma: implement access barriers that hurt humans or suffer downtime from bots.

The Bottom Line

We are watching the slow death of open access in real-time. If cultural institutions are forced to put up paywalls or login gates just to keep their servers from crashing, we lose the public good that made the early internet valuable. AI companies need to fix their scraping etiquette, or they will destroy the very data ecosystems they depend on.