A new index report from ReadableByAI reveals that approximately one-quarter of Y Combinator's Fall 2025 cohort has implemented measures to block AI training crawlers, effectively rendering their public-facing websites as blank pages to machine learning systems. The findings, shared via Hacker News on August 13th, highlight a growing tension between companies building AI products and those seeking to prevent their data from being used in model training.

What the Data Shows

The report analyzed publicly accessible web properties belonging to startups in Y Combinator's most recent batch, focusing on how these companies configure their robots.txt files and other crawler-access headers. Rather than relying solely on automated scanning, ReadableByAI's methodology appears to have involved manual verification of each startup's stance toward AI training data collection.

Why Startups Are Opting Out

For developers and technical founders, the decision to block AI crawlers isn't simply about protecting proprietary content. Many YC-backed startups are acutely aware that their documentation, product descriptions, and even error messages could end up being used to train competitor models or generic AI systems that might eventually displace them.

The Infrastructure Perspective

From an infrastructure standpoint, implementing crawler blocks is relatively straightforward but comes with tradeoffs. Standard approaches include adding entries to robots.txt files, using meta tags like "max-snippet:0" and "max-image-preview:none," or blocking known AI crawler user agents at the edge via CDN configurations. However, these measures also prevent legitimate search indexing by Google and Bing, potentially impacting organic discovery.

Implications for the AI Training Pipeline

The trend among YC startups reflects a broader shift in how companies think about their web content's role in AI training datasets. As model capabilities improve and more organizations seek to build on top of foundation models, the question of who controls training dataβ€”and at what costβ€”becomes increasingly contentious.

Key Takeaways

  • Approximately 25% of YC Fall 2025 startups block AI crawlers from their public sites
  • Blocking mechanisms range from robots.txt directives to CDN-level user agent filtering
  • The trend signals growing concern among early-stage companies about data rights
  • Search engine optimization often suffers as a side effect of crawler blocking

The Bottom Line

This YC cohort is essentially drawing a line in the sand on AI training dataβ€”something that's only going to become more complicated as the industry matures. Whether you're building infrastructure tooling, developer platforms, or SaaS products, you'd better have a clear policy on how your content gets used by machines. The startups that figure this out early will avoid awkward conversations with investors and customers down the road.