The author of Deallocated Objects has overhauled their static blog to better accommodate AI agents, recognizing that a significant portion of current traffic comes from tools like Claude, ChatGPT, and Perplexity rather than human readers. The update, completed in a single evening, transforms the nginx-served site into an agent-friendly resource without relying on complex frameworks or heavy dependencies.

Markdown Delivery via Content Negotiation

The core of the update involves serving every post as plain markdown in addition to HTML. Users can append '.md' to any post URL to retrieve the raw markdown, which includes YAML frontmatter with metadata like title, canonical URL, and dates. More impressively, the site implements HTTP content negotiation; if a client sends an 'Accept: text/markdown' header, nginx automatically rewrites the request to serve the markdown version. This approach is fortified with a 'Vary: Accept' header to prevent cache poisoning, ensuring agents get markdown while browsers receive HTML.

llms.txt and Structured Data

The blog now features an llms.txt file at the root, a plain markdown document that summarizes the site, lists all posts with links to their markdown versions, and explains citation preferences. A companion file, llms-full.txt, aggregates the entire site's content into a single ~225 KB file for easy ingestion. Additionally, the site serves a JSON Feed where each item includes a custom '_markdown_url' key, alongside standard Open Graph tags and JSON-LD structured data to help both search engines and AI crawlers understand the content hierarchy.

Explicit Robots and IndexNow Integration

The robots.txt file has been updated to explicitly allow major AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, using a new 'Content-Signal' directive to permit search, AI input, and training. The author also integrated IndexNow, a protocol that pings Bing and other search engines immediately upon publication. While Google does not currently support IndexNow, the author notes that Bing's index feeds many AI search tools, making the 30-line script a worthwhile investment for visibility.

Key Takeaways

  • Static sites can be made AI-ready with simple nginx config changes and build-time markdown generation.
  • Content negotiation via the 'Accept' header allows seamless switching between HTML and markdown formats.
  • The 'llms.txt' standard provides a clear entry point for agents to understand site structure and content.
  • Explicit robots.txt directives and IndexNow integration help ensure AI crawlers and search engines index content correctly.

The Bottom Line

Making your content machine-readable is no longer optional for technical blogs; if you aren't serving clean markdown and structured metadata, you are invisible to the agents that are increasingly becoming your primary audience.