It is easy to blame the market when your new side project gets zero traction. But before you pivot or give up, check your infrastructure plumbing. A developer using the handle haruwalks recently discovered that his micro-shop, selling personalized letters based on Street View walks for $1.20, had received only one payment in five months. The culprit wasn't lack of demand or poor copywriting. It was a single, immutable line in a robots.txt file provided by his hosting platform.
The Default Block That Can't Be Overridden
The shop was hosted on a free subdomain at surge.sh. When haruwalks investigated why traffic was nonexistent, he ran a simple curl command against the robots.txt endpoint. The result was stark: 'User-agent: *' followed by 'Disallow: /'. This configuration tells every web crawler, including Googlebot, to stay away from the entire site. The critical failure point here is that uploading a custom robots.txt to the site root does not override the platform's default. The edge server keeps its own version, effectively sealing the site off from search engine indexing regardless of what the developer uploads.
Orphaned Content and The Illusion of Visibility
Even if the robots.txt issue is resolved, the underlying platform architecture presents another hurdle. The app haruwalks used to publish content serves 'Allow: /' and renders post bodies server-side, which is good for SEO. However, the sitemap exposes only 22 static pages, none of which link to individual blog posts or product pages. This creates an orphaned content problem. Google can crawl the homepage, but there is no crawlable path to the specific text haruwalks published. Without internal links or sitemap entries pointing to the individual letters, the content remains invisible to search engines, rendering the 'Allow' directive moot.
Debugging Demand vs. Discoverability
The developer's experience highlights a common trap for indie builders: conflating low conversion rates with low traffic. When you cannot see your own content in search results, metrics like dwell time and conversion rates are measuring an empty room. Haruwalks noted that searching for the specific village name he wrote about, PrΔ dzonka in Pomerania, returned Wikipedia, a Polish statistics site, and a Google Maps pin, but not his letter. The content existed, but it was structurally isolated from the discovery mechanisms of the web.
Key Takeaways
- Always verify your robots.txt file using 'curl -s -A "Googlebot/2.1" https://your.site/robots.txt' to see exactly what crawlers are allowed to do.
- Free hosting tiers often impose rigid infrastructure rules, such as non-overridable robots.txt files, that can silently kill SEO efforts.
- Sitemap completeness matters more than just having one; ensure all dynamic content pages are linked and indexed to avoid orphaned posts.
- Before assuming product-market fit failure, rule out technical discoverability issues, especially on managed platforms with opaque edge configurations.
The Bottom Line
Stop guessing about demand and test the plumbing instead. If your robots.txt says Disallow, your funnel is closed at the front door, and every number downstream is meaningless.