Integrating Alishahryar1/free-claude-code into production CI pipelines requires more than just checking if it works; it requires knowing where it breaks. A new benchmark test conducted on AWS t3.medium instances simulated 50 concurrent IDE requests against the tool's multi-provider architecture to measure token throughput, tail latency, and failover reliability. The results expose a critical divide between median performance, which looks competitive, and p99 latency, which reveals a severe scaling bottleneck.

Median Performance vs. Tail Latency Reality

For the 47 successful requests out of 50, free-claude-code delivered a median Time-to-First-Token (TTFT) of 287ms, which is 8% faster than the B-Lost relay baseline of 312ms. Median throughput also edged out the baseline, hitting 42.3 tokens per second compared to 38.1. However, these aggregate numbers mask a deteriorating experience at the upper percentiles. The p95 TTFT spiked to 1,840ms (a 107% increase over baseline), and the p99 TTFT reached 4,210ms, nearly tripling the baseline's 1,520ms. This indicates that while the tool is efficient for light loads, it struggles significantly as concurrency increases.

Failover Mechanics and Error Rates

The benchmark observed a 6% error rate, with 3 requests exceeding the 30-second client timeout. Logs revealed 8 automatic failovers from Claude to Gemini during the latency spike. Each provider switch added approximately 600ms of handshake overhead, contributing to the tail latency. The analysis highlighted that the tool's retry logic attempts three sequential providers before failing, but lacks a circuit breaker. This absence means that when the primary provider is degraded, requests queue up rather than failing fast, exacerbating the latency problem for all concurrent users.

Production Thresholds and Mitigation Strategies

The author recommends using free-claude-code for solo developer workflows or CI jobs with fewer than 20 parallel tasks, where the 1.3B+ free token pool makes it economically superior to metered APIs. For team-scale integrations exceeding 50 concurrent users, the p99 spike becomes a UX liability. To mitigate this, the test team implemented a pre-flight provider health check to fail fast if the primary provider is down, switching to a fallback URL immediately. Additionally, adjusting the asyncio.Semaphore concurrency limit to 30 and adding request deduplication for identical prompts dropped the error rate to 1.2% and p99 latency to 2.1s.

Key Takeaways

  • Median TTFT and throughput are competitive with paid relays, but p99 latency is 177% worse than the B-Lost baseline.
  • 50 concurrent requests triggered 8 provider failovers, adding ~600ms overhead per switch and causing timeouts.
  • Lack of a circuit breaker in retry logic leads to request queuing during provider degradation.
  • Reducing concurrency limits to 30 and adding request deduplication significantly improves stability for team use.

The Bottom Line

Free-tier LLM tools are great for hobbyists, but without circuit breakers and strict concurrency controls, they become a latency nightmare for teams. Don't let median stats fool you; your users feel the p99. If you are building a real-time IDE integration, you need managed reliability or aggressive client-side throttling. The 1.3B free tokens are a trap if your users are waiting 4 seconds for the first token.