The token-saver add-on market is a graveyard of broken promises, but JetBrains just published the first numbers that actually hold up. Their latest benchmark tested Ponytail Skill for Claude Code across 80 paired tasks using HarnessHarbor 0.18, and the results tell a more nuanced story than the headline claims: −15% code written (not 54%), −10.3% cost savings (not 20%), but this time it's statistically significant at p=0.004. That's the first solid positive result in their ongoing series dissecting these tools.

The Catch Nobody Talks About

Here's the thing that'll bite you if you just copy SKILL.md into a skills folder and walk away: Ponytail self-activates exactly zero times out of ten test sessions when left to Claude Code's own judgment. Zero. Never. The skill sits there installed, visible, ready — and the model never reaches for it. That's not a bug; it's by design. Ponytail ships as a plugin with a SessionStart hook that injects its ruleset automatically. If you don't use that installation path, you're measuring nothing at all.

How the Ladder Actually Works

Ponytail's core premise is elegant: before writing anything, climb a decision ladder — does this need to exist, is it already in the codebase, does the standard library handle it, can it be one line? Only then write the minimum viable code. Validation, error handling, security, and accessibility are explicitly off the chopping block. In practice, that looked like this: both agents were asked to export a three.js scene to Blender-ready OBJ format. The plain agent built a wrapper Group object with named intermediate steps across ten statements. Ponytail rotated the existing root directly in five statements. Same geometry exported, same 1.0 score — but half the code.

Where the Effect Lives and Dies

The −15% headline masks something more interesting: the savings concentrate hard on tasks where there's room to over-build. On larger builds, code fell 31%. On tasks where the plain agent already wrote lean code? Ponytail barely moved — though it actually persisted more lines in some cases because it wrote scripts to files while the baseline piped logic through an interpreter as heredocs. That seven-task gap is worth sitting with. The counter saw zero lines for one arm and 51–230 for the other, not because Ponytail wrote more code but because file-persisted code gets counted differently than inline throwaway analysis. Net effect on the headline: roughly 190 lines out of 10,205 — under 2%.

The Numbers That Matter

A typical task cost 10.3% less with Ponytail installed across all 80 pairs: cheaper on 46 tasks, dearer on 34. That's p=0.004 — the strongest positive cost result in this series by a country mile. RTK gave you a +7.6% increase (statistically significant, just pointing the wrong direction). Caveman landed around 10% cheaper once they removed a pricing-tier outlier, but that was fragile. What didn't move: input tokens fell 8.4%, fresh tokens 3.9% — neither statistically significant. Ponytail attacks what the model writes, not what it reads. On this workload, that's where the money lived.

Quality Remains Indistinguishable

Nine tasks scored slightly worse with Ponytail, six slightly better, 65 identical. That's a null result, not a clean bill of health — and JetBrains is upfront about that. This run wasn't powered to prove equivalence; proving quality genuinely unchanged would take several hundred paired tasks per arm rather than 80. What the data rules out is the obvious failure mode where writing less quietly breaks tests. One small note on adherence: Ponytail's ruleset asks models to mark deliberate shortcuts with a "ponytail" comment naming the ceiling and upgrade path. Across 80 trials with the ruleset demonstrably in context, that happened exactly once. The ladder gets followed; the paperwork does not.

Key Takeaways

  • Installation method is everything: use the plugin SessionStart hook or get zero activation
  • Real cost savings of ~10% are defensible on this workload — advertise 20% and you're overselling by half
  • Code reduction concentrates where over-building exists; expect 31% cuts on larger builds, near-zero elsewhere
  • Quality impact is undetectable at this sample size — not proven safe, just no obvious signal

The Bottom Line

Ponytail actually delivers real savings. That's more than most tools in this space can claim after independent testing. But if you're expecting the 54% code reduction from their marketing materials, you need to be running tasks stuffed with front-end over-building traps — not data and analysis work. Know your workload before you trust the numbers.