The hype around AI skills has hit a wall of reality. A recent audit of 777 skills from major public GitHub collections found that 267 failed basic security and validity checks. The study, conducted by developer Dongdarealest, used a custom tool called SkillDoctor to scan for common pitfalls like credential exposure and prompt injection risks. While curated sets like Anthropicβs remained clean, the larger, community-driven collections showed significant hygiene issues.
The SkillDoctor Methodology
SkillDoctor is a zero-dependency Python script that evaluates each SKILL.md file against four critical criteria: validity, security patterns, conflicts, and token cost. The audit covered six distinct collections, ranging from small, maintained repositories like superpowers (15 skills) to massive open repositories like ECC (500+ skills sampled). The results were stark: 310 skills passed, 200 warned, and 267 failed outright. The failures were heavily concentrated in the ECC collection, which scored a health rating of 76 compared to scores above 90 for curated sets.
Security Risks and Token Bloat
The audit highlighted specific security concerns across the dataset. 136 skills mentioned API keys, tokens, or passwords, and 97 referenced credential stores. Most alarming were the 8 skills that contained high-severity patterns, such as instructions to read or send credential material over the network. Beyond security, the audit exposed a hidden cost: token bloat. If all 777 skills were loaded into a single context window, they would consume approximately 1.76 million tokens. This 'rent' on every conversation is often overlooked by users who install skills as casually as editor themes.
The Illusion of Safety in Open Collections
The disparity between curated and open collections suggests that maintenance scale directly impacts security. Small, actively maintained sets like mattpocock/skills and addyosmani/agent-skills maintained high health scores (99 and 92, respectively), while the larger, less-managed ECC collection suffered from duplicate names and near-identical descriptions that confuse AI agents. The author notes that while a 'fail' verdict doesn't automatically mean malware, it indicates a lack of scrutiny that is unacceptable given the rising privilege level of AI agents.
Key Takeaways
- Failure Rate: 267 out of 777 audited skills failed basic security and validity checks.
- Credential Exposure: 97 skills referenced credential stores, with 8 showing high-risk exfiltration patterns.
- Token Costs: Installing all audited skills would cost ~1.76 million tokens per context load.
- Curated vs. Open: Small, maintained collections scored 90+ health, while large open collections like ECC scored 76.
- Tooling Available: SkillDoctor is now open-source, allowing users to audit their own skill libraries via
pip install skillcheckup.
The Bottom Line
Stop treating AI skills like browser extensions. If you aren't reading the SKILL.md or running a checker like SkillDoctor, you are blindly granting shell access and token budget to unverified instructions. The ecosystem is growing faster than its security practices.
Resources
- Audit Report: https://github.com/Dongdarealest/skilldoctor/blob/main/AUDIT.md
- SkillDoctor Tool: https://github.com/Dongdarealest/skilldoctor
- Curated Packs: https://github.com/Dongdarealest/skillport