In the world of filesystem management, most tools are either too dumb to be useful or too smart to trust. Zygo Blaxell's 'bees' (Best-Effort Extent-Same) sits firmly in the latter category, but with a hacker-friendly twist. This btrfs deduplication agent is designed to scale up to large filesystems without melting your RAM. It combines offline dedupe with an incremental data scan, minimizing the time data spends on disk before being optimized.

Why This Beats the Naive Approach

The standout feature here is the space-efficient hash table. Bees can manage 10 TB of unique data using as little as 1 GB of hash table memory. That’s a 0.1GB/TB ratio, which is absurdly efficient for a userspace tool. Unlike other dedupers that balloon in resource usage as your dataset grows, bees maintains a constant hash table size. It doesn't care if your dataset shrinks; the RAM usage stays pinned, preventing unexpected spikes during heavy I/O operations.

Operational Reality Check

This isn't a set-and-forget magic trick. Bees requires root privileges, specifically CAP_SYS_ADMIN, and works exclusively with btrfs. It operates on live data without scheduled downtime, thanks to automatic self-throttling that reduces system load when resources get tight. However, the first run can increase metadata space usage if you have a lot of snapshots. It’s a trade-off: you get whole-filesystem dedupe that scans data only once, even with complex snapshot and reflink structures, but you pay for it in metadata overhead initially.

The Hacker's Caveat

Bees has no include/exclude filters. It doesn't accept file lists. It scans everything. This 'best-effort' approach means it prioritizes the largest extents first to recover the most free space during maintenance windows. It works with btrfs compression, deduping any combination of compressed and uncompressed files. But remember: it is btrfs only. If you're on ext4 or ZFS, you're out of luck. The project has been around since 2015, licensed under GPL v3+, and maintained by Blaxell via email and GitHub.

Key Takeaways

  • Bees uses a persistent hash table that allows for rapid restarts after shutdowns, ensuring continuity in long-running daemons.
  • It recovers more free space from btrfs than naive dedupers by leveraging extent-same logic and largest-extents-first processing.
  • The tool is designed for users who want incremental, background deduplication without the pain of manual maintenance windows.

The Bottom Line

Bees is the unglamorous workhorse your btrfs server needs if you’re drowning in redundant data but can’t afford the RAM for other dedup solutions. It’s technical, specific, and brutally efficient.