The Rust Project maintains an ecosystem of developer tools—Cargo, Clippy, rustfmt, Rust Analyzer, and Miri—that live in separate git repositories outside the main rust-lang/rust monorepo. Keeping these tools in their own repos enables faster CI pipelines, independent deployment cycles, and scoped permissions for maintainers. But integrating them back into the central compiler repository for distribution and breaking-change coordination has been a persistent headache—until they found Josh.

Why Submodules and Subtrees Both Failed

The team tried git submodules first, which work as simple pointers to external repositories. In practice, developers frequently forgot to properly initialize submodules with git clone --recursive or accidentally committed unrelated submodule changes due to git's "dirty" state behavior when switching branches. More critically, submodules prevented atomic PRs that touched both parent and subproject code—making breaking internal API changes a multi-step dance requiring separate PRs and careful coordination. Git subtrees offered a different approach by embedding complete repository history directly into the parent repo, enabling single-PR workflows for coordinated changes. However, performance became untenable at Rust's scale. While a patched version worked adequately for some tools, Miri's complex history—which involved moving large code chunks from Miri to rustc in a history-preserving way—made subtree operations simply never finish, even after hours of waiting.

Enter Josh: Git Subtree on Steroids

Josh is a Rust-written tool implementing sophisticated git filtering operations that can transparently split repositories into subrepositories, rearrange directory structures, exclude paths, linearize merge commit histories, and more. The Rust team uses it primarily as a dramatically faster alternative to git subtree with cleaner resulting history. To simplify their workflow across all Josh-managed projects, the team built josh-sync—a lightweight wrapper providing unified pull and push operations for bidirectional synchronization. They even open-sourced a reusable GitHub Actions workflow that periodically performs sync pulls, automatically opens PRs when needed, and alerts maintainers on Zulip if manual intervention is required.

Current Usage and Migration Plans

Josh currently powers subtree synchronization for Miri, Rust Analyzer, compiler-builtins, stdarch, and the Rust Compiler Development Guide. While a few projects like Clippy still use traditional git subtrees, the plan is to migrate everything over time. The team credits Josh maintainers—particularly Christian Schilling—for being highly cooperative in improving the tool to handle Rust's demanding use cases. One remaining pain point: pull syncs generate excessive merge commits in subprojects. The Josh team has already improved their trivial-merge-avoidance logic and is actively helping with migration to these better filters. Over years of use, Rust's workflows have served as a valuable stress test for Josh's performance capabilities, uncovering edge cases that have strengthened the tool for everyone.

Key Takeaways

  • Separate repositories enable faster CI and independent deployment but require robust synchronization
  • Git submodules break atomic PRs; git subtrees become unusable at scale with complex histories
  • Josh provides order-of-magnitude speed improvements over subtree operations
  • The Rust team built josh-sync to unify cross-project sync workflows

The Bottom Line

This is exactly the kind of tooling innovation that keeps large open-source projects from grinding to a halt. Rather than accepting git's limitations, the Rust team found and helped improve Josh—turning a multi-hour synchronization nightmare into something that's actually maintainable at scale.