In a recent blog post, the Project Lead for open-source marketing automation platform Mautic detailed their experiment with "Ruth AI," a custom Google Gemini Gem designed to deputize for them during a three-month ordination retreat. The initiative, inspired by a concept from the Exit Five podcast, aimed to preserve institutional knowledge and reduce the burden on volunteers and leadership who were covering the lead's responsibilities. The tool served as a first-line resource for the Mautic Council and leadership team, answering queries that typically required the lead's personal context.

The Technical Setup

The core of the implementation was a custom Gem in Google Gemini, configured with a lengthy system prompt and approximately ten uploaded reference files. The lead exported years of relevant AI chat history to capture their reasoning patterns and created a private Obsidian vault to document internal workflows, contact details, and task assignments. This vault was exported as a PDF using the Better Export PDF plugin and uploaded to the Gem, ensuring the AI had access to both public documentation and private operational knowledge. The Gem was shared with the leadership team, the Council, and key staff members, effectively turning a personal AI assistant into a shared organizational resource.

Successes and Hallucinations

The team actively used Ruth AI for routine tasks, such as locating email addresses for fiscal hosts, understanding translation review processes, and drafting responses. It proved particularly valuable for newer team members who hesitated to ask "basic" questions, providing a low-stakes environment for learning. However, the AI struggled with gaps in documentation. In one instance, it confidently directed a contributor to a non-existent release documentation folder, and in another, it invented payment methods for services. The lead noted that because the AI's tone mimicked theirs, the hallucinations were deceptively convincing, highlighting the danger of AI confidence without verified sources.

The Documentation Lesson

The primary takeaway from the experiment was that the AI was only as effective as the underlying documentation. The process of preparing Ruth AI forced a rigorous audit of the lead's daily, weekly, and annual tasks, revealing outdated habits and undocumented processes. The lead emphasized that the true value lay not in the AI itself, but in the plain-text, version-controlled knowledge base (the Obsidian vault) that it queried. This approach aligns with Mautic's digital sovereignty principles, ensuring that knowledge remains owned by the community rather than locked within a proprietary tool or a single individual's head.

Key Takeaways

  • Start early: The lead shared the AI four months before departure to allow for testing and feedback.
  • Treat questions as gaps: Every time a human had to answer a question, the answer was added to the documentation and the Gem updated.
  • Own your data: Keep knowledge in plain text (e.g., Obsidian) to avoid vendor lock-in and ensure portability.
  • Verify sensitive info: Always check financial, access, or decision-related answers with a human, as AI can be confidently wrong.
  • Audit your chat history: Exporting personal AI chats provided rich context for reasoning but requires redaction of sensitive data.

The Bottom Line

Ruth AI proved that an LLM is only as smart as its source material, turning a continuity experiment into a forced documentation audit. For open-source communities, the real win wasn't the AI deputy, but the plain-text knowledge base that outlasts any single tool or leader.