RhinoDevel has released Poldi, a Private Offline Local Digital Intelligence designed specifically for users with limited GPU resources. Unlike many local AI solutions that assume high-end hardware, Poldi is engineered to run effectively on consumer-grade systems, targeting what the developer calls the "GPU poor." The tool functions as a personal chat partner and assistant, featuring both speech and terminal interfaces while ensuring all data remains local.

Optimized for Low-VRAM Efficiency

The core philosophy of Poldi is aggressive optimization for RAM usage and module selection to support smaller large language models (LLMs). The project explicitly critiques other solutions that fail when deployed with 12B or 27B parameter models, arguing that their described capabilities do not hold up on modest hardware. By tuning speech interface components and long-term memory implementation, Poldi aims to provide a functional always-on AI experience without the context overflow issues common in less optimized stacks.

Persistent Memory and Infinite Chat Features

Poldi introduces an "infinite chat" feature that automatically manages context limits. When the LLM's context window fills, the system summarizes the earlier half of the conversation, clears the context, and reinserts the summary alongside the recent dialogue. This mechanism allows for continuous interaction without manual resets. Additionally, the software supports automatic resume capabilities; if the application is powered off and restarted, it reimports the last chat state, allowing users to pick up exactly where they left off, with the system tracking time gaps in conversation continuity.

Modular Architecture and Local Search

The system relies on a modular architecture that includes built-in tools for Wikipedia and internet search via SearXNG instances, as well as chat history search. These modules are activated dynamically by a classifier, which determines when external knowledge retrieval is necessary. All chat data is stored in a local SQLite database, enabling the extraction of "memory statements" about the user. These memories are automatically injected into prompts when relevant, creating a personalized experience that remains entirely private, barring the specific exceptions of web-based searches.

Key Takeaways

  • Poldi is explicitly designed for consumer hardware, optimizing for low VRAM and CPU-only inference scenarios.
  • The software features an "infinite chat" mode that auto-summarizes context to prevent overflow and maintain conversation flow.
  • Data privacy is prioritized, with local SQLite storage for chat history and memory, though Wikipedia and web searches require internet access.
  • The tool supports both hands-free speech clients with wakeword detection and a web client accessible via browsers, including smartphones.

The Bottom Line

Poldi offers a pragmatic, builder-focused approach to local AI, stripping away the bloat that makes many open-source assistants unusable on mid-range hardware. Its emphasis on persistent state and modular, local-first design makes it a compelling tool for developers seeking a private, always-on companion without the cloud subscription tax.