If you would never hardcode a changing price into a compiled binary, why are you fine-tuning changing facts into a model's weights? That is the core argument from Nabeel Baghoor, an engineer at Null Studio, who likens fine-tuning on factual data to burying configuration in an opaque blob that no one can grep, diff, or audit. The industry's reflex to ask "can we train it on our data?" often ignores the operational reality that weights are terrible at handling dynamic information.

The Storage Architecture of AI

Baghoor proposes a clear mental model for where knowledge lives: prompts are config, retrieval is the database, and fine-tuning is compiled code. Prompts allow for instant, reversible behavioral changes without deployment. Retrieval-Augmented Generation (RAG) acts as the source of truth, allowing developers to update a document and have the next query reflect that change immediately, complete with source attribution and permission controls. Fine-tuning, by contrast, is best reserved for narrow behavioral tasks like strict output formatting or style consistency, where the pattern is hard to describe in text but easy to demonstrate.

Why Facts in Weights Break Production

Treating facts as weights creates significant maintenance debt. Every time a business detail changes, such as pricing or hours, a fine-tuned model requires a new training run, test cycle, and deployment, whereas a RAG system only needs a database row update. Furthermore, fine-tuned models lack a "stack trace"; they cannot point to the specific source of an answer, making debugging nearly impossible. Baghoor also highlights security risks, noting that once restricted information is trained into a model, it can potentially be extracted by anyone with API access, as system prompts are not true security boundaries.

A Practical Order of Operations

The article outlines a disciplined workflow that starts with building an evaluation set before touching any models. Developers should first fix the prompt with clear rules and examples, then add retrieval for dynamic facts, and only consider fine-tuning if there is a measured performance gap that prompting and RAG cannot close. Baghoor cites his own work with Retell voice agents, where nine client agents used zero retrieval and kept all facts in the prompt because the data sets were small enough to fit, proving that simplicity often beats architectural complexity for small business use cases.

Key Takeaways

  • Fine-tuning is for behavior, not facts: Use it for style, format, and narrow tasks, not for dynamic data like prices or hours.
  • Retrieval is the database: Keep facts in documents you can update, audit, and permission-check without retraining.
  • Prompts are config: They offer the fastest iteration cycle for behavioral changes and should be the first line of defense.
  • Hidden costs of fine-tuning: Data preparation, model churn, and vendor lock-in often outweigh the benefits for factual tasks.

The Bottom Line

Stop treating model weights like a database. If you cannot point to the source of an answer, you do not have a reliable system; you have a guessing game with a training bill.