Pablo Felps, the developer behind the local-first Markdown editor Marksage, recently detailed a rigorous engineering process that resulted in shipping only two small AI models while discarding three others. The project challenges the prevailing trend of integrating large generative models into every application by demonstrating the efficacy of 'decision models'β€”tiny, specialized classifiers that pick answers from fixed lists rather than generating text. The shipped models weigh in at just 385 KB and execute in approximately 0.1 milliseconds within the browser, proving that high-precision, low-latency AI is achievable without heavy infrastructure.

The Case Against Generative Overkill

Marksage’s architecture relies on a strict contract where the AI never writes content; it only selects from predefined options. This approach eliminates the need for output parsing and validation, which are common failure points in generative AI integrations. The core interface, DecisionEngine, exposes three methods: choice for selecting an option, score for placing text on a scale, and noul for calculating statement probability. By restricting the AI to these deterministic tasks, Felps ensures that every suggestion is measurable against a ground-truth label, a luxury that generative output rarely affords.

Why the Better Model Didn't Ship

The most instructive part of the project is the rejection of 'Code Model v3.' On October 30, this new model achieved a higher overall precision of 93.8% compared to the incumbent's 93.1%. However, it failed its specific mission: distinguishing between 'near-twin' languages like TypeScript and JavaScript. On these critical pairs, v3’s precision dropped to 88.0%, below the incumbent’s 89.8%. The model was better on average but worse at its core purpose, leading to its exclusion from the production build.

Data Quality Trumps Architecture

The failure of v3 wasn't due to a flawed architecture but rather noisy training labels. Analysis revealed that only 42% of code blocks tagged as TypeScript in the training data actually contained TypeScript-specific syntax; the rest were valid JavaScript tagged by project origin. This insight underscores a critical lesson for LLM practitioners: no amount of sophisticated pooling or model tuning can overcome fundamentally ambiguous or mislabeled training data. The decision to stick with the slightly less accurate but more reliable v2 model highlights the importance of domain-specific precision over general benchmark scores.

The Bottom Line

Marksage proves that 'small' AI isn't just a budget constraintβ€”it's a design choice that prioritizes trust and measurability. In a landscape obsessed with parameter counts, the real win is shipping a model that knows exactly when to stay silent.

Key Takeaways

  • Decision models offer superior measurability compared to generative models because answers are binary right/wrong against labels.
  • A model with higher overall accuracy was rejected because it performed worse on its specific target task (language disambiguation).
  • Training data quality is the bottleneck; ambiguous labels for TypeScript/JavaScript prevented v3 from succeeding despite architectural improvements.
  • Local-first AI can be extremely efficient, with 385 KB models running in 0.1 ms in-browser without GPU acceleration.