The abstraction layer is a lie. Developers relying on Microsoft.Extensions.AI to swap hosted LLMs for local Ollama instances are falling into a trap of silent failures. While registering a new IChatClient gets the code to compile, the operational assumptions of a shared, robust cloud API do not transfer to a single-process local server. Michael Maurice, writing for DEV.to, details five specific mistakes that cause confident but wrong answers and catastrophic latency spikes in .NET applications running local models.

The Silent Truncation Trap

The most dangerous failure mode is the silent context window truncation. Hosted APIs explicitly reject oversized prompts, but Ollama’s default 4096-token context window often cuts the input, logs a warning on the server side, and returns a standard HTTP 200 response. Maurice’s simulator data showed that 15 of 29 long-document questions returned wrong answers despite the 200 status code. The oldest content, frequently the system prompt, gets discarded first. Developers must inspect the prompt_eval_count in the response to detect truncation, as there is no explicit flag in the API.

Context Is a Request Property, Not a Model Property

A common misconception is that the context window is inherent to the model. In reality, it is a property of the request. If you do not send num_ctx, the server uses its default, potentially ignoring the model's actual capacity. ChatOptions in .NET has no typed property for this, making it easy to overlook. The fix requires the gateway to send the window and keep-alive on every request, with a planner deciding what fits before the call to avoid silent truncation. A pessimistic estimate that occasionally refuses a prompt with a clear 413 error is preferable to a quietly wrong answer.

Concurrency and Cold Start Nightmares

Local Ollama defaults to OLLAMA_NUM_PARALLEL=1, forcing all concurrent requests into an invisible server-side queue. In Maurice’s tests, 30 simultaneous requests resulted in a p95 latency of 3,311 ms. Unlike cloud providers that scale, a local server simply makes users wait. The solution requires a bounded gate in the application layer that sheds load with a fast 429 response rather than letting requests pile up. Furthermore, the default five-minute idle timeout means the first request after a pause pays the cost of model loading. A cold start can take nearly 1,700 ms compared to 96 ms when warm, making readiness checks based on process uptime useless.

Resilience Handlers Need Tuning

The standard .NET resilience handler is misconfigured for local LLMs. AddStandardResilienceHandler defaults to a 10-second timeout and retries on failure. For local models, a cold load can exceed 10 seconds, and retrying a timed-out request often doubles the load on a server that is still computing the original request. Maurice recommends setting attempt timeouts to match realistic time-to-finish metrics and restricting retries to connection failures or 503 errors, never timeouts.

Key Takeaways

  • Ollama returns HTTP 200 even when truncating prompts; always compare prompt_eval_count against your input size.
  • Explicitly set num_ctx on every request; do not rely on server defaults or model capabilities.
  • Default concurrency is 1; implement application-level gating to shed load with 429s instead of queuing.
  • 'Healthy' process status does not mean the model is loaded; use warm-up workers and readiness checks.
  • Disable retries on timeouts for local LLMs to avoid doubling server load during cold starts.

The Bottom Line

Treating local Ollama like a hosted API is a recipe for silent data corruption and poor user experience. You must build explicit controls for context planning, concurrency gating, and warm-up readiness into your .NET application layer to make local inference viable.

Sources

https://dev.to/michael_maurice/5-mistakes-i-made-swapping-a-hosted-llm-for-local-ollama-in-net-4hke