If you are building autonomous agents and relying on temperature=0 for deterministic reproducibility, you are operating on a dangerous assumption. A recent discussion on Hacker News highlights a persistent reality in LLM inference: identical prompts can yield divergent outputs even when the temperature parameter is set to zero. For developers shipping agentic workflows, this isn't just a theoretical annoyance; it is a fundamental stability issue that breaks unit tests, evaluation harnesses, and production reliability.

The Myth of Deterministic Sampling

The core of the issue lies in the definition of temperature. In standard sampling, temperature controls the randomness of the next-token prediction by scaling the logits. Setting it to 0 theoretically forces the model to always pick the highest probability token (argmax). However, in practice, this 'greedy' decoding is not always bit-for-bit reproducible. Floating-point non-determinism across different hardware backends, batching effects, and parallel processing optimizations in modern inference engines (like vLLM or TensorRT-LLM) introduce microscopic variations. When an agent executes a multi-step chain, these tiny deviations cascade. A single token difference in step one changes the context window for step two, leading to a completely different trajectory by step ten.

Compounding Errors in Multi-Step Chains

The divergence becomes critical when agents are tasked with complex, multi-step reasoning. Unlike single-turn Q&A, agents maintain state across iterations. If an agent's internal monologue or tool selection diverges slightly due to non-deterministic inference, the subsequent actions can branch into entirely different logical paths. One run might choose to search the web, while another decides to use a calculator, leading to different final answers. This 'butterfly effect' in agent loops makes it nearly impossible to debug failures based solely on logs if you cannot reproduce the exact execution path.

When Does It Actually Matter?

So, when should you panic? If you are running a creative writing bot, divergence is a feature, not a bug. But for agentic workflows involving code execution, database transactions, or financial calculations, non-determinism is a showstopper. The Hacker News thread suggests that while we often treat LLMs as black boxes, we must accept that temperature=0 is a statistical preference for determinism, not a hard guarantee. For critical infrastructure, you need more than just a low temperature setting.

Key Takeaways

  • temperature=0 does not guarantee identical outputs across runs due to floating-point non-determinism and hardware variations.
  • Multi-step agent chains amplify small token-level differences into major trajectory divergences.
  • Reproducibility requires more than just parameter tuning; it needs pinned seeds, fixed hardware environments, and potentially deterministic inference engines.
  • Treat agent outputs as probabilistic distributions, not deterministic functions, especially in production environments.

The Bottom Line

Stop treating temperature=0 as a magic bullet for determinism. If your agent pipeline cannot tolerate variance, you need to implement robust state checkpointing, deterministic inference backends, or accept that your 'reproducible' tests are actually just probabilistic snapshots.