The Randomness You Didn't Ask For: Understanding Non-Determinism in LLMs
·1704 words·8 mins
Non-determinism in LLMs creates real operational problems: flaky tests, irreproducible bugs, compliance nightmares, and unreliable agents. Most people only know about token sampling, but randomness creeps in across six distinct layers—from floating-point variance to hidden system prompts. Temperature=0 and random seeds help less than you’d hope because they constrain token selection, not reasoning paths. The solution requires structural constraints, not parameter tuning. Until then, you’re rolling dice in production.