Non-Determinism
The property that a system can produce different outputs from the same input. It is the root cause of why AI agents are hard to make reliable and why ordinary software-testing intuitions fail on them. A traditional function is deterministic — same input, same output, forever — so you can test it once and trust it. An agent is non-deterministic: the language model samples from a probability distribution over next tokens (even at temperature zero, floating-point and infrastructure effects leave residual variance), the context it sees shifts as memory and retrieval change, and the world it acts on differs every run. You can *reduce* variance (low temperature, tight prompts, structured outputs, pinned model versions) but not eliminate it, because the flexibility that lets an agent handle novel tasks is the same flexibility that makes its behavior vary — determinism and adaptability are the same dial. The practical consequence: an agent is never 'verified once and trusted forever'; it must be measured continuously, and any single passing run is one sample from a distribution, not a guarantee.
Written by Max Zeshut
Founder at Agentmelt · Last updated Sep 9, 2026
Example
Asked twice to 'reconcile this invoice against the PO,' an agent produces two valid but different plans — one checks totals first, the other line items; one calls the tax API, the other reasons about tax itself. Both may be correct, but they are not identical, which is why a `expect(result).toBe(x)` assertion either rejects good behavior or gets loosened until it tests nothing. The fix is statistical evaluation over many runs, not exact-match tests.
Frequently asked questions
- Can I make an AI agent deterministic?
- Not fully, and usually you shouldn't want to. Setting temperature to zero, pinning the model version, and demanding structured output reduce variance, but residual non-determinism remains from infrastructure and from the changing context and world the agent operates in. More importantly, driving the flexibility all the way down turns the agent back into a rigid script — the reason you reached for an agent was to handle cases a script couldn't. The goal is to engineer reliability around non-determinism, not to remove it.
- Why does non-determinism break normal testing?
- Because standard tests assume one correct output and that a green run is a guarantee. With a non-deterministic agent, a valid task can have several correct outputs, and passing once only proves the task is possible — not reliable. Honest agent testing runs each case many times and reports a pass rate (pass^k), grading behavior against a rubric that tolerates valid variation while catching real errors, rather than exact-match assertions.