Determinism in LLM Generation

If you send the same prompt to an LLM ten times, do you get ten identical answers? How would you make them as consistent as possible? [1]

Short answer: no. There are two causes: sampling, and the floating-point computation underneath it.

Cause 1: Sampling#

Assume the computation is exactly the same each time, so the logits $z$ are identical. The next token is sampled from a softmax with temperature $T$:

$$ p_i=\frac{\exp\left(\frac{z_i}{T}\right)}{\sum_{j}\exp\left(\frac{z_j}{T}\right)}. $$

A larger $T$ flattens the distribution. As $T \to 0$, all the probability moves to the largest logit, so decoding becomes greedy argmax: same logits, same answer.

Cause 2: Floating-Point Computation#

In practice the computation is not the same each time, so even $T=0$ is not enough. The root cause is one fact:

Floating-point addition is not associative. Each operation rounds its result, so the grouping matters: (1 + 1e-16) - 1 = 0 but 1e-16 + (1 - 1) = 1e-16. To add them, 1e-16 is aligned to the exponent of 1 by shifting its mantissa right by 54 bits. All of its bits are shifted out of FP64’s 52-bit fraction, so it adds nothing: 1 + 1e-16 stays 1.

So whenever a sum is grouped differently, the result can change. One common trigger is the batch size. Your output $y_j=\sum_{k} x_k w_{kj}$ only uses your own row; other requests never enter the sum. But the batch size $B$ decides how the kernel splits the work across the GPU’s compute units (SMs):

Same numbers, different summation grouping, so the results differ slightly [2]. The same regrouping shows up wherever a sum is split differently: attention computed in different chunks under prefix caching or chunked prefill, gradients from many GPUs added in a different order, … Once a tiny difference flips one token, every later token follows a different path.

People deal with this by fixing the grouping. For example, batch-invariant kernels keep the same split whatever $B$ is, so your result no longer depends on who else is in the batch [2].

How to Get Deterministic Outputs#

Sampling (Cause 1). Use $T=0$ or a fixed seed.

Associative law (Cause 2). Fix the summation order:

• Batch-invariant kernels (deterministic modes in vLLM and SGLang), so the batch size no longer changes the split.
• Prefix caching and chunked prefill off, so a request’s attention is computed the same way whether or not part of it was cached or chunked.
• The same hardware, library versions, and model snapshot, so the kernels pick the same split.
• …

Supplement: Intuition for Precision Formats#

A float spends its bits on two things: exponent bits set its range (how big or small a number can be), and mantissa bits set its precision (how exact it is). Fewer bits are faster and use less memory, so each part of the pipeline keeps only what it needs.

Common formats split their bits like this:

Format	Sign	Exponent	Mantissa
FP32	1	8	23
BF16	1	8	7
FP16	1	5	10
FP8 (E5M2)	1	5	2
FP8 (E4M3)	1	4	3

• Training runs in BF16. ML training can give up some precision, but not range: tiny gradients must not underflow to 0. BF16 keeps FP32’s range with fewer mantissa bits, at half the memory [3].
• Summing needs precision. Mixed-precision training keeps the same weights in two precisions, one for each stage of a training step [4]:
  • Forward and backward are mostly matmuls, which run much faster on 16-bit inputs (the sums inside are still accumulated in FP32).
  • The update adds to an FP32 master copy, because an update is often thousands of times smaller than the weight and would be lost in 16 bits, like 1 + 1e-16. It is only one addition per weight, so FP32 costs little here.
• Inference tolerates lower precision. The weights are fixed, so there are no tiny updates to lose. Rounding each weight does change the output, but usually only slightly, so weights are often stored in 8 or even 4 bits at a small cost in quality [5].

References#

1. 山景城小路 (@Zen_with_AI). An Interesting AI Interview Question. Post on X, Oct 12, 2026.
2. Horace He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, Sep 2025.
3. Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, et al. A Study of BFLOAT16 for Deep Learning Training. arXiv, 2019.
4. Paulius Micikevicius, Sharan Narang, Jonah Alben, et al. Mixed Precision Training. ICLR, 2018.
5. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.