Understanding Cross-Entropy as a Loss Function

Loss Landscape#

The loss landscape provides a useful perspective on the training process. For LLMs, the goal is to model language. The model, loss function, and training data together define a landscape over the model parameters. The goal is to find a good surrogate loss function and training data that help us reach a place by gradient descent in that landscape where the model distribution is a good estimate of the target distribution.

Choosing a Loss Function#

So, first we want our loss function to have certain characteristics, and then we try to find a function that meets them. We want the expected loss to:

• Penalize distribution mismatch.
• Be minimized when the two distributions are the same.

For $L(P,Q)=\sum_i p_i f(q_i)$ with differentiable $f$ and at least three outcomes, requiring a unique minimum at $Q=P$ for every strictly positive $P$ gives [2]:

This gives the log-loss form $f(q)=-a\log q+b$.

Entropy#

Entropy is the minimum average number of bits per symbol needed for lossless encoding. For independent symbols, this limit can be approached by encoding long sequences [1, 2].

In a prefix code, no codeword can be the beginning of another. For 16 equally likely messages, four bits each is optimal. In the figure below, shortening $m_7$ to 011 rules out both 0110 and 0111 as other codewords. The illustrated rearrangement gives $m_5$ and $m_6$ the codes 01010 and 01011: one message saves a bit, but two others each gain a bit. With equal probabilities, the average length therefore increases. Independent, fair random bits cannot be compressed further on average.

To see how code lengths $\ell_i=-\log_2 p_i$ turn independent symbols into fair random bits, consider this example:

Symbol	Probability	Code
A	$1/2$	0
B	$1/4$	10
C	$1/8$	110
D	$1/8$	111

The pattern is simple: every branch of this code splits the probability equally between 0 and 1. With independent source symbols, every next bit is therefore a fair coin flip, regardless of the preceding bits.

The encoded stream is now independent random bits, so further compression is impossible on average. This makes the encoding optimal. Its average length is entropy:

$$ H(P)=\sum_i p_i\underbrace{(-\log_2 p_i)}_{\text{ideal code length}}. $$

Cross-Entropy and KL Divergence#

Cross-entropy is the average number of bits per symbol needed to encode data from $P$ using an ideal code optimized for $Q$.

Now suppose we design our code using a predicted distribution $Q$ instead of the true distribution $P$. Its ideal code lengths are $-\log_2 Q(i)$, but the symbols still arrive with frequencies $P(i)$. The average cost is therefore:

$$ H(P,Q)=-\sum_i P(i)\log_2 Q(i). $$

KL divergence measures the extra coding cost caused by using $Q$ instead of $P$:

$$ D_{\mathrm{KL}}(P\|Q)=H(P,Q)-H(P). $$

Since $H(P)$ is the optimal coding cost, $H(P,Q)\geq H(P)$, so $D_{\mathrm{KL}}(P\|Q)\geq 0$.

Cross-entropy includes both the source’s own uncertainty and the cost of a mismatched model. KL isolates the mismatch.

Cross-Entropy as a Loss Function#

When training a model, we want the model distribution $Q$ to match the data distribution $P$. Since $H(P)$ is fixed, minimizing $D_{\mathrm{KL}}(P\|Q)=H(P,Q)-H(P)$ is equivalent to minimizing cross-entropy. Using cross-entropy directly is simpler: we do not need to compute $H(P)$, which does not affect the optimization.

For outcomes $y$ drawn from the data distribution $P$, the expected loss is:

$$ L(Q)=\mathbb{E}_{y\sim P}\left[-\log Q(y)\right]. $$

Cross-Entropy and NLL Loss#

The negative log-likelihood (NLL) loss for an observed outcome $y$ is $-\log Q(y)$. Its connection to cross-entropy becomes clear when we average over the dataset: grouping identical outcomes turns their frequencies into probability weights.

$$ \underbrace{-\frac{1}{N}\sum_{n=1}^{N}\log Q(y_n)}_{\text{average NLL}} =-\sum_y\underbrace{\frac{\operatorname{count}(y)}{N}}_{\widehat P(y)}\log Q(y) =H(\widehat P,Q). $$

Average NLL is exactly cross-entropy under the empirical distribution $\widehat P$, and estimates cross-entropy under the true distribution $P$ when the examples are sampled from $P$.

References#

1. Grant Sanderson (3Blue1Brown). Reinventing Entropy | Compression is Intelligence Part 1. Video, 2026.
2. Grant Sanderson (3Blue1Brown). But what is cross-entropy? | Compression is Intelligence Part 2. Video, 2026.