Differential Entropy & Maximum Entropy

Continuous entropy h(X) = −∫ f log f. Why the Gaussian is nature's default distribution.

Differential Entropy & Maximum Entropy

Swap the sum in Shannon entropy for an integral and you get differential entropy, for continuous variables. It can go negative, and it explains why the Gaussian shows up everywhere.

The Differential Entropy Explorer

Pick a distribution, drag its parameters, and switch between bits and nats. Shrink a Gaussian's σ below about 0.24 and the readout turns red: differential entropy can go negative.
family:
closed form
h(X) = ½ log(2π e σ²)
h(X)
2.0471 bits

The classic max-entropy density on the real line at fixed mean and variance. Push σ below 1/√(2πe) ≈ 0.242 to see h(X) go negative.

Go deeper:The math objects

Shannon's entropy was defined for discrete random variables. To push the same idea into the continuous world we replace the sum with an integral and arrive at differential entropy: for a random variable X with density f(x), h(X) = −∫ f(x) log f(x) dx. The shape of the formula is identical to its discrete cousin, but its behaviour is meaningfully different — h can be negative, and its value depends on the units you measure x in.

  • Probability density f(x): a non-negative function on the real line whose integral is 1. The continuous analogue of a probability mass function.
  • Differential entropy h(X) = −∫ f log f dx: the expected value of −log f(X). It can be negative — for example, a very narrow Gaussian has h(X) < 0 — because f(x) can exceed 1 in places.
  • Closed-form values: h_uniform = log(b − a), h_exp = 1 − ln λ in nats, h_Gaussian = ½ log(2π e σ²), h_Laplace = 1 + ln(2b) in nats. These are easy to remember and cover most of practice.
  • Coordinate dependence: if Y = c X then h(Y) = h(X) + log |c|. So differential entropy depends on the units of measurement — meters versus millimeters changes h by log 1000. This is the price you pay for replacing a sum by an integral.
  • Why it still matters: differential entropy appears everywhere downstream — in mutual information for continuous variables, in the channel capacity of the additive white Gaussian noise channel, and in the entropy power inequality.

The Three Canonical Max-Entropy Distributions

Adjust each constraint and watch its maximum-entropy density change: uniform for bounded support, exponential for a fixed positive mean, Gaussian for a fixed mean and variance.

Three different constraint sets, three different max-entropy distributions. Adjust each panel's constraint and watch its PDF (and entropy) change.

Uniform on [a, b]
constraint: support is [a, b]
h = log(b − a)h = 1.000 bits
Exponential, rate 1/μ
constraint: X ≥ 0 and E[X] = μ
h = 1 + ln μ (nats)h = 1.443 bits
Gaussian N(μ, σ²)
constraint: E[X] = μ, Var(X) = σ²
h = ½ log(2π e σ²)h = 2.047 bits
bounded support
positive with mean μ
rate λ = 1/μ = 1.000
mean μ & variance σ²

Each density is the unique max-entropy choice given its constraint. Replace any of these distributions with a different shape that meets the same constraint, and h(X) will be strictly smaller — that is what "maximum entropy" literally means.

Go deeper:The maximum-entropy principle

Differential entropy gets interesting through the maximum-entropy principle. Among all densities satisfying a given set of constraints, the one that maximizes h(X) is, in a precise sense, the "least biased" choice — it assumes nothing beyond the constraints themselves. Three classic results follow: the uniform distribution is max-entropy on a bounded interval, the exponential is max-entropy on the positive reals at a fixed mean, and the Gaussian is max-entropy on the whole real line at a fixed mean and variance. That last result is one of the deepest reasons Gaussians appear everywhere — in measurement noise, in statistical models, in the central limit theorem, in physics.

  • Maximum-entropy principle: subject to a set of moment constraints, the density that maximizes h(X) is uniquely determined and has the form f(x) ∝ exp(λ₀ + Σᵢ λᵢ Tᵢ(x)) — an exponential family with one Lagrange multiplier per constraint.

Why the Gaussian Wins

A Gaussian, a Laplace, and a uniform, all with mean 0 and variance 1: the Gaussian has the highest h. Drag the morph slider toward Laplace and watch h drop.

All three densities have mean 0 and variance 1. The Gaussian wins — on the real line at fixed (μ, σ²), no other density has a higher differential entropy.

Mixture (1 − t)·Gaussian + t·Laplace, dashed white. h(X) = 2.0471 bits. Notice how it stays at or below the pure-Gaussian value.

Gaussian N(0, 1)h = 2.0471 bitsmax
Laplace (μ=0, b=1/√2)h = 1.9427 bits
Uniform [−√3, √3]h = 1.7925 bits

This is the maximum-entropy theorem in concrete form. Constrain the first two moments of a real-valued random variable, and the unique density that maximizes h(X) is the Gaussian. Anything else with the same mean and variance — heavier tails, lighter tails, compact support — has strictly less entropy.

Key takeaways

  • Differential entropy h(X) = −∫ f log f dx is entropy for continuous variables.
  • Unlike discrete entropy, h(X) can be negative and depends on the units of x.
  • Maximum entropy picks the least biased density that meets the constraints.
  • At a fixed mean and variance, the Gaussian has the most entropy.

Spot a mistake or have an idea?

Search