Many of the probability distributions you encounter—like the normal, Poisson, and Bernoulli—belong to a powerful, unified class known as the exponential family. This article introduces the concept, explains its structure, and rewrites familiar distributions in exponential form with step-by-step math.

What is the exponential family?

A distribution belongs to the exponential family if its probability density or mass function can be written as:

Each part plays a specific role:

• $h(x)$: base measure
• $\eta(\theta)$: natural parameter
• $T(x)$: sufficient statistic
• $A(\theta)$: log-partition function (normalizes the distribution)

Why is this form useful?

• Sufficient statistics are made explicit
• Maximum likelihood estimation (MLE) becomes easier
• Conjugate priors exist for Bayesian analysis
• It enables unified analysis and transformation across distributions

Example 1: Bernoulli Distribution

The Bernoulli distribution is:

Step 1: Convert to exponential form

Step 2: Identify exponential family parts

• $\eta(\theta) = \log\left( \frac{\theta}{1 - \theta} \right)$
• $T(x) = x$
• $A(\theta) = -\log(1 - \theta)$
• $h(x) = 1$

✅ Bernoulli is in the exponential family.

Example 2: Poisson Distribution

The Poisson distribution is:

Step 1: Exponential form

Step 2: Map to the exponential form

• $\eta(\lambda) = \log \lambda$
• $T(x) = x$
• $A(\lambda) = \lambda$
• $h(x) = \frac{1}{x!}$

✅ Poisson is in the exponential family.

Example 3: Normal Distribution (with known variance)

Consider the normal distribution with known variance $\sigma^2$:

Step 1: Expand and rearrange

Step 2: Identify components

• $\eta(\mu) = \frac{\mu}{\sigma^2}$
• $T(x) = x$
• $A(\mu) = \frac{\mu^2}{2\sigma^2}$
• $h(x) = \exp\left( -\frac{x^2}{2\sigma^2} \right) / \sqrt{2\pi \sigma^2}$

✅ Normal (with fixed variance) is also in the exponential family.

The log-partition function $A(\theta)$

This function ensures that the probability integrates (or sums) to 1. Formally:

It also plays a key role in computing expectations and variances in exponential families.

Visual Interactive Demo

Use the demo below to explore how the shape of distributions in the exponential family changes with the natural parameter $\eta$. You can also experiment with different sufficient statistics and see how the log-partition function affects normalization.

Summary

The exponential family provides a unified framework for many common distributions. Understanding how different distributions fit this structure helps in both frequentist and Bayesian inference.