Expectation, Variance & Moments

Let $X$ be a random variable.

Discrete case. If $X$ takes values in a countable set $\{x_1, x_2, \ldots\}$ with PMF $p_X$, the expectation of $X$ is

$E[X] = \sum_{i} x_i \, p_X(x_i)$

provided $\sum_i |x_i| \, p_X(x_i) < \infty$ (absolute convergence).

Continuous case. If $X$ has PDF $f_X$, the expectation of $X$ is

$E[X] = \int_{-\infty}^{\infty} x \, f_X(x) \, dx$

provided $\int_{-\infty}^{\infty} |x| \, f_X(x) \, dx < \infty$ (absolute integrability).

The expectation is also written $\mu$, $\mu_X$, or $\mathbb{E}[X]$.

The absolute convergence condition $E[|X|] < \infty$ is not a technicality — without it, the expectation can depend on the order of summation or the way we partition the integral. The Cauchy distribution with PDF $f(x) = \frac{1}{\pi(1 + x^2)}$ is the standard example: $\int_{-\infty}^{\infty} |x| f(x) \, dx = \infty$, so $E[X]$ does not exist. If you compute the Cauchy principal value $\lim_{R \to \infty} \int_{-R}^{R} x f(x) \, dx$, you get 0 — but that’s a cancellation artifact, not a genuine expectation. The absolute convergence condition from formalCalculus: Sequences & Limits ensures the expectation is well-defined regardless of ordering.

Let $X$ be a random variable and $g : \mathbb{R} \to \mathbb{R}$ a function.

Discrete case: $\displaystyle E[g(X)] = \sum_{x} g(x) \, p_X(x)$

Continuous case: $\displaystyle E[g(X)] = \int_{-\infty}^{\infty} g(x) \, f_X(x) \, dx$

provided the sum/integral converges absolutely.

Roll a fair die. $X \in \{1,2,3,4,5,6\}$ with $p_X(k) = 1/6$ for each $k$.

$E[X] = \sum_{k=1}^{6} k \cdot \frac{1}{6} = \frac{1}{6}(1 + 2 + 3 + 4 + 5 + 6) = \frac{21}{6} = 3.5$

Notice that $E[X] = 3.5$ is not a value $X$ can actually take — this is normal. The expectation is the center of mass, not a mode or a median.

Using LOTUS, $E[X^2] = \sum_{k=1}^{6} k^2 \cdot \frac{1}{6} = \frac{1}{6}(1 + 4 + 9 + 16 + 25 + 36) = \frac{91}{6} \approx 15.17$.

Let $X \sim \text{Exp}(\lambda)$ with PDF $f_X(x) = \lambda e^{-\lambda x}$ for $x \geq 0$.

$E[X] = \int_0^{\infty} x \cdot \lambda e^{-\lambda x} \, dx$

Using integration by parts (with $u = x$, $dv = \lambda e^{-\lambda x} dx$) from formalCalculus: Integration by Parts :

$= \left[-x e^{-\lambda x}\right]_0^{\infty} + \int_0^{\infty} e^{-\lambda x} \, dx = 0 + \frac{1}{\lambda} = \frac{1}{\lambda}$

An Exponential($\lambda$) random variable has mean $1/\lambda$. If a server processes requests at rate $\lambda = 5$ per second, the mean inter-arrival time is $1/5 = 0.2$ seconds.

For any random variables $X$ and $Y$ (with finite expectations) and constants $a, b \in \mathbb{R}$:

$E[aX + bY] = a \, E[X] + b \, E[Y]$

This is worth emphasizing: $E[X + Y] = E[X] + E[Y]$ always, even when $X$ and $Y$ are dependent. The proof uses only marginalization, not factorization of the joint. This makes linearity enormously useful — we can compute $E[\text{sum}]$ as a sum of expectations even when the summands are tangled together in complex ways. The classic application: expected number of fixed points in a random permutation (Example 3 below).

If $X \leq Y$ almost surely (i.e., $P(X \leq Y) = 1$), then $E[X] \leq E[Y]$.

For any constant $c \in \mathbb{R}$: $E[c] = c$.

If $X$ and $Y$ are independent random variables with finite expectations, then

$E[XY] = E[X] \cdot E[Y]$

The converse is false. If $X \sim \text{Uniform}\{-1, 0, 1\}$ and $Y = X^2$, then $E[XY] = E[X^3] = 0 = E[X] \cdot E[Y]$, but $X$ and $Y$ are clearly dependent ($Y$ is a deterministic function of $X$). The condition $E[XY] = E[X]E[Y]$ is called uncorrelatedness — it is strictly weaker than independence.

Randomly shuffle $n$ cards labeled $1, \ldots, n$. A match (or fixed point) occurs at position $i$ if card $i$ lands in position $i$. Let $M = \sum_{i=1}^{n} X_i$ where $X_i = \mathbf{1}\{\text{card } i \text{ is in position } i\}$.

The $X_i$‘s are dependent (if card 1 is in position 1, the remaining cards are shuffled among $n - 1$ positions, changing the probabilities for $X_2, \ldots, X_n$). But linearity doesn’t care:

$E[M] = \sum_{i=1}^{n} E[X_i] = \sum_{i=1}^{n} P(\text{card } i \text{ in position } i) = \sum_{i=1}^{n} \frac{1}{n} = 1$

The expected number of matches is exactly 1, regardless of $n$. This surprising result — the same whether you shuffle 10 cards or 10 million — follows effortlessly from linearity.

The variance of a random variable $X$ with mean $\mu = E[X]$ is

$\text{Var}(X) = E[(X - \mu)^2]$

The standard deviation is $\sigma_X = \sqrt{\text{Var}(X)}$.

Variance is also written $\sigma^2$, $\sigma_X^2$, or $\text{Var}(X)$.

$\text{Var}(X) = E[X^2] - (E[X])^2$

1. $\text{Var}(X) \geq 0$, with equality iff $X$ is constant a.s.
2. $\text{Var}(aX + b) = a^2 \, \text{Var}(X)$ for constants $a, b$
3. If $X \perp Y$, then $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$

Property 3 requires independence. In general, $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X,Y)$. If $X$ and $Y$ are positively correlated ($\text{Cov}(X,Y) > 0$), the variance of their sum is larger than the sum of variances. If negatively correlated, it’s smaller. This is the mathematical foundation of portfolio diversification: combining negatively correlated assets reduces total variance.

For a fair die with $E[X] = 3.5$ and $E[X^2] = 91/6 \approx 15.17$:

$\text{Var}(X) = E[X^2] - (E[X])^2 = \frac{91}{6} - \left(\frac{7}{2}\right)^2 = \frac{91}{6} - \frac{49}{4} = \frac{182 - 147}{12} = \frac{35}{12} \approx 2.92$

Standard deviation: $\sigma = \sqrt{35/12} \approx 1.71$.

Variant A: win 5 dollars with probability 0.4, else 0. $E[A] = 2$, $\text{Var}(A) = 0.4 \cdot 25 - 4 = 6$.

Variant B: win 20 dollars with probability 0.1, else 0. $E[B] = 2$, $\text{Var}(B) = 0.1 \cdot 400 - 4 = 36$.

Both have the same mean payout (2 dollars), but Variant B is 6x more variable. In an A/B test, you’d need far more samples to detect a treatment effect in B than in A — because the noise-to-signal ratio is much higher. This is why variance matters for experimental design.

The covariance of random variables $X$ and $Y$ is

$\text{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)]$

where $\mu_X = E[X]$ and $\mu_Y = E[Y]$.

$\text{Cov}(X, Y) = E[XY] - E[X] \cdot E[Y]$

1. $\text{Cov}(X, Y) = \text{Cov}(Y, X)$ (symmetry)
2. $\text{Cov}(aX + b, cY + d) = ac \, \text{Cov}(X, Y)$ (bilinearity with constants)
3. $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y)$ (general variance of sum)
4. $\text{Cov}\bigl(\sum_i X_i, \sum_j Y_j\bigr) = \sum_i \sum_j \text{Cov}(X_i, Y_j)$ (multilinearity)

The Pearson correlation coefficient of $X$ and $Y$ is

$\rho(X, Y) = \frac{\text{Cov}(X, Y)}{\sigma_X \, \sigma_Y} = \frac{\text{Cov}(X, Y)}{\sqrt{\text{Var}(X) \, \text{Var}(Y)}}$

provided both variances are positive.

$-1 \leq \rho(X, Y) \leq 1$

with $|\rho(X, Y)| = 1$ if and only if $Y = aX + b$ for some constants $a, b$ (i.e., $X$ and $Y$ are related by a perfect linear function).

Independence $\implies$ $\text{Cov}(X,Y) = 0$ $\implies$ $\rho(X,Y) = 0$ (Theorem 5). But the converse fails: uncorrelatedness ($\rho = 0$) does not imply independence. The example from Remark 3 ($X$ uniform on $\{-1, 0, 1\}$, $Y = X^2$) has $\rho = 0$ but complete functional dependence. Correlation measures linear association only — it can miss nonlinear dependencies entirely. This distinction matters in ML: two features can be uncorrelated yet carry highly redundant information through nonlinear relationships.

If $X \geq 0$ a.s. and $a > 0$, then

$P(X \geq a) \leq \frac{E[X]}{a}$

For any random variable $X$ with $E[X] = \mu$ and $\text{Var}(X) = \sigma^2 < \infty$:

$P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}$

for any $k > 0$. Equivalently, $P(|X - \mu| \geq \varepsilon) \leq \text{Var}(X)/\varepsilon^2$.

A quality control process produces items with mean weight $\mu = 100$g and standard deviation $\sigma = 2$g. What fraction of items can weigh more than 106g?

Using Chebyshev with $k = 3$ (since $|106 - 100| = 6 = 3\sigma$):

$P(|X - 100| \geq 6) \leq \frac{1}{3^2} = \frac{1}{9} \approx 11.1\%$

If we know the weights are normally distributed, the true probability is $P(|Z| \geq 3) \approx 0.27\%$ — 40x smaller. Chebyshev’s power is that it works regardless of the distribution shape.

If $g$ is a convex function and $E[X]$ exists, then

$g(E[X]) \leq E[g(X)]$

If $g$ is concave, the inequality reverses: $g(E[X]) \geq E[g(X)]$.

Let $g(x) = -\log(x)$ (convex on $(0, \infty)$). Jensen gives:

$-\log(E[X]) \leq E[-\log(X)] = -E[\log(X)]$

So $\log(E[X]) \geq E[\log(X)]$, or equivalently $E[X] \geq e^{E[\log X]}$. For $n$ equal-probability values $x_1, \ldots, x_n$:

$\frac{x_1 + \cdots + x_n}{n} \geq (x_1 \cdots x_n)^{1/n}$

This is the arithmetic-mean ≥ geometric-mean inequality — a pure consequence of Jensen.

ML application: Jensen’s inequality with $g(x) = -\log(x)$ is exactly what gives us the evidence lower bound (ELBO) in variational inference: $\log p(\mathbf{x}) \geq E_q[\log p(\mathbf{x}, \mathbf{z}) - \log q(\mathbf{z})]$. See formalML: Information Theory for the full derivation.

If $B$ is an event with $P(B) > 0$, the conditional expectation of $X$ given $B$ is

$E[X \mid B] = \sum_x x \, P(X = x \mid B) \quad \text{(discrete)}$

$E[X \mid B] = \int_{-\infty}^{\infty} x \, f_{X|B}(x) \, dx \quad \text{(continuous)}$

This is just the ordinary expectation computed using the conditional distribution.

For discrete $X$ and $Y$ with joint PMF $p_{X,Y}$:

$E[X \mid Y = y] = \sum_x x \, p_{X|Y}(x \mid y) = \sum_x x \, \frac{p_{X,Y}(x, y)}{p_Y(y)}$

For continuous $X$ and $Y$ with joint PDF $f_{X,Y}$:

$E[X \mid Y = y] = \int_{-\infty}^{\infty} x \, f_{X|Y}(x \mid y) \, dx = \int_{-\infty}^{\infty} x \, \frac{f_{X,Y}(x, y)}{f_Y(y)} \, dx$

Here $E[X \mid Y = y]$ is a function of $y$ — we write it as $h(y) = E[X \mid Y = y]$.

The conditional expectation $E[X \mid Y]$ is the random variable obtained by evaluating the function $h(y) = E[X \mid Y = y]$ at $Y$:

$E[X \mid Y] = h(Y)$

This is a random variable — it inherits its randomness from $Y$. Different realizations of $Y$ produce different “best guesses” of $X$.

Let $(X, Y)$ be bivariate normal with means $\mu_X, \mu_Y$, standard deviations $\sigma_X, \sigma_Y$, and correlation $\rho$. From Topic 3, §8:

$E[X \mid Y = y] = \mu_X + \rho \frac{\sigma_X}{\sigma_Y}(y - \mu_Y)$

This is a linear function of $y$ — it’s the regression line. The slope is $\rho \sigma_X / \sigma_Y$, and when $\rho = 0$, the conditional mean equals the unconditional mean $\mu_X$ (knowing $Y$ provides no information about $X$).

The conditional variance is $\text{Var}(X \mid Y = y) = \sigma_X^2(1 - \rho^2)$, which does not depend on $y$. This homoscedasticity is special to the bivariate normal.

$E[X] = E[E[X \mid Y]]$

More precisely, if $Y$ is discrete with values $\{y_1, y_2, \ldots\}$:

$E[X] = \sum_j E[X \mid Y = y_j] \, P(Y = y_j)$

$\text{Var}(X) = E[\text{Var}(X \mid Y)] + \text{Var}(E[X \mid Y])$

In words: total variance = expected within-group variance + between-group variance.

A company has two customer segments: Casual (60%) with mean spending of 50, and Power Users (40%) with mean spending of 120. Let $Y$ indicate the segment.

$E[\text{Spending}] = E[\text{Spending} \mid \text{Casual}] \cdot P(\text{Casual}) + E[\text{Spending} \mid \text{Power}] \cdot P(\text{Power})$

$= 50 \cdot 0.6 + 120 \cdot 0.4 = 30 + 48 = 78$

The unconditional mean is a weighted average of the conditional means.

Continuing Example 9, suppose $\text{Var}(\text{Spending} \mid \text{Casual}) = 400$ (standard deviation 20) and $\text{Var}(\text{Spending} \mid \text{Power}) = 2500$ (standard deviation 50).

Within-group variance (expected conditional variance):

$E[\text{Var}(\text{Spending} \mid Y)] = 400 \cdot 0.6 + 2500 \cdot 0.4 = 240 + 1000 = 1240$

Between-group variance (variance of conditional means): The conditional means are 50 and 120, with weights 0.6 and 0.4. Their mean is 78 (from Example 9).

$\text{Var}(E[\text{Spending} \mid Y]) = (50 - 78)^2 \cdot 0.6 + (120 - 78)^2 \cdot 0.4 = 784 \cdot 0.6 + 1764 \cdot 0.4 = 470.4 + 705.6 = 1176$

Total variance: $1240 + 1176 = 2416$.

About half the variance comes from within segments (customers vary within their segment) and half from between segments (the segments have different means).

The moment-generating function (MGF) of a random variable $X$ is

$M_X(t) = E[e^{tX}]$

defined for all $t \in \mathbb{R}$ where the expectation exists. Explicitly:

$M_X(t) = \sum_x e^{tx} p_X(x) \quad \text{(discrete)}$

$M_X(t) = \int_{-\infty}^{\infty} e^{tx} f_X(x) \, dx \quad \text{(continuous)}$

If $M_X(t)$ exists in an open interval around $t = 0$, then

$M_X^{(n)}(0) = E[X^n]$

where $M_X^{(n)}$ denotes the $n$th derivative.

If $M_X(t) = M_Y(t)$ for all $t$ in some open interval $(-\delta, \delta)$ around 0, then $X$ and $Y$ have the same distribution.

If $X$ and $Y$ are independent, then

$M_{X+Y}(t) = M_X(t) \cdot M_Y(t)$