4.5 Expected Value, Variance & Covariance

Expected value, variance and covariance are indispensable in estimating returns and risks of any investments. Naturally, they are a popular test subject in interviews as well. The basic knowledge includes the following:

If $E[x_i]$ is finite for all $i=1,\cdots,n$ , then $E[X_1+\cdots+X_n]=E[X_1]+\cdots+E[X_n]$ . The relationship holds whether the $x_i$ ’s are independent of each other or not.

If $X$ and $Y$ are independent, then $E[g(X)h(Y)]=E[g(x)]E[h(Y)]$ .

Covariance: $\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])]=E[XY]-E[X]E[Y]$ .

Correlation: $\rho(X,Y)=\dfrac{\operatorname{Cov}(X,Y)}{\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}}$

If $X$ and $Y$ are independent, $\operatorname{Cov}(X,Y)=0$ and $\rho(X,Y)=0$ .²⁶

General rules of variance and covariance:

\operatorname{Cov}\left(\sum_{i=1}^{n}a_iX_i,\sum_{j=1}^{m}b_jY_j\right)=\sum_{i=1}^{n}\sum_{j=1}^{m}a_ib_j\operatorname{Cov}(X_i,Y_j)

\operatorname{Var}\left(\sum_{i=1}^{n}X_i\right)=\sum_{i=1}^{n}\operatorname{Var}(X_i)+2\sum\sum_{i<j}\operatorname{Cov}(X_i,X_j)

Conditional expectation and variance

For discrete distribution:

E[g(X)\mid Y=y]=\sum_x g(x)p_{X\mid Y}(x\mid y)=\sum_x g(x)p(X=x\mid Y=y)

For continuous distribution:

E[g(X)\mid Y=y]=\int_{-\infty}^{\infty}g(x)f_{X\mid Y}(x\mid y)\,dx

Law of total expectation:

E[X]=E[E[X\mid Y]]=\begin{cases} \displaystyle\sum_y E[X\mid Y=y]p(Y=y),&\text{for discrete }Y\\[6pt] \displaystyle\int_{-\infty}^{\infty}E[X\mid Y=y]f_Y(y)\,dy,&\text{for continuous }Y \end{cases}

Connecting noodles

You have 100 noodles in your soup bowl. Being blindfolded, you are told to take two ends of some noodles (each end on any noodle has the same probability of being chosen) in your bowl and connect them. You continue until there are no free ends. The number of loops formed by the noodles this way is stochastic. Calculate the expected number of circles.

Solution: Again do not be frightened by the large number 100. If you have no clue how to start, let’s begin with the simplest case where $n=1$ . Surely you have only one choice (to connect both ends of the noodle), so $E[f(1)]=1$ . How about 2 noodles? Now you have 4 ends ( $2\times2$ ) and you can connect any two of them. There are $\binom{4}{2}=\dfrac{4\times3}{2}=6$ combinations. Among them, 2 combinations will connect both ends of the same noodle together and yield 1 circle and 1 noodle. The other 4 choices will yield a single noodle. So the expected number of circles is

E[f(2)]=2/6\times(1+E[f(1)])+4/6\times E[f(1)]=1/3+E[f(1)]=1/3+1.

We now move on to 3 noodles with $\binom{6}{2}=\dfrac{6\times5}{2}=15$ choices. Among them, 3 choices will yield 1 circle and 2 noodles; the other 12 choices will yield 2 noodles only, so

E[f(3)]=3/15\times(1+E[f(2)])+12/15\times E[f(2)]=1/5+E[f(2)]=1/5+1/3+1.

See the pattern? For any $n$ noodles, we will have $E[f(n)]=1+1/3+1/5+\cdots+1/(2n-1)$ , which can be easily proved by induction. Plug 100 in, we will have the answer.

Actually after the 2-noodle case, you probably have found the key to this question. If you start with $n$ noodles, among $\binom{2n}{2}=n(2n-1)$ possible combinations, we have $\dfrac{n}{n(2n-1)}=\dfrac{1}{2n-1}$ probability to yield 1 circle and $n-1$ noodles and $\dfrac{2n-2}{2n-1}$ probability to yield $n-1$ noodles only, so $E[f(n)]=E[f(n-1)]+\dfrac{1}{2n-1}$ . Working backward, you can get the final solution as well.

Optimal hedge ratio

You just bought one share of stock $A$ and want to hedge it by shorting stock $B$ . How many shares of $B$ should you short to minimize the variance of the hedged position? Assume that the variance of stock $A$ ’s return is $\sigma_A^2$ ; the variance of $B$ ’s return is $\sigma_B^2$ ; their correlation coefficient is $\rho$ .

Solution: Suppose that we short $h$ shares of B, the variance of the portfolio return is

\operatorname{var}(r_A-hr_B)=\sigma_A^2-2\rho h\sigma_A\sigma_B+h^2\sigma_B^2

The best hedge ratio should minimize $\operatorname{var}(r_A-hr_B)$ . Take the first order partial derivative with respect to $h$ and set it to zero:

\frac{\partial\operatorname{var}}{\partial h}=-2\rho\sigma_A\sigma_B+2h\sigma_B^2=0\Rightarrow h=\rho\frac{\sigma_A}{\sigma_B}.

To confirm it’s the minimum, we can also check the second-order partial derivative: $\dfrac{\partial^2\operatorname{var}}{\partial h^2}=2\sigma_B^2>0$ . So Indeed when $h=\rho\dfrac{\sigma_A}{\sigma_B}$ , the hedge portfolio has the minimum variance.

Dice game

Suppose that you roll a dice. For each roll, you are paid the face value. If a roll gives 4, 5 or 6, you can roll the dice again. Once you get 1, 2 or 3, the game stops. What is the expected payoff of this game?

Solution: This is an example of the law of total expectation. Clearly your payoff will be different depending on the outcome of first roll. Let $E[X]$ be your expected payoff and $Y$ be the outcome of your first throw. You have 1/2 chance to get $Y\in\{1,2,3\}$ , in which case the expected value is the expected face value 2, so $E[X\mid Y\in\{1,2,3\}]=2$ ; you have

1/2 chance to get $Y\in\{4,5,6\}$ , in which case you get expected face value 5 and extra throw(s). The extra throw(s) essentially means you start the game again and have an extra expected value $E[X]$ . So we have $E[X\mid Y\in(4,5,6)]=5+E[X]$ . Apply the law of total expectation, we have

E[X]=E[E[X\mid Y]]=\tfrac12\times2+\tfrac12\times(5+E[X])\Rightarrow E[X]=7.

²⁷

Card game

What is the expected number of cards that need to be turned over in a regular 52-card deck in order to see the first ace?

Solution: There are 4 aces and 48 other cards. Let’s label them as card $1,2,\cdots,48$ . Let

X_i=\begin{cases}1,&\text{if card }i\text{ is turned over before 4 aces}\\0,&\text{otherwise}\end{cases}

The total number of cards that need to be turned over in order to see the first ace is $X=1+\sum_{i=1}^{48}X_i$ , so we have $E[X]=1+\sum_{i=1}^{48}E[X_i]$ . As shown in the following sequence, each card $i$ is equally likely to be in one of the five regions separated by 4 aces:

1\quad A\quad 2\quad A\quad 3\quad A\quad 4\quad A\quad 5

So the probability that card $i$ appears before all 4 aces is 1/5, and we have $E[X_i]=1/5$ .

Therefore, $E[X]=1+\sum_{i=1}^{48}E[X_i]=1+48/5=10.6$ .

This is just a special case for random ordering of $m$ ordinary cards and $n$ special cards. The expected position of the first special card is $1+\sum_{i=1}^{m}E[X_i]=1+\dfrac{m}{n+1}$ .

Sum of random variables

Assume that $X_1,X_2,\cdots,$ and $X_n$ are independent and identically-distributed (IID) random variables with uniform distribution between 0 and 1. What is the probability that $S_n=X_1+X_2+\cdots+X_n\le1$ ?²⁸

Solution: This problem is a rather difficult one. The general principle to start with the simplest cases and try to find a pattern will again help you approach the problem; even though it may not give you the final answer. When $n=1$ , $P(S_1\le1)$ is 1. As shown in Figure 4.6, when $n=2$ , the probability that $X_1+X_2\le1$ is just the area under $X_1+X_2\le1$ within the square with side length 1 (a triangle). So $P(S_2\le1)=1/2$ . When $n=3$ , the probability becomes the tetrahedron ABCD under the plane $X_1+X_2+X_3\le1$ within the cube with side length 1. The volume of tetrahedron ABCD is 1/6.²⁹ So $P(S_3\le1)=1/6$ . Now we can guess that the solution is $1/n!$ . To prove it, let’s again resort to induction. Assume $P(S_n\le1)=1/n!$ . We need to prove that $P(S_{n+1}\le1)=1/(n+1)!$ .

n = 2: unit square with axes X1 and X2, endpoint labels 0 and 1, and shaded triangular region below X1 + X2 = 1. — Figure 4.6 Probability that $S_n\le1$ when $n=2$ or $n=3$ .

n = 3: unit cube with axes X1, X2, X3; vertices A, B, C and origin 0; the tetrahedral region under X1 + X2 + X3 = 1 is shaded, with hidden edges dashed. — Figure 4.6 Probability that $S_n\le1$ when $n=2$ or $n=3$ .

Here we can use probability by conditioning. Condition on the value of $X_{n+1}$ , we have

P(S_{n+1}\le1)=\int_0^1 f(X_{n+1})P(S_n\le1-X_{n+1})\,dX_{n+1},

where $f(X_{n+1})$ is the probability density function of $X_{n+1}$ , so $f(X_{n+1})=1$ . But how do we calculate $P(S_n\le1-X_{n+1})$ ? The cases of $n=2$ and $n=3$ have provided us with some clue. For $S_n\le1-X_{n+1}$ instead of $S_n\le1$ , we essentially need to shrink every dimension of the $n$ -dimensional simplex³⁰ from 1 to

$1-X_{n+1}$ . So its volume should be $\dfrac{(1-X_{n+1})^n}{n!}$ instead of $\dfrac{1}{n!}$ . Plugging in these results, we have

P(S_{n+1}\le1)=\int_0^1\frac{(1-X_{n+1})^n}{n!}\,dX_{n+1}=\frac{1}{n!}\left[-\frac{(1-X_{n+1})^{n+1}}{n+1}\right]_0^1=\frac{1}{n!}\times\frac{1}{n+1}=\frac{1}{(n+1)!}.

So the general result is true for $n+1$ as well and we have $P(S_n\le1)=1/n!$ .

Coupon collection

There are $N$ distinct types of coupons in cereal boxes and each type, independent of prior selections, is equally likely to be in a box.

If a child wants to collect a complete set of coupons with at least one of each type, how many coupons (boxes) on average are needed to make such a complete set?
If the child has collected $n$ coupons, what is the expected number of distinct coupon types?³¹

Solution: For part $A$ , let $X_i,\ i=1,2,\cdots,N$ , be the number of additional coupons needed to obtain the $i$ -th type after $(i-1)$ distinct types have been collected. So the total number of coupons needed is $X=X_1+X_2+\cdots+X_N=\sum_{i=1}^{N}X_i$ .

For any $i$ , $i-1$ distinct types of coupons have already been collected. It follows that a new coupon will be of a different type with probability $1-(i-1)/N=(N-i+1)/N$ . Essentially to obtain the $i$ -th distinct type, the random variable $X_i$ follows a geometric distribution with $p=(N-i+1)/N$ and $E[X_i]=N/(N-i+1)$ . For example, if $i=1$ , we simply have $X_i=E[X_i]=1$ .

\therefore\ E[X]=\sum_{i=1}^{N}E[X_i]=\sum_{i=1}^{N}\frac{N}{N-i+1}=N\left(\frac1N+\frac{1}{N-1}+\cdots+\frac11\right).

For part $B$ , let $Y$ be the number of distinct types of coupons in the set of $n$ coupons. We introduce indicator random variables $I_i,\ i=1,2,\cdots,N$ , where

\begin{cases}I_i=1,&\text{if at least one coupon of the }i\text{-th type is in the set of }n\text{ coupons}\\ I_i=0,&\text{otherwise}\end{cases}

So we have $Y=I_1+I_2+\cdots+I_N=\sum_{i=1}^{N}I_i$

For each collected coupon, the probability that it is not the $i$ -th coupon type is $\dfrac{N-1}{N}$ . Since all $n$ coupons are independent, the probability that none of the $n$ coupons is the $i$ -th coupon type is $P(I_i=0)=\left(\dfrac{N-1}{N}\right)^n$ and we have $E[I_i]=P(I_i=1)=1-\left(\dfrac{N-1}{N}\right)^n$ .

\therefore\ E[Y]=\sum_{i=1}^{N}E[I_i]=N-N\left(\frac{N-1}{N}\right)^n.

³²

Joint default probability

If there is a 50% probability that bond $A$ will default next year and a 30% probability that bond $B$ will default. What is the range of probability that at least one bond defaults and what is the range of their correlation?

Solution: The range of probability that at least one bond defaults is easy to find. To have the largest probability, we can assume whenever $A$ defaults, $B$ does not default; whenever $B$ defaults, $A$ does not default. So the maximum probability that at least one bond defaults is $50\%+30\%=80\%$ . (The result only applies if $P(A)+P(B)\le1$ ). For the minimum, we can assume whenever $A$ defaults, $B$ also defaults. So the minimum probability that at least one bond defaults is 50%.

To calculate the corresponding correlation, let $I_A$ and $I_B$ be the indicator for the event that ‘bond A/B defaults next year’ and $\rho_{AB}$ be their correlation. Then we have $E[I_A]=0.5,\ E[I_B]=0.3,\ \operatorname{var}(I_A)=p_A\times(1-p_A)=0.25,\ \operatorname{var}(I_B)=0.21$ .

\begin{aligned} P(A\text{ or }B\text{ defaults})&=E[I_A]+E[I_B]-E[I_AI_B]\\ &=E[I_A]+E[I_B]-\bigl(E[I_A]E[I_B]-\operatorname{cov}(I_A,I_B)\bigr)\\ &=0.5+0.3-(0.5\times0.3-\rho_{AB}\sigma_A\sigma_B)\\ &=0.65-\sqrt{0.21}/2\,\rho_{AB} \end{aligned}

For the maximum probability, we have $0.65-\sqrt{0.21}/2\,\rho_{AB}=0.8\Rightarrow\rho_{AB}=-\sqrt{3/7}$ .

For the minimum probability, we have $0.65-\sqrt{0.21}/2\,\rho_{AB}=0.5\Rightarrow\rho_{AB}=\sqrt{3/7}$ .

In this problem, do not start with $P(A\text{ or }B\text{ defaults})=0.65-\sqrt{0.21}/2\,\rho_{AB}$ and try to set $\rho_{AB}=\pm1$ to calculate the maximum and minimum probability since the correlation cannot be $\pm1$ . The range of correlation is restricted to $\left[-\sqrt{3/7},\sqrt{3/7}\right]$ .