4.3 Conditional Probability and Bayes’ formula

Many financial transactions are responses to probability adjustments based on new—and most likely incomplete—information. Conditional probability surely is one of the most popular test subjects in quantitative interviews. So in this section, we focus on basic conditional probability definitions and theorems.

Conditional probability $P(A\mid B)$: If $P(B)>0$, then $P(A\mid B)=\dfrac{P(AB)}{P(B)}$ is the fraction of $B$ outcomes that are also $A$ outcomes.

Multiplication Rule: $P(E_1E_2\cdots E_n)=P(E_1)P(E_2\mid E_1)P(E_3\mid E_1E_2)\cdots P(E_n\mid E_1\cdots E_{n-1})$.

Law of total probability: for any mutually exclusive events $\{F_i\},\ i=1,2,\cdots,n$, whose union is the entire sample space ($F_i\cap F_j=\Phi,\ \forall i\ne j;\ \bigcup_{i=1}^{n}F_i=\Omega$), we have

Independent events: $P(EF)=P(E)P(F)\Rightarrow P(EF^C)=P(E)P(F^C)$.

Independence is a symmetric relation: X is independent of Y $\Leftrightarrow$ Y is independent of X.

Bayes’ Formula: $P(F_j\mid E)=\dfrac{P(E\mid F_j)P(F_j)}{\sum_{i=1}^{n}P(E\mid F_i)P(F_i)}$ if $F_i,\ i=1,\cdots,n$, are mutually exclusive events whose union is the entire sample space.

As the following examples will demonstrate, not all conditional probability problems have intuitive solutions. Many demand logical analysis instead.

Boys and girls

Part A. A company is holding a dinner for working mothers with at least one son. Ms. Jackson, a mother with two children, is invited. What is the probability that both children are boys?

Solution: The sample space of two children is given by $\Omega=\{(b,b),(b,g),(g,b),(g,g)\}$ (e.g., $(g,b)$ means the older child is a girl and the younger child a boy), and each outcome has the same probability. Since Ms. Jackson is invited, she has at least one son. Let $B$ be the event that at least one of the children is a boy and $A$ be the event that both children are boys, we have

Part B. Your new colleague, Ms. Parker is known to have two children. If you see her walking with one of her children and that child is a boy, what is the probability that both children are boys?

Solution: the other child is equally likely to be a boy or a girl (independent of the boy you’ve seen), so the probability that both children are boys is 1/2.

Notice the subtle difference between part A and part B. In part A, the problem essentially asks given there is at least one boy in two children, what is the conditional probability that both children are boys. Part B asks that given one child is a boy, what is the conditional probability that the other child is also a boy. For both parts, we need to assume that each child is equal likely to be a boy or a girl.

All-girl world?

In a primitive society, every couple prefers to have a baby girl. There is a 50% chance that each child they have is a girl, and the genders of their children are mutually independent. If each couple insists on having more children until they get a girl and once they have a girl they will stop having more children, what will eventually happen to the fraction of girls in this society?

Solution: It was surprising that many interviewees—include many who studied probability—have the misconception that there will be more girls. Do not let the word “prefer” and a wrong intuition misguide you. The fraction of baby girls are driven by nature, or at least the X and Y chromosomes, not by the couples’ preference. You only need to look at the key information: 50% and independence. Every new-born child has equal probability of being a boy or a girl regardless of the gender of any other children. So the fraction of girls born is always 50% and the fractions of girls in the society will stay stable at 50%.

Unfair coin

You are given 1000 coins. Among them, 1 coin has heads on both sides. The other 999 coins are fair coins. You randomly choose a coin and toss it 10 times. Each time, the coin turns up heads. What is the probability that the coin you choose is the unfair one?

Solution: This is a classic conditional probability question that uses Bayes’ theorem. Let $A$ be the event that the chosen coin is the unfair one, then $A^c$ is the event that the chosen coin is a fair one. Let $B$ be the event that all ten tosses turn up heads. Apply Bayes’ theorem we have $P(A\mid B)=\dfrac{P(B\mid A)P(A)}{P(B)}=\dfrac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid A^c)P(A^c)}$.

The priors are $P(A)=1/1000$ and $P(A^c)=999/1000$. If the coin is unfair, it always turns up heads, so $P(B\mid A)=1$. If the coin is fair, each time it has 1/2 probability turning 75 up heads. So $P(B\mid A^c)=(1/2)^{10}=1/1024$. Plug in all the available information and we have the answer:

Fair probability from an unfair coin

If you have an unfair coin, which may bias toward either heads or tails at an unknown probability, can you generate even odds using this coin?

Solution: Unlike fair coins, we clearly can not generate even odds with one toss using an unfair coin. How about using 2 tosses? Let $p_H$ be the probability the coin will yield head, and $p_T$ be the probability the coin will yield tails ($p_H+p_T=1$). Consider two independent tosses. We have four possible outcomes $HH,\ HT,\ TH$ and $TT$ with probabilities $P(HH)=p_Hp_H,\ P(HT)=p_Hp_T,\ P(TH)=p_Tp_H,$ and $P(TT)=p_Tp_T$. So we have $P(HT)=P(TH)$. By assigning $HT$ to winning and $TH$ to losing, we can generate even odds.¹²

Dart game

Jason throws two darts at a dartboard, aiming for the center. The second dart lands farther from the center than the first. If Jason throws a third dart aiming for the center, what is the probability that the third throw is farther from the center than the first? Assume Jason's skillfulness is constant.

Solution: A standard answer directly applies the conditional probability by enumerating all possible outcomes. If we rank the three darts’ results from the best (A) to the worst (C), there are 6 possible outcomes with equal probability:

The information from the first two throws eliminates outcomes 2, 4 and 6. Conditioned on outcomes 1, 3, and 5, the outcomes that the 3rd throw is worse than the 1st throw are outcomes 1 and 3. So there is 2/3 probability that the third throw is farther from the center than the first.

This approach surely is reasonable. Nevertheless, it is not an efficient approach. When the number of darts is small, we can easily enumerate all outcomes. What if it is a more complex version of the original problem:

Jason throws $n$ ($n\ge5$) darts at a dartboard, aiming for the center. Each subsequent dart is farther from the center than the first dart. If Jason throws the $(n+1)$th dart, what is the probability that it is also farther from the center than his first?

This question is equivalent to a simple question: what is the probability that the $(n+1)$th throw is not the best among all $(n+1)$ throws? Since the 1st throw is the best among the first $n$ throws, essentially I am saying the event that the $(n+1)$th throw is the best of all $(n+1)$ throws (let’s call it $A_{n+1}$) is independent of the event that the 1st throw is the best of the first $n$ throws (let’s call it $A_1$). In fact, $A_{n+1}$ is independent of the order of the first $n$ throws. Are these two events really independent? The answer is a resounding yes. If it is not obvious to you that $A_{n+1}$ is independent of the order of the first $n$ throws, let’s look at it another way: the order of the first $n$ throws is independent of $A_{n+1}$. Surely this claim is conspicuous. But independence is symmetric! Since the probability of $A_{n+1}$ is $1/(n+1)$, the probability that $(n+1)$th throw is not the best is $n/(n+1)$.¹³

For the original version, three darts are thrown independently, they each have a 1/3 chance of being the best throw. As long as the third dart is not the best throw, it will be worse than the first dart. Therefore the answer is 2/3.

Birthday line

At a movie theater, a whimsical manager announces that she will give a free ticket to the first person in line whose birthday is the same as someone who has already bought a ticket. You are given the opportunity to choose any position in line. Assuming that you 77 don’t know anyone else's birthday and all birthdays are distributed randomly throughout the year (assuming 365 days in a year), what position in line gives you the largest chance of getting the free ticket?¹⁴

Solution: If you have solved the problem that no two people have the same birthday in an $n$-people group, this new problem is just a small extension. Assume that you choose to be the $n$-th person in line. In order for you to get the free ticket, all of the first $n-1$ individuals in line must have different birthdays and your birthday needs to be the same as one of those $n-1$ individuals.

It is intuitive to argue that when $n$ is small, increasing $n$ will increase your chance of getting the free ticket since the increase of $p(\text{yours among those }n-1\text{ birthdays})$ is more significant than the decrease in $p(\text{first }n-1\text{ people have no same birthday})$. So when $n$ is small, we have $P(n)>P(n-1)$. As $n$ increases, gradually the negative impact of $p(\text{first }n-1\text{ people have no same birthday})$ will catch up and at a certain point we will have $P(n+1)<P(n)$. So we need to find such an $n$ that satisfies $P(n)>P(n-1)$ and $P(n)>P(n+1)$.

Hence,

You should be the 20th person in line.

Dice order

We throw 3 dice one by one. What is the probability that we obtain 3 points in strictly increasing order?¹⁵

Solution: To have 3 points in strictly increasing order, first all three points must be different numbers. Conditioned on three different numbers, the probability of strictly increasing order is simply $1/3!=1/6$ (one specific sequence out of all possible permutations). So we have

Monty Hall problem

Monty Hall problem is a probability puzzle based on an old American show Let’s Make a Deal. The problem is named after the show’s host. Suppose you're on the show now, and you're given the choice of 3 doors. Behind one door is a car; behind the other two, goats. You don’t know ahead of time what is behind each of the doors.

You pick one of the doors and announce it. As soon as you pick the door, Monty opens one of the other two doors that he knows has a goat behind it. Then he gives you the option to either keep your original choice or switch to the third door. Should you switch? What is the probability of winning a car if you switch?

Solution: If you don’t switch, whether you win or not is independent of Monty’s action of showing you a goat, so your probability of winning is 1/3. What if you switch? Many would argue that since there are only two doors left after Monty shows a door with goat, the probability of winning is 1/2. But is this argument correct?

If you look at the problem from a different perspective, the answer becomes clear. Using a switching strategy, you win the car if and only if you originally pick a door with a goat, which has a probability of 2/3 (You pick a door with a goat, Monty shows a door with another goat, so the one you switch to must have a car behind it). If you originally picked the door with the car, which has a probability of 1/3, you will lose by switching. So your probability of winning by switching is actually 2/3.

Amoeba population

There is a one amoeba in a pond. After every minute the amoeba may die, stay the same, split into two or split into three with equal probability. All its offspring, if it has any, will behave the same (and independent of other amoebas). What is the probability the amoeba population will die out?

Solution: This is just another standard conditional probability problem once you realize we need to derive the probability conditioned on what happens to the amoeba one minute later. Let $P(E)$ be the probability that the amoeba population will die out and apply the law of total probability conditioned on what happens to the amoeba one minute later:

For the original amoeba, as stated in the question, there are four possible mutually exclusive events each with probability 1/4. Let’s denote $F_1$ as the event the amoeba dies; $F_2$ as the event that it stays the same; $F_3$ as the event that it splits into two; $F_4$ as the event that it splits into three. For event $F_1,\ P(E\mid F_1)=1$ since no amoeba is left. $P(E\mid F_2)=P(E)$ since the state is the same as the beginning. For $F_3$, there are two amoebas; either behaves the same as the original one. The total amoeba population will die only if both amoebas die out. Since they are independent, the probability that they both will die out is $P(E)^2$. Similarly we have $P(F_4)=P(E)^3$. Plug in all the numbers, the equation becomes $P(E)=1/4\times1+1/4\times P(E)+1/4\times P(E)^2+1/4\times P(E)^3$. Solve this equation with the restriction $0<P(E)<1$, and we will get $P(E)=\sqrt2-1\approx0.414$ (The other two roots of the equation are 1 and $-\sqrt2-1$).

Candies in a jar

You are taking out candies one by one from a jar that has 10 red candies, 20 blue candies, and 30 green candies in it. What is the probability that there are at least 1 blue candy and 1 green candy left in the jar when you have taken out all the red candies?¹⁶

Solution: At first look, this problem appears to be a combinatorial one. However, a conditional probability approach gives a much more intuitive answer. Let $T_r,T_b$ and $T_g$ 80 be the number that the last red, blue, and green candies are taken out respectively. To have at least 1 blue candy and 1 green candy left when all the red candies are taken out, we need to have $T_r<T_b$ and $T_r<T_g$. In other words, we want to derive $P(T_r<T_b\cap T_r<T_g)$. There are two mutually exclusive events that satisfy $T_r<T_b$ and $T_r<T_g:\ T_r<T_b<T_g$ and $T_r<T_g<T_b$.

$T_r<T_b<T_g$ means that the last candy is green ($T_g=60$). Since each of the 60 candies are equally likely to be the last candy and among them 30 are green ones, we have $P(T_g=60)=\dfrac{30}{60}$. Conditioned on $T_g=60$, we need $P(T_r<T_b\mid T_g=60)$. Among the 30 red and blue candies, each candy is again equally likely to be the last candy and there are 20 blue candies, so $P(T_r<T_b\mid T_g=60)=\dfrac{20}{30}$ and $P(T_r<T_b<T_g)=\dfrac{30}{60}\times\dfrac{20}{30}$. Similarly, we have $P(T_r<T_g<T_b)=\dfrac{20}{60}\times\dfrac{30}{40}$.

Hence,

Coin toss game

Two players, $A$ and $B$, alternatively toss a fair coin ($A$ tosses the coin first, then $B$ tosses the coin, then $A$, then $B$…). The sequence of heads and tails is recorded. If there is a head followed by a tail ($HT$ subsequence), the game ends and the person who tosses the tail wins. What is the probability that $A$ wins the game?¹⁷

Solution: Let $P(A)$ be the probability that $A$ wins; then the probability that $B$ wins is $P(B)=1-P(A)$. Let’s condition $P(A)$ on $A$’s first toss, which has 1/2 probability of $H$ (heads) and 1/2 probability of $T$ (tails).

If $A$’s first toss is $T$, then B essentially becomes the first to toss (An $H$ is required for the HT subsequence). So we have $P(A\mid T)=P(B)=1-P(A)$.

If $A$’s first toss ends in $H$, let’s further condition on $B$’s first toss. $B$ has 1/2 probability of getting $T$, in that case $A$ loses. For the 1/2 probability that $B$ gets $H$, $B$ essentially 81 becomes the first one to toss an $H$. In that case, $A$ has $(1-P(A\mid H))$ probability of winning. So $P(A\mid H)=1/2\times0+1/2(1-P(A\mid H))\Rightarrow P(A\mid H)=1/3$

Combining all the available information, we have

Sanity check: we can see that $P(A)<1/2$, which is reasonable since $A$ cannot win in his first toss, yet $B$ has 1/4 probability to win in her first toss.

Russian roulette series

Let’s play a traditional version of Russian roulette. A single bullet is put into a 6-chamber revolver. The barrel is randomly spun so that each chamber is equally likely to be under the hammer. Two players take turns to pull the trigger—with the gun unfortunately pointing at one’s own head—without further spinning until the gun goes off and the person who gets killed loses. If you, one of the players, can choose to go first or second, how will you choose? And what is your probability of loss?

Solution: Many people have the wrong impression that the first person has higher probability of loss. After all, the first player has a 1/6 chance of getting killed in the first round before the second player starts. Unfortunately, this is one of the few times that intuition is wrong. Once the barrel is spun, the position of the bullet is fixed. If you go first, you lose if and only if the bullet is in chamber 1, 3 and 5. So the probability that you lose is the same as the second player, 1/2. In that sense, whether to go first or second does not matter.

Now, let’s change the rule slightly. We will spin the barrel again after every trigger pull. Will you choose to be the first or the second player? And what is your probability of loss?

Solution: The difference is that each run now becomes independent. Assume that the first player’s probability of losing is $p$, then the second player’s probability of losing is $1-p$. Let’s condition the probability on the first person’s first trigger pull. He has 1/6 probability of losing in this run. Otherwise, he essentially becomes the second player in the game with new (conditional) probability of losing $1-p$. That happens with probability 5/6. That gives us $p=1\times1/6+(1-p)\times5/6\Rightarrow p=6/11$. So you should choose to be the second player and have 5/11 probability of losing.

If instead of one bullet, two bullets are randomly put in the chamber. Your opponent played the first and he was alive after the first trigger pull. You are given the option whether to spin the barrel. Should you spin the barrel?

Solution: if you spin the barrel, the probability that you will lose in this round is 2/6. If you don’t spin the barrel, there are only 5 chambers left and your probability of losing in this round (conditioned on that your opponent survived) is 2/5. So you should spin the barrel.

What if the two bullets are randomly put in two consecutive positions? If your opponent survived his first round, should you spin the barrel?

Solution: Now we have to condition our probability on the fact that the positions of the two bullets are consecutive. As shown in Figure 4.3, let’s label the empty chambers as 1, 2, 3 and 4; label the ones with bullets 5 and 6. Since your opponent survived the first round, the possible position he encountered is 1, 2, 3 or 4 with equal probability. With 1/4 chance, the next one is a bullet (the position was 4). So if you don’t spin, the chance of survival is 3/4. If you spin the barrel, each position has equal probability of being chosen, and your chance of survival is only 2/3. So you should not spin the barrel.

Aces

Fifty-two cards are randomly distributed to 4 players with each player getting 13 cards. What is the probability that each of them will have an ace?

Solution: The problem can be answered using standard counting methods. To distribute 52 cards to 4 players with 13 cards each has $\dfrac{52!}{13!13!13!13!}$ permutations. If each player 83 needs to have one ace, we can distribute the aces first, which has $4!$ ways. Then we distribute the rest 48 cards to 4 players with 12 cards each, which has $\dfrac{48!}{12!12!12!12!}$ permutations. So the probability that each of them will have an Ace is

The logic becomes clearer if we use a conditional probability approach. Let’s begin with any one of the four aces; it has probability $52/52=1$ of belonging to a pile. The second ace can be any of the remaining 51 cards, among which 39 belong to a pile different from the first ace. So the probability that the second ace is not in the pile of the first ace is 39/51. Now there are 50 cards left, among which 26 belong to the other two piles. So the conditional probability that the third ace is in one of the other 2 piles given the first two aces are already in different piles is 26/50. Similarly, the conditional probability that the fourth ace is in the pile different from the first three aces given that the first three aces are in different piles is 13/49. So the probability that each pile has an ace is $1\times\dfrac{39}{51}\times\dfrac{26}{50}\times\dfrac{13}{49}$.

Gambler’s ruin problem

A gambler starts with an initial fortune of $i$ dollars. On each successive game, the gambler wins $1 with probability $p,\ 0<p<1$, or loses $1 with probability $q=1-p$. He will stop if he either accumulates $N$ dollars or loses all his money. What is the probability that he will end up with $N$ dollars?

Solution: This is a classic textbook probability problem called the Gambler’s Ruin Problem. Interestingly, it is still widely used in quantitative interviews.

From any initial state $i$ (the dollars the gambler has), $0\le i\le N$, let $P_i$ be the probability that the gambler’s fortune will reach $N$ instead of 0. The next state is either $i+1$ with probability $p$ or $i-1$ with probability $q$. So we have

We also have the boundary probabilities $P_0=0$ and $P_N=1$.

So starting from $P_2$, we can successively evaluate $P_i$ as an expression of $P_1$:

Extending this expression to $P_N$, we have

Basketball scores

A basketball player is taking 100 free throws. She scores one point if the ball passes through the hoop and zero point if she misses. She has scored on her first throw and missed on her second. For each of the following throw the probability of her scoring is the fraction of throws she has made so far. For example, if she has scored 23 points after the 40th throw, the probability that she will score in the 41th throw is 23/40. After 100 throws (including the first and the second), what is the probability that she scores exactly 50 baskets?¹⁸

Solution: Let $(n,k),\ 1\le k\le n$, be the event that the player scores $k$ baskets after $n$ throws and $P_{n,k}=P((n,k))$. The solution is surprisingly simple if we use an induction approach starting with $n=3$. The third throw has 1/2 probability of scoring. So we have $P_{3,1}=1/2$ and $P_{3,2}=1/2$. For the case when $n=4$, let’s apply the law of total probability

The results indicate that $P_{n,k}=\dfrac1{n-1},\ \forall k=1,2,\cdots,n-1$, and give the hint that the law of total probability can be used in the induction step.

Induction step: given that $P_{n,k}=\dfrac1{n-1},\ \forall k=1,2,\cdots,n-1$, we need to prove $P_{n+1,k}=\dfrac1{(n+1)-1}=\dfrac1n,\ \forall k=1,2,\cdots,n$. To show it, simply apply the law of total probability:

The equation is also applicable to the $P_{n+1,1}$ and $P_{n+1,n}$, although in these cases $\dfrac{k-1}n=0$ and $\left(1-\dfrac kn\right)=0$, respectively. So we have $P_{n,k}=\dfrac1{n-1},\ \forall k=1,2,\cdots,n-1$ and $\forall n\ge2$. Hence, $P_{100,50}=1/99$.

Cars on road

If the probability of observing at least one car on a highway during any 20-minute time interval is 609/625, then what is the probability of observing at least one car during any 5-minute time interval? Assume that the probability of seeing a car at any moment is uniform (constant) for the entire 20 minutes.

Solution: We can break down the 20-minute interval into a sequence of 4 non-overlapping 5-minute intervals. Because of constant default probability (of observing a car), the probability of observing a car in any 5-minute interval is constant. Let’s denote the probability to be $p$, then the probability that in any 5-minute interval we do not observe a car is $1-p$.

The probability that we do not observe any car in all four of such independent 5-minute intervals is $(1-p)^4=1-609/625=16/625$, which gives $p=3/5$.