4.4 Discrete and Continuous Distributions

In this section, we review a variety of distribution functions for random variables that are widely used in quantitative modeling. Although it may not be necessary to memorize the properties of these distributions, having an intuitive understanding of the distributions and having the ability to quickly derive important properties are valuable skills in practice. As usual, let’s begin with the theories:

Common function of random variables

Table 4.1 summarizes how the basic properties of discrete and continuous random variables are defined or calculated. These are the basics you should commit to memory.

Table 4.1 Basic properties of discrete and continuous random variables
Random variable $(X)$	Discrete	Continuous¹⁹
Cumulative distribution function/cdf	$F(a)=P\{X\le a\}$	$F(a)=\int_{-\infty}^{a}f(x)dx$
Probability mass function /pmf Probability density function /pdf	pmf: $p(x)=P\{X=x\}$	pdf: $f(x)=\frac{d}{dx}F(x)$
Expected value/ $E[X]$	$\displaystyle\sum_{x:p(x)>0}xp(x)$	$\displaystyle\int_{-\infty}^{\infty}xf(x)dx$
Expected value of g(X)/ $E[g(X)]$	$\displaystyle\sum_{x:p(x)>0}g(x)p(x)$	$\displaystyle\int_{-\infty}^{\infty}g(x)f(x)dx$
Variance of $X/var(X)$	$E[(X-E[X])^2]=E[X^2]-(E[X])^2$
Standard deviation of $X/std(X)$	$\sqrt{var(X)}$

Discrete random variables

Table 4.2 includes some of the most widely-used discrete distributions. Discrete uniform random variable represents the occurrence of a value between number $a$ and $b$ when all values in the set $\{a,a+1,\cdots,b\}$ have equal probability. Binomial random variable represents the number of successes in a sequence of $n$ experiments when each trial is 87 independently a success with probability $p$ . Poisson random variable represents the number of events occurring in a fixed period of time with the expected number of occurrences $\lambda t$ when events occur with a known average rate $\lambda$ and are independent of the time since the last event. Geometric random variable represents the trial number $(n)$ to get the first success when each trial is independently a success with probability $p$ . Negative Binomial random variable represents the trial number to get to the $r$ -th success when each trial is independently a success with probability $p$ .

Table 4.2 Probability mass function, expected value and variance of discrete random variables
Name	Probability mass function (pmf)	$E[X]$	$var(X)$
Uniform	$P(x)=\frac{1}{b-a+1},\quad x=a,a+1,\cdots,b$	$\frac{b+a}{2}$	$\frac{(b-a+1)^2-1}{12}$
Binomial	$P(x)=\binom{n}{x}p^x(1-p)^{n-x},\quad x=0,1,\cdots,n$	$np$	$np(1-p)$
Poisson	$P(x)=\frac{e^{-\lambda t}(\lambda t)^x}{x!},\quad x=0,1,\cdots$ ²⁰	$\lambda t$	$\lambda t$
Geometric	$P(x)=(1-p)^{x-1}p,\quad x=1,2,\cdots$	$\frac{1}{p}$	$\frac{1-p}{p^2}$
Negative Binomial	$P(x)=\binom{x-1}{r-1}p^r(1-p)^{x-r},\quad x=r,r+1,\cdots$	$\frac{r}{p}$	$\frac{r(1-p)}{p^2}$

Continuous random variables

Table 4.3 includes some of the commonly encountered continuous distributions. Uniform distribution describes a random variable uniformly distributed over the interval $[a,b]$ . Because of the central limit theorem, normal distribution/Gaussian distribution is by far the most popular continuous distribution. Exponential distribution models the arrival time of an event if it has a constant arrival rate $\lambda$ . Gamma distribution with parameters $(\alpha,\lambda)$ often arises, in practice, as the distribution of the amount of time one has to wait until a total of $n$ events occur. Beta distributions are used to model events 88 that are constrained within a defined interval. By adjusting the shape parameters α and β, it can model different shapes of probability density functions.²¹

Table 4.3 Probability density function, expected value and variance of continuous random variables
Name	Probability density function (pdf)	$E[X]$	$var(X)$
Uniform	$\frac{1}{b-a},\quad a\le x\le b$	$\frac{b+a}{2}$	$\frac{(b-a)^2}{12}$
Normal	$\frac{1}{\sqrt{2\pi}\sigma}e^{\frac{-(x-\mu)^2}{2\sigma^2}},\quad x\in(-\infty,\infty)$	$\mu$	$\sigma^2$
Exponential	$\lambda e^{-\lambda x},\quad x\ge0$	$1/\lambda$	$1/\lambda^2$
Gamma	$\frac{\lambda e^{-\lambda x}(\lambda x)^{\alpha-1}}{\Gamma(\alpha)},\quad x\ge0,\quad\Gamma(a)=\int_0^\infty e^{-y}y^{a-1}$	$\alpha/\lambda$	$\alpha/\lambda^2$
Beta	$\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1},\quad 0<x<1$	$\frac{\alpha}{\alpha+\beta}$	$\frac{\alpha\beta}{(\alpha+\beta+1)(\alpha+\beta)^2}$

Meeting probability

Two bankers each arrive at the station at some random time between 5:00 am and 6:00 am (arrival time for either banker is uniformly distributed). They stay exactly five minutes and then leave. What is the probability they will meet on a given day?

Solution: Assume banker $A$ arrives $X$ minutes after 5:00 am and $B$ arrives $Y$ minutes after 5:00 am. $X$ and $Y$ are independent uniform distribution between 0 and 60. Since both only stay exactly five minutes, as shown in Figure 4.4, $A$ and $B$ meet if and only if $|X-Y|\le5$ .

So the probability that $A$ and $B$ will meet is simply the area of the shadowed region divided by the area of the square (the rest of the region can be combined to a square with size length 55):

\frac{60\times60-2\times(1/2\times55\times55)}{60\times60}=\frac{(60+55)\times(60-55)}{60\times60}=\frac{23}{144}.

Figure 4.4: X and Y axes from 0 to 60. The shaded diagonal band labeled A and B meet lies between Y=X−5 and Y=X+5. Both axes mark 5, 55 and 60; dashed guides mark 55. — Figure 4.4 Distributions of Banker A’s and Banker B’s arrival times

Probability of triangle

A stick is cut twice randomly (each cut point follows a uniform distribution on the stick), what is the probability that the 3 segments can form a triangle?²²

Solution: Without loss of generality, let’s assume that the length of the stick is 1. Let’s also label the point of the first cut as $x$ and the second cut as $y$ .

If $x<y$ , then the three segments are $x$ , $y-x$ and $1-y$ . The conditions to form a triangle are

A unit stick divided into three successive segments labeled x, y−x, and 1−y.

\begin{aligned}x+(y-x)&>1-y\Rightarrow y>1/2\\x+(1-y)&>y-x\Rightarrow y<1/2+x\\(y-x)+(1-y)&>x\Rightarrow x<1/2\end{aligned}

The feasible area is shown in Figure 4.5. The case for $x<y$ is the left gray triangle. Using symmetry, we can see that the case for $x>y$ is the right gray triangle.

Figure 4.5 as printed: a unit square with axes X and Y marked 0, 1/2 and 1, and an inscribed diamond joining the midpoints. The two upper corner triangles are gray. — Figure 4.5 Distribution of cuts X and Y

The total shadowed area represents the region where 3 segments can form a triangle, which is 1/4 of the square. So the probability is 1/4.

Property of Poisson process

You are waiting for a bus at a bus station. The buses arrive at the station according to a Poisson process with an average arrival time of 10 minutes $(\lambda=0.1/\mathrm{min})$ . If the buses have been running for a long time and you arrive at the bus station at a random time, what is your expected waiting time? On average, how many minutes ago did the last bus leave?

Solution: Considering the importance of jump-diffusion processes in derivative pricing and the role of Poisson processes in studying jump processes, let’s elaborate more on exponential random variables and the Poison process. Exponential distribution is widely used to model the time interval between independent events that happen at a constant average rate (arrival rate) $\lambda$ : $f(t)=\begin{cases}\lambda e^{-\lambda t}&(t\ge0)\\0&(t<0)\end{cases}$ . The expected arrival time is $1/\lambda$ and the variance is $1/\lambda^2$ . Using integration, we can calculate the cdf of an exponential distribution to be $F(t)=P(\tau\le t)=1-e^{-rt}$ and $P(\tau>t)=e^{-rt}$ , where $\tau$ is the random variable for arrival time. One unique property of exponential distribution is memorylessness: $P\{\tau>s+t\mid\tau>s\}=P(\tau>t\}$ .²³ That means if we have waited for $s$ time units, the extra waiting time has the same distribution as the waiting time when we start at time 0.

When the arrivals of a series of events each independently follow an exponential distribution with arrival rate $\lambda$ , the number of arrivals between time 0 and $t$ can be modeled as a Poisson process $P(N(t)=x)=\frac{e^{-\lambda t}\lambda t^x}{x!},\quad x=0,1,\cdots$ .²⁴ The expected number of arrivals is $\lambda t$ and the variance is also $\lambda t$ . Because of the memoryless nature of exponential distribution, the number of arrivals between time $s$ and $t$ is also a Poisson process $P(N(t-s)=x)=\frac{e^{-\lambda(t-s)}(\lambda(t-s))^x}{x!}$ .

Taking advantage of the memoryless property of exponential distribution, we know that the expected waiting time is $1/\lambda=10\ \mathrm{min}$ . If you look back in time, the memoryless property still applies. So on average, the last bus arrived 10 minutes ago as well.

This is another example that your intuition may misguide you. You may be wondering that if the last bus on average arrived 10 minutes ago and the next bus on average will arrive 10 minutes later, shouldn’t the average arrival time be 20 minutes instead of 10? The explanation to the apparent discrepancy is that when you arrive at a random time, you are more likely to arrive in a long time interval between two bus arrivals than in a short one. For example, if one interval between two bus arrivals is 30 minutes and another is 5 minutes, you are more likely to arrive at a time during that 30-minute interval rather than 5-minute interval. In fact, if you arrive at a random time, the expected residual life (the time for the next bus to arrive) is $\frac{E[X^2]}{2E[X]}$ for a general distribution.²⁵

Moments of normal distribution

If $X$ follows standard normal distribution $(X\sim N(0,1))$ , what is $E[X^n]$ for $n=1,2,3$ and 4?

Solution: The first to fourth moments of the standard normal distribution are essentially the mean, the variance, the skewness and the kurtosis. So you probably have remembered that the answers are 0, 1, 0 (no skewness), and 3, respectively.

Standard normal distribution has pdf $f(x)=\frac{1}{\sqrt{2\pi}}e^{-x^2/2}$ . Using simple symmetry we have $E[x^n]=\int_{-\infty}^{\infty}x^n\frac{1}{\sqrt{2\pi}}e^{-x^2/2}dx=0$ when $n$ is odd. For $n=2$ , integration by parts are often used. To solve $E[X^n]$ for any integer $n$ , an approach using moment generating functions may be a better choice. Moment generating functions are defined as

M(t)=E[e^{tX}]=\begin{cases}\displaystyle\sum_x e^{tx}p(x),&\textit{if x is discrete}\\\displaystyle\int_{-\infty}^{\infty}e^{tx}f(x)dx,&\textit{if x is continuous}\end{cases}

Sequentially taking derivative of $M(t)$ , we get one frequently-used property of $M(t)$ :

M'(t)=\frac{d}{dt}E[e^{tX}]=E[Xe^{tX}]\Rightarrow M'(0)=E[X],

M''(t)=\frac{d}{dt}E[Xe^{tX}]=E[X^2e^{tX}]\Rightarrow M''(0)=E[X^2],

and $M^n(0)=E[X^n],\ \forall n\ge1$ in general.

We can use this property to solve $E[X^n]$ for $X\sim N(0,1)$ . For standard normal distribution

M(t)=E[e^{tX}]=\int_{-\infty}^{\infty}e^{tx}\frac{1}{\sqrt{2\pi}}e^{-x^2/2}dx=e^{t^2/2}\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}}e^{-(x-t)^2/2}dx=e^{t^2/2}.

( $\frac{1}{\sqrt{2\pi}}e^{-(x-t)^2/2}$ is the pdf of normal distribution $X\sim N(t,1)$ , so $\int_{-\infty}^{\infty}f(x)dx=1$ ).

Taking derivatives, we have

M'(t)=te^{t^2/2}\Rightarrow M'(0)=0,\quad M''(t)=e^{t^2/2}+t^2e^{t^2/2}\Rightarrow M''(0)=e^0=1,

M^3(t)=te^{t^2/2}+2te^{t^2/2}+t^3e^{t^2/2}=3te^{t^2/2}+t^3e^{t^2/2}\Rightarrow M^3(0)=0,

and

M^4(t)=3e^{t^2/2}+3t^2e^{t^2/2}+3t^2e^{t^2/2}+3t^4e^{t^2/2}\Rightarrow M^4(0)=3e^0=3.

¹⁹ For continuous random variables, $P(X=x)=0,\ \forall x\in(-\infty,\infty)$ , so $P\{X\le x\}=P\{X<x\}$ . ↩

²⁰ Here we use the product of arrival rate $\lambda$ and time to define the parameter (expected value) since it is the definition used in many Poisson process studies. ↩

²¹ For example, beta distribution is widely used in modeling loss given default in risk management. If you are familiar with Bayesian statistics, you will also recognize it as a popular conjugate prior function. ↩

²² Hint: Let the first cut point be $x$ , the second one be $y$ , use the figure to show the distribution of $x$ and $y$ . ↩

²³ $P\{\tau>s+t\mid\tau>s\}=e^{-\lambda(s+t)}/e^{-\lambda s}=e^{-\lambda t}=P(x>t\}$ . ↩

²⁴ More rigorously, $N(t)$ is defined as a right-continuous function. ↩

²⁵ The residual life is explained in Chapter 3 of “Discrete Stochastic Process” by Robert G. Gallager. ↩