3.4 Important Calculus Methods

Taylor’s series

One-dimensional Taylor’s series expands function $f(x)$ as the sum of a series using the derivatives at a point $x=x_0$:

If $x_0=0$, $\displaystyle f(x)=f(0)+f'(0)x+\frac{f''(0)}{2!}x^2+\cdots+\frac{f^{(n)}(0)}{n!}x^n+\cdots$

Taylor’s series are often used to represent functions in power series terms. For example, Taylor’s series for three common transcendental functions, $e^x$, $\sin x$ and $\cos x$, at $x_0=0$ are

The Taylor’s series can also be expressed as the sum of the nth-degree Taylor polynomial $\displaystyle T_n(x)=f(x_0)+f'(x_0)(x-x_0)+\frac{f''(x_0)}{2!}(x-x_0)^2+\cdots+\frac{f^{(n)}(x_0)}{n!}(x-x_0)^n$ and a remainder $R_n(x)$: $f(x)=T_n(x)+R_n(x)$.

For some $\tilde{x}$ between $x_0$ and $x$, $\displaystyle R_n(x)=\frac{f^{(n+1)}(\tilde{x})}{(n+1)!}|x-x_0|^{n+1}$. Let $M$ be the maximum of $\left|f^{(n+1)}(\tilde{x})\right|$ for all $\tilde{x}$ between $x_0$ and $x$, we get constraint $\displaystyle |R_n(x)|\leq\frac{M\times|x-x_0|^{n+1}}{(n+1)!}$.

A. What is $i^i$?

Solution: The solution to this problem uses Euler’s formula, $e^{i\theta}=\cos\theta+i\sin\theta$, which can be proven using Taylor’s series. Let’s look at the proof. Applying Taylor’s series to $e^{i\theta}$, $\cos\theta$ and $\sin\theta$, we have

Combining these three series, it is apparent that $e^{i\theta}=\cos\theta+i\sin\theta$.

When $\theta=\pi$, the equation becomes $e^{i\pi}=\cos\pi+i\sin\pi=-1$. When $\theta=\pi/2$, the equation becomes $e^{i\pi/2}=\cos(\pi/2)+i\sin(\pi/2)=i$.³ So $\ln i=\ln\left(e^{i\pi/2}\right)=i\pi/2$.

Hence, $\ln(i^i)=i\ln i=i(i\pi/2)=-\pi/2\Rightarrow i^i=e^{-\pi/2}$.

B. Prove $(1+x)^n\geq1+nx$ for all $x>-1$ and for all integers $n\geq2$.

Solution: Let $f(x)=(1+x)^n$. It is clear that $1+nx$ is the first two terms in the Taylor’s series of $f(x)$ with $x_0=0$. So we can consider solving this problem using Taylor’s series.

For $x_0=0$ we have $(1+x)^n=1$ for $\forall n\geq2$. The first and secondary derivatives of $f(x)$ are $f'(x)=n(1+x)^{n-1}$ and $f''(x)=n(n-1)(1+x)^{n-2}$. Applying Taylor’s series, we have

where $x\leq\tilde{x}\leq0$ if $x<0$ and $x\geq\tilde{x}\geq0$ if $x>0$.

Since $x>-1$ and $n\geq2$, we have $n>0$, $(n-1)>0$, $(1+\tilde{x})^{n-2}>0$, $x^2\geq0$.

Hence, $n(n-1)(1+\tilde{x})^{n-2}x^2\geq0$ and $f(x)=(1+x)^n\geq1+nx$.

If Taylor’s series does not jump to your mind, the condition that $n$ is an integer may give you the hint that you can try the induction method. We can rephrase the problem as: for every integer $n\geq2$, prove $(1+x)^n\geq1+nx$ for $x>-1$.

The base case: show $(1+x)^n\geq1+nx,\forall x>-1$ when $n=2$, which can be easily proven since $(1+x)^2\geq1+2x+x^2\geq1+2x,\forall x>-1$.

The induction step: show that if $(1+x)^n\geq1+nx,\forall x>-1$ when $n=k$, the same statement holds for $n=k+1$: $(1+x)^{k+1}\geq1+(k+1)x,\forall x>-1$. This step is straightforward as well.

So the statement holds for all integers $n\geq2$ when $x>-1$.

Newton’s method

Newton’s method, also known as the Newton-Raphson method or the Newton-Fourier method, is an iterative process for solving the equation $f(x)=0$. It begins with an initial value $x_0$ and applies the iterative step $\displaystyle x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}$ to solve $f(x)=0$ if $x_1,x_2,\cdots$ converge.⁴

Convergence of Newton’s method is not guaranteed, especially when the starting point is far away from the correct solution. For Newton’s method to converge, it is often necessary that the initial point is sufficiently close to the root; $f(x)$ must be differentiable around the root. When it does converge, the convergence rate is quadratic, which means $\displaystyle\frac{(x_{n+1}-x_f)^2}{(x_n-x_f)^2}\leq\delta<1$, where $x_f$ is the solution to $f(x)=0$.

A. Solve $x^2=37$ to the third digit.

Solution: Let $f(x)=x^2-37$, the original problem is equivalent to solving $f(x)=0$. $x_0=6$ is a natural initial guess. Applying Newton’s method, we have

($6.083^2=37.00289$, which is very close to 37.)

If you do not remember Newton’s method, you can directly apply Taylor’s series for function $f(x)=\sqrt{x}$ with $f'(x)=\frac{1}{2}x^{-1/2}$:

Alternatively, we can use algebra since it is obvious that the solution should be slightly higher than 6. We have $(6+y)^2=37\Rightarrow y^2+12y-1=0$. If we ignore the $y^2$ term, which is small, then $y=0.083$ and $x=6+y=6.083$.

B. Could you explain some root-finding algorithms to solve $f(x)=0$? Assume $f(x)$ is a differentiable function.

Solution: Besides Newton’s method, the bisection method and the secant method are two alternative methods for root-finding.⁵

Bisection method is an intuitive root-finding algorithm. It starts with two initial values $a_0$ and $b_0$ such that $f(a_0)<0$ and $f(b_0)>0$. Since $f(x)$ is differentiable, there must be an $x$ between $a_0$ and $b_0$ that makes $f(x)=0$. At each step, we check the sign of $f((a_n+b_n)/2)$. If $f((a_n+b_n)/2)<0$, we set $b_{n+1}=b_n$ and $a_{n+1}=(a_n+b_n)/2$; If $f((a_n+b_n)/2)>0$, we set $a_{n+1}=a_n$ and $b_{n+1}=(a_n+b_n)/2$; If $f((a_n+b_n)/2)=0$, or its absolute value is within allowable error, the iteration stops and $x=(a_n+b_n)/2$. The bisection method converges linearly, $\displaystyle\frac{x_{n+1}-x_f}{x_n-x_f}\leq\delta<1$, which means it is slower than Newton’s method. But once you find an $a_0/b_0$ pair, convergence is guaranteed.

Secant method starts with two initial values $x_0$, $x_1$ and applies the iterative step $\displaystyle x_{n+1}=x_n-\frac{x_n-x_{n-1}}{f(x_n)-f(x_{n-1})}f(x_n)$. It replaces the $f'(x_n)$ in Newton’s method with a linear approximation $\displaystyle\frac{f(x_n)-f(x_{n-1})}{x_n-x_{n-1}}$. Compared with Newton’s method, it does not require the calculation of derivative $f'(x_n)$, which makes it valuable if $f'(x)$ is difficult to calculate. Its convergence rate is $(1+\sqrt{5})/2$, which makes it faster than the bisection method but slower than Newton’s method. Similar to Newton’s method, convergence is not guaranteed if initial values are not close to the root.

Lagrange multipliers

The method of Lagrange multipliers is a common technique used to find local maximums/minimums of a multivariate function with one or more constraints.⁶

Let $f(x_1,x_2,\cdots,x_n)$ be a function of $n$ variables $x=(x_1,x_2,\cdots,x_n)$ with gradient vector $\displaystyle\nabla f(x)=\left\langle\frac{\partial f}{\partial x_1},\frac{\partial f}{\partial x_2},\cdots,\frac{\partial f}{\partial x_n}\right\rangle$. The necessary condition for maximizing or minimizing $f(x)$ subject to a set of $k$ constraints

is that $\nabla f(x)+\lambda_1\nabla g_1(x)+\lambda_2\nabla g_2(x)+\cdots+\lambda_k\nabla g_k(x)=0$, where $\lambda_1,\cdots,\lambda_k$ are called the Lagrange multipliers.

What is the distance from the origin to the plane $2x+3y+4z=12$?

Solution: The distance ($D$) from the origin to a plane is the minimum distance between the origin and points on the plane. Mathematically, the problem can be expressed as

Applying the Lagrange multipliers, we have

In general, for a plane with equation $ax+by+cz=d$, the distance to the origin is $\displaystyle D=\frac{|d|}{\sqrt{a^2+b^2+c^2}}$.