Lecture Notes

on

Numerical Analysis

by

Ruennhwa Ferng

Department of Applied Mathematics

National Chiao Tung University Hsin-Chu, TAIWAN

Contents

1 Mathematical Preliminaries 1

1.1 Review of Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Limit, Continuity, Derivative, and Integral . . . . . . . . . . . . . . . 1 1.1.2 Taylor’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Review of Linear Algebra and Matrix Analysis . . . . . . . . . . . . . . . . . 6 1.2.1 Matrices and Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.2 Vector Space, Range Space, and Null Space . . . . . . . . . . . . . . 9 1.2.3 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.3 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3.1 Vector Norm Definition and Properties . . . . . . . . . . . . . . . . . 11 1.3.2 Matrix Norm Definition and Properties . . . . . . . . . . . . . . . . . 13

1.4 SVD: The Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . 17

2 Computer Arithmetic 21 2.1 Floating-Point Number and Roundoff Error . . . . . . . . . . . . . . . . . . 21 2.2 Loss of Significance, Stability, and Conditioning . . . . . . . . . . . . . . . . 26

2.2.1 Loss of Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.2.2 Numerical Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.2.3 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.3 Floating-Point Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3 Direct Methods for Solving Systems of Linear Equations 37

3.1 Triangular Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.1.1 Diagonal System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.1.2 Forward Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.1.3 Back Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.2 Gaussian Elimination and LU Factorization . . . . . . . . . . . . . . . . . . 41 3.2.1 Gaussian Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2.2 Gaussian Transformation and LU Factorization . . . . . . . . . . . . 44 3.2.3 Existence and Uniqueness of LU Factorization . . . . . . . . . . . . . 50

3.3 Pivoting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.3.1 The Need for Pivoting . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.3.2 Partial Pivoting and Complete Pivoting . . . . . . . . . . . . . . . . 52 3.3.3 Scaled Row Pivoting . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.4 Some Special Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

3.4.1 Symmetric Positive Definite System and Cholesky Factorization … 59 3.4.2 Diagonally Dominant Systems . . . . . . . . . . . . . . . . . . . . . . 61 3.4.3 Tridiagonal System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.4.4 General Banded Systems . . . . . . . . . . . . . . . . . . . . . . . . . 64

3.5 Perturbation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4 Iterative Methods for Solving Systems of Linear Equations 69

4.1 Classic Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.1.1 Basic Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.1.2 Richard’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.1.3 Jacobi Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.1.4 Gauss-Seidel Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.1.5 Successive Over Relaxation (SOR) Method . . . . . . . . . . . . . . . 72 4.1.6 Symmetric Successive Over Relaxation (SSOR) Method . . . . . . . . 73

4.2 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

5 Solutions of Non-linear Equations 81 5.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.2 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.3 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.3.1 Derivation of Newton’s Method . . . . . . . . . . . . . . . . . . . . . 85 5.3.2 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.3.3 Examples and Pitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.3.4 System of Nonlinear Equations . . . . . . . . . . . . . . . . . . . . . 90

5.4 Quasi-Newton’s Method (Secant Method) . . . . . . . . . . . . . . . . . . . 91 5.4.1 The Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.4.2 Error Analysis of Secant Method . . . . . . . . . . . . . . . . . . . . 93

5.5 Fixed Point and Functional Iteration . . . . . . . . . . . . . . . . . . . . . . 97 5.5.1 Functional Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.5.2 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

6 Interpolation 103

6.1 Polynomial Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.1.1 Existence And Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . 103 6.1.2 Naive Approach for Polynomial Interpolation . . . . . . . . . . . . . 104 6.1.3 Lagrange Form and Neville’s Method . . . . . . . . . . . . . . . . . . 105 6.1.4 Newton’s Form of Polynomial Interpolation . . . . . . . . . . . . . . 107 6.1.5 Divided Differences Scheme . . . . . . . . . . . . . . . . . . . . . . . 109 6.1.6 Error Analysis for Polynomial Interpolation . . . . . . . . . . . . . . 115

6.2 Hermite Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 6.2.1 Existence and Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . 117 6.2.2 Lagrange Form for Hermite Interpolation . . . . . . . . . . . . . . . . 118 6.2.3 Divided Difference Method for Hermite Interpolation . . . . . . . . . 118 6.2.4 Error Analysis for Hermite Interpolation . . . . . . . . . . . . . . . . 120 6.3 Spline Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

6.3.1 Cubic Spline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

7 Numerical Differentiation and Integration 125

7.1 Numerical Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 7.1.1 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . 125 7.1.2 Polynomial Interpolation Method . . . . . . . . . . . . . . . . . . . . 126 7.1.3 Richardson Extrapolation Method . . . . . . . . . . . . . . . . . . . . 128

7.2 Numerical Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 7.2.1 Elements of Numerical Integration . . . . . . . . . . . . . . . . . . . 130 7.2.2 Newton-Cotes Formulas . . . . . . . . . . . . . . . . . . . . . . . . . 131 7.2.3 Composite Newton-Cotes Forumlas . . . . . . . . . . . . . . . . . . . 137

7.3 Gaussian Quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 7.3.1 Orthogonal Polynomials and Quadrature Rule . . . . . . . . . . . . . 140 7.3.2 Gaussian Quadrature Rule . . . . . . . . . . . . . . . . . . . . . . . . 145 7.3.3 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

7.4 Adaptive Quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 7.5 Romberg Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

8 Numerical Solutions of Ordinary Differential Equations 151 8.1 Existence and Uniqueness of Solutions . . . . . . . . . . . . . . . . . . . . . 151 8.2 Euler’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 8.3 Runge-Kutta Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 8.4 Systems and Higher-Order Ordinary Differential Equations . . . . . . . . . . 154

9 Boundary-Value Problems for Ordinary Differential Equations 157 9.1 Mathematical Theories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 9.2 Finite Difference Method For Linear Problems . . . . . . . . . . . . . . . . . 162

9.2.1 The Finite Difference Formulation . . . . . . . . . . . . . . . . . . . . 163 9.2.2 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 9.2.3 Higher Order Approximations . . . . . . . . . . . . . . . . . . . . . . 165

9.3 Finite Difference Method For Nonlinear Problems . . . . . . . . . . . . . . . 166 9.4 Shooting Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

List of Tables

2.1 Some characteristics of IEEE standard floating-point numbers . . . . . . . . 25 2.2 IEEE exception handling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

List of Figures

2.1 32-bit single precision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.2 64-bit double precision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Chapter 1

Mathematical Preliminaries

In this chapter, we review some of the most important topics from Calculus, Linear Algebra, and Matrix Analysis that will be required in the subsequent chapters throught this course. Throught this book, ℝ will denote the set of all real numbers.

1.1 Review of Calculus

We start with the review of some definitions and theorems in Calculus including limit, continuity, derivative, and Riemann integral. We shall emphasize the various forms of Taylor’s theorem.

1.1.1 Limit, Continuity, Derivative, and Integral

Unless otherwise stated, f will denote a real-valued function defined in a domain Ω ⊆ ℝ, i.e., f : Ω → ℝ.

Definition 1.1 (Limit) We say f has the limit L at c, written as lim_{x→c} f(x) = L, if

∀ ε > 0, ∃ δ > 0, such that |x − c| < δ ⇒ |f(x) − L| < ε.

If there is no number L with this property, then we say the limit of f at c does not exist.

Definition 1.2 (Continuous) The function f is said to be continuous at c if

lim_{x→c} f(x) = f(c).

Definition 1.3 Let {x_{i}}_{i=1}^{∞}, where x_{i} ∈ ℝ or ℂ, be an infinite sequence. The sequence converges to x^{⋆}, write lim_{i→∞} x_{i} = x^{⋆} or x_{i} → x^{⋆} as i → ∞, if for any ε > 0, there exists a positive integer N(ε) such that i > N implies |x_{i} − x^{⋆}| < ε.

Theorem 1.1 Suppose f : Ω → ℝ, where Ω ⊆ ℝ, is a function and x^{⋆} ∈ X. Then the following are equivalent.

1. f is continuous at x^{⋆}.

2. If {x_{i}}_{i=1}^{∞} is any sequence in X and lim_{i→∞} x_{i} = x^{⋆}, then lim_{i→∞} f(x_{i}) = f(x^{⋆}).

Definition 1.4 (Derivative) The derivative of the function f at c is the limit, if it exists, defined by

f′(c) = lim_{x→c} \frac{f(x) − f(c)}{x − c}.

If f′(c) exists, then we say f is defferentiable at c.

Remarks 1.1 Another form of derivative:

f′(x) = lim_{h→0} \frac{f(x + h) − f(x)}{h}.

Theorem 1.2 If f is differentiable at c, then f must be continuous at c. But the converse is not true.

Definition 1.5 By convention, we use the following definitions and notations.

• C(Ω): the set of all functions that are continuous everywhere in Ω.

• C^{1}(Ω): the set of all functions f for which f′ exists and is continuous everywhere in Ω.

• C^{n}(Ω): the set of all functions f for which f^{(n)} exists and is continuous everywhere in Ω.

• C^{∞}(Ω): the set of all functions each of whose derivatives is continuous.

Property 1.1 C^{∞}(Ω) ⊂ … ⊂ C^{2}(Ω) ⊂ C^{1}(Ω) ⊂ C(Ω)

Remarks 1.2 Similarly, we define C^{n}[a, b] to be the set of all functions f defined on [a, b] such that f^{(n)} exists and is continuous everywhere in the interval (a, b).

Theorem 1.3 (Rolle’s Theorem) If f ∈ C[a, b], f′ exists on (a, b), and f(a) = f(b) = 0, then there exists at least one ξ ∈ (a, b) such that f′(ξ) = 0.

Theorem 1.4 (Generalized Roll’s Theorem) Suppose f ∈ C[a, b] is n times differentiable on (a, b). If f(x) = 0 for x = x_{0}, x_{1}, …, x_{n}, where x_{i} ∈ [a, b], are distinct, then there is a ξ ∈ (a, b) such that f^{(n)}(ξ) = 0.

Theorem 1.5 (Mean Value Theorem) If f ∈ C[a, b] and f′ exists on (a, b), then for any x, c ∈ [a, b], there exist a number ξ between x and c, such that

f(x) = f(c) + f′(ξ)(x − c),

or, equivalently,

f′(ξ) = \frac{f(x) − f(c)}{x − c}.

Remarks 1.3 Rolle’s theorem is a special case of the Mean Value theorem.

Theorem 1.6 (Intermediate Value Theorem) If f ∈ C[a, b] and η is some number between f(a) and f(b), then there exists ξ ∈ (a, b) such that f(ξ) = η.

Theorem 1.7 (Extreme Value Theorem) If f ∈ C[a, b], then there exist ξ and η in [a, b] such that f(ξ) ≤ f(x) ≤ f(η) for all x ∈ [a, b]. If, in addition, f is differentiable on (a, b), then ξ and η occur either at the end-points of [a, b] or where f′ is zero.

Definition 1.6 (Riemann Integral) Suppose f is a function defined on [a, b]. The Riemann integral of f on [a, b] is the limit, if it exists,

∫_{a}^{b} f(x)dx = lim_{max △x_{i}→0} ∑_{i=1}^{n} f(z_{i}) △ x_{i},

where △x_{i} = x_{i} − x_{i−1}, z_{i} ∈ [x_{i−1}, x_{i}] is arbitrary, and a = x_{0} ≤ x_{1} ≤ ⋯ ≤ x_{n} = b is a partition of [a, b].

Remarks 1.4 In the case of equally spaced partition and z_{i} = x_{i}, one has

∫_{a}^{b} f(x)dx = lim_{n→∞} \frac{b − a}{n} ∑_{i=1}^{n} f(x_{i}),

where x_{i} = a + i(b − 1)/n.

Theorem 1.8 (Mean Value Theorem for Integral) If f is continuous on [a, b] and g is integrable and does not change sign on [a, b], then there exists ξ ∈ (a, b), such that

∫_{a}^{b} f(x)g(x)dx = f(ξ) ∫_{a}^{b} g(x)dx.

In the special case when g(x) ≡ 1 on [a, b], then one has

∫_{a}^{b} f(x)dx = f(ξ)(b − a).

1.1.2 Taylor’s Theorem

Theorem 1.9 (Taylor’s Theorem with Lagrange Formula) Suppose f ∈ C^{n}[a, b] and f^{(n+1)} exists on (a, b). Then for any x and c in [a, b], there exists a number ξ, depending on x, between x and c such that

f(x) = P_{n}(x) + E_{n}(x),  (1.1)

where

P_{n}(x) = ∑_{k=0}^{n} \frac{1}{k!}f^{(k)}(c)(x − c)^{k}  (1.2)

is called the n-th Taylor polynomial for f centered at c, and

E_{n}(x) = \frac{1}{(n + 1)!}f^{(n+1)}(ξ)(x − c)^{n+1}  (1.3)

is called the remainder term, error term, or truncation error associated with P_{n}(x). The infinite series by taking the limit of P_{n}(x) as n → ∞ is called the Taylor series for f(x).

Remarks 1.5 The mean-value theorem is a special case of Tayler’s theorem by taking n = 0.

Theorem 1.10 (Taylor’s Theorem with Integral Form) If f ∈ C^{n+1}[a, b], then for any x and c in [a, b],

f(x) = ∑_{k=0}^{n} \frac{1}{k!}f^{(k)}(c)(x − c)^{k} + R_{n}(x),  (1.4)

where

R_{n}(x) = \frac{1}{n!} ∫_{c}^{x} f^{(n+1)}(t)(x − t)^{n}dt.  (1.5)

Theorem 1.11 (Other forms of Taylor’s Theorem) If f ∈ C^{n+1}[a, b], then for any x ∈ (a, b) and h ≥ 0 in a neighborhood of x, there exists ξ such that

f(x + h) = ∑_{k=0}^{n} \frac{1}{k!}f^{(k)}(x)h^{k} + \frac{1}{(n + 1)!}f^{(n+1)}(ξ)h^{n+1},  (1.6)

if x < ξ < x + h, and

f(x − h) = ∑_{k=0}^{n}(−1)^{k} \frac{1}{k!}f^{(k)}(x)h^{k} + \frac{(−1)^{n+1}}{(n + 1)!} f^{(n+1)}(ξ)h^{n+1},  (1.7)

if x − h < ξ < x.

Theorem 1.12 (Taylor’s Theorem in Two Variables) Suppose that f(x, y) and all its partial derivatives of order less than or equal to n + 1 are continuous on D = {(x, y)|a ≤ x ≤ b, c ≤ y ≤ d}, and let (x_{0}, y_{0}) ∈ D. For every (x, y) ∈ D, there exists ξ between x and x_{0} and μ between y and y_{0} with

f(x, y) = P_{n}(x, y) + R_{n}(x, y),  (1.8)

where

P_{n}(x, y) = f(x_{0}, y_{0}) + [(x − x_{0})\frac{∂f}{∂x}(x_{0}, y_{0}) + (y − y_{0})\frac{∂f}{∂y} (x_{0}, y_{0})]

+ [\frac{(x − x_{0})^{2}}{2} \frac{∂^{2}f}{∂x^{2}} (x_{0}, y_{0}) + (x − x_{0})(y − y_{0})\frac{∂^{2}f}{∂x∂y} (x_{0}, y_{0})

+\frac{(y − y_{0})^{2}}{2} \frac{∂^{2}f}{∂y^{2}} (x_{0}, y_{0})] + · · · · · ·

+ [\frac{1}{n!} ∑_{j=0}^{n} \binom{n}{j}(x − x_{0})^{n−j}(y − y_{0})^{j} \frac{∂^{n}f}{∂x^{n−j}∂y^{j}} (x_{0}, y_{0})]  (1.9)

and

R_{n}(x, y) = \frac{1}{(n + 1)!} ∑_{j=0}^{n+1} \binom{n + 1}{j}(x − x_{0})^{n+1−j}(y − y_{0})^{j} \frac{∂^{n+1}f}{∂x^{n+1−j}∂y^{j}} (ξ, μ).  (1.10)

The function P_{n}(x, y) is called the n-th Taylor polynomial in two variables for thefunction f(x, y), and R_{n}(x, y) is the remainder term associated with P_{n}(x, y).

Example 1.1 Consider the function f(x) = ln x in the interval [1, 2]. Since f′(x) = x^{−1}, f′′(x) = −x^{−2}, f′′′(x) = 2x^{−3}, ⋯ and so on. In general, f^{(k)}(x) = (−1)^{k−1}(k − 1)!x^{−k}. Hence, with c = 1, Taylor’s theorem 1.9 gives,

ln x = ∑_{k=1}^{n} \frac{(−1)^{k−1}}{k} (x − 1)^{k} + \frac{(−1)^{n}}{n + 1} ξ^{−(n+1)}(x − 1)^{n+1},

where 1 ≤ x ≤ 2 and 1 < ξ < x. The error term is

|E_{n}| = \frac{1}{n + 1}ξ^{−(n+1)}(x − 1)^{n+1} ≤ \frac{1}{n + 1}(x − 1)^{n+1},

since ξ > 1 and ξ^{−(n+1)} < 1. To estimate ln 2 with an error less than 10^{−8}, it would require

|E_{n}| ≤ \frac{1}{n + 1} ≤ 10^{−8}.

This means n ≥ 10^{8} − 1. Similarly, to estimate ln 1.5 with the same precision, it requires

|E_{n}| ≤ \frac{1}{n + 1} (\frac{1}{2})^{n+1} ≤ 10^{−8}.

It is sufficient to attain the accuracy if n ≥ 26.

Example 1.2 Consider the function f(x) = cos x defined on ℝ. Since f ∈ C^{∞}(ℝ), Taylor’s theorem can be applied for any n > 0. With n = 2 and c = 0, we have

f(x) = cos x = 1 − \frac{1}{2}x^{2} + \frac{1}{6}x^{3} sin ξ,

where 0 < ξ < x. To estimate cos(0.01) using the Taylor’s theorem, one has

cos(0.01) = 1 − \frac{1}{2}(0.01)^{2} + \frac{1}{6}(0.01)^{3} sin ξ = 0.99995 + \frac{1}{6} × 10^{−6} sin ξ,

where 0 < ξ < 0.01. Since | sin ξ| < 1, the error is

| cos(0.01) − 0.99995| ≤ \frac{1}{6} × 10^{−6} ≈ 0.167 × 10^{−6}.

This means that the approximation 0.99995 matches at least the first five digits of cos(0.01). On the other hand, with n = 3, we also have

f(x) = cos x = 1 − \frac{1}{2}x^{2} + \frac{1}{24}x^{4} cos ξ.

The apporximation of cos(0.01) with this formula remains the same, still 0.99995, but we now have much better accuracy assurance since the error is now

|\frac{1}{24}x^{4} cos ξ| ≤ \frac{1}{24}(0.01)^{4} ≈ 0.42 × 10^{−9}.

This means that the approximation 0.99995 matches at least the first eight digits when compared to the 11-digit accuracy cos(0.01) = 0.99995000042.

These examples illustrate the two objectives of numerical analysis. The first is to find approximation, which both Taylor polynomials provide. The second objective is to determine the accuracy of the approximation. In the previous example, the later analysis is much more informative, even though both gave the same approximation.

1.2 Review of Linear Algebra and Matrix Analysis

1.2.1 Matrices and Vectors

We let ℝ^{n} denote the set of all real n-vectors (column vectors):

x ∈ ℝ^{n} ⟺ x = [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] , x_{i} ∈ ℝ, ∀ i = 1, 2, …, n,

in which x_{i} is referred as the i-th component of x. By convection, ℝ^{n×1} = ℝ^{n} denotes the set of column vectors and ℝ^{1×n} the row vectors, x ∈ ℝ^{1×n}, x = [x_{1}, x_{2}, …, x_{n}], x_{i} ∈ ℝ.

ℝ^{m×n} will denote the set of all m × n matrices:

A ∈ ℝ^{m×n} ⟺ A = [a_{ij}] = [ \begin{matrix}a_{11} & a_{12} & ⋯ & a_{1n} \\ a_{21} & a_{22} & ⋯ & a_{2n} \\ ⋮ & ⋮ & ... & ⋮ \\ a_{m1} & a_{m2} & ⋯ & a_{mn} \end{matrix} ] ,

in which a_{ij} ∈ ℝ is called the (i, j) component of A. Frequently, we use the notation

A = [a_{1}, a_{2}, …, a_{n}] ,

in which

a_{j} = [ a_{1j} ; a_{2j} ; ⋮ ; a_{mj} ] ∈ ℝ^{m}

denotes the j-th column of A. Analogously, we use ℂ^{n} and ℂ^{m×n} to denote the set of complex n-vectors and complex m × n matrices, respectively.

The transpose of matrix A is A^{T} ∈ ℝ^{n×m} and

A^{T} = [ \begin{matrix}a_{11} & a_{21} & ⋯ & a_{m1} \\ a_{12} & a_{22} & ⋯ & a_{m2} \\ ⋮ & ⋮ & ... & ⋮ \\ a_{1n} & a_{2n} & ⋯ & a_{mn} \end{matrix} ] .

For complex matrix A ∈ ℂ^{m×n}, the Hermitian of A is

A^{H} = [ \begin{matrix}ā_{11} & ā_{21} & ⋯ & ā_{m1} \\ ā_{12} & ā_{22} & ⋯ & ā_{m2} \\ ⋮ & ⋮ & ... & ⋮ \\ ā_{1n} & ā_{2n} & ⋯ & ā_{mn} \end{matrix} ] ,

where a_{ij} ∈ ℂ andā_{ij} denotes the complex conjugate of a_{ij}.

Some basic matrix-vector operations include

• dot-product: for x, y ∈ ℝ^{n},

x^{T} y = x_{1}y_{1} + ⋯ + x_{n}y_{n} = ∑_{i=1}^{n} x_{i}y_{i} ∈ ℝ.

• outer product: for x ∈ ℝ^{m}, y ∈ ℝ^{n},

xy^{T} = [x_{i}y_{j}] = [ \begin{matrix}x_{1}y_{1} & x_{1}y_{2} & ⋯ & x_{1}y_{n} \\ x_{2}y_{1} & x_{2}y_{2} & ⋯ & x_{2}y_{n} \\ ⋮ & ⋮ & ... & ⋮ \\ x_{m}y_{1} & x_{m}y_{2} & ⋯ & x_{m}y_{n} \end{matrix} ] ∈ ℝ^{m×n}.

• matrix-vector multiplication: for A = [a_{1}, a_{2}, …, a_{n}] ∈ ℝ^{m×n} and x ∈ ℝ^{n},

y = Ax = x_{1}a_{1} + x_{2}a_{2} + ⋯ + x_{n}a_{n}, or y_{i} = ∑_{k=1}^{n} a_{ik}x_{k}, i = 1, 2, …, n.

• matrix-matrix multiplication: for A ∈ ℝ^{m×p} and B ∈ ℝ^{p×n},

C = AB ∈ ℝ^{m×n} with c_{ij} = ∑_{k=1}^{p} a_{ik}b_{kj}.

• saxpy:

z = αx + y, or z_{i} = αx_{i} + y_{i}

with x, y, z ∈ ℝ^{n}, α ∈ ℝ.

• outer-product update:

A ← A + xy^{T} or a_{ij} ← a_{ij} + x_{i}y_{j}

with A ∈ ℝ^{m×n}, x ∈ ℝ^{m}, y ∈ ℝ^{n}.

• Gaxpy:

z = Ax + y or z_{i} = ∑_{k=1}^{n} a_{ik}x_{k} + y_{i}

with A ∈ ℝ^{m×n}, x ∈ ℝ^{n}, y ∈ ℝ^{m}, z ∈ ℝ^{m}.

The identity matrix of order n, I_{n} = [δ_{ij}], is a special matrix with entries

δ_{ij} = \begin{cases}1, & if i = j, \\ 0 & if i ≠ j. \end{cases}

The columns of I_{n},

e_{1} = [ 1 0 0 ⋮ 0 ] ,  e_{2} = [ 0 1 0 ⋮ 0 ] ,  ⋯,  e_{n} = [ 0 0 0 ⋮ 1 ] ,

are called the standard unit vectors. When the size of I_{n} is clear from the content, this matrix is generally written simply as I.

A square matrix A ∈ ℝ^{n×n} is said to be nonsingular (or invertible) if there exists B ∈ ℝ^{n×n} such that BA = AB = I_{n}, the n × n identity matrix. The matrix B is called the inverse of A and is denoted as A^{−1}.

When A and B are both n-by-n non-singular matrices, we have

(AB)^{−1} = B^{−1}A^{−1} and (A^{−1})^{T} = (A^{T})^{−1} = A^{−T} .

Suppose A ∈ ℝ^{n×n} is nonsingular and A^{−1} is known. The the inverse (A + xy^{T})^{−1}, where x, y ∈ ℝ^{n}, can be computed by the so-called Sherman-Morrison-Woodbury formula

(A + xy^{T})^{−1} = A^{−1} − \frac{A^{−1}xy^{T} A^{−1}}{1 + y^{T} A^{−1}x}

if 1 + y^{T} A^{−1}x ≠ 0. The extension to rank-k formula for U, V ∈ ℝ^{n×k} is

(A + UV^{T})^{−1} = A^{−1} − A^{−1}U(I + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}

if I + V^{T} A^{−1}U is nonsingular.

We list in the following some matrices which has either special structure or property.

Definition 1.7 Let A = [a_{ij}] be a square or rectangular matrix, A is called

1.2 Review of Linear Algebra and Matrix Analysis  9

• diagonal if a_{ij} = 0, ∀ i ≠ j;

• tridiagonal if a_{ij} = 0 if |i − j| > 1;

• upper bidiagonal if a_{ij} = 0, ∀ i > j or j > i + 1;

• upper triangular if a_{ij} = 0, ∀ i > j;

• strictly upper triangular if a_{ij} = 0, ∀ i ≥ j;

• upper Hessenberg matrix if a_{ij} = 0, ∀ i > j + 1;

• symmetric if A^{T} = A;

• skew symmetric if A^{T} = −A;

• if positive if a_{ij} > 0, ∀ i, j;

• positive definite if x^{T} Ax > 0, ∀ x ≠ 0, x ∈ ℝ^{n};

• non-negative definite or positive semi-definite if x^{T} Ax ≥ 0, ∀ x ∈ ℝ^{n};

• indefinite if there exist x, y ∈ ℝ^{n} such that (x^{T} Ax)(y^{T} Ay) < 0;

• idempotent if A^{2} = A.

• nilpotent if A^{k} = 0 for some positive integer k;

• diagonal dominant if |a_{ii}| > ∑_{j≠i} |a_{ij}|, ∀ i;

• orthogonal if A^{T} A = I (that is, A^{T} = A^{−1}) when A ∈ ℝ^{n×n} and unitary if A^{H}A = I when A ∈ ℂ^{n×n};

• normal if A^{T} A = AA^{T} when A ∈ ℝ^{n×n} and A^{H}A = AA^{H} when A ∈ ℂ^{n×n}.

1.2.2 Vector Space, Range Space, and Null Space

Definition 1.8 (Linearly Independent) We say vectors x_{1}, x_{2}, …, x_{m} ∈ ℝ^{n} are linearly independent if α_{1}, α_{2}, …, α_{m} ∈ ℝ

α_{1}x_{1} + α_{2}x_{2} + … + α_{m}x_{m} = 0 ⟹ α_{1} = α_{2} = … = α_{m} = 0.

If there exists any α_{k} ≠ 0 but ∑_{i=1}^{m} α_{i}x_{i} = 0, then x_{1}, x_{2}, …, x_{m} are said to be linearly dependent.

A subspace spanned by x_{1}, x_{2}, …, x_{m} is denoted and defined as

span {x_{1}, x_{2}, …, x_{m}} = {y ∈ ℝ^{n} | y = ∑_{i=1}^{m} α_{i}x_{i}for someα_{i} ∈ ℝ} .

Suppose S ⊆ ℝ^{n} is a subspace and B = {b_{1}, …, b_{m}} ⊆ S is a subset. If b_{1}, …, b_{m} are linearly independent and span(B) = S, then B is called a basis for S and dim(S) = m.

Definition 1.9 (Range and Null Space) Let A = [a_{1}, …, a_{n}] ∈ ℝ^{m×n}, where a_{i} ∈ ℝ^{m} are columns of A. The range of A is denoted and defined as

R(A) = {y ∈ ℝ^{m}| y = Ax, for some x ∈ ℝ^{n}} = span {a_{1}, a_{2}, …, a_{n}} ,  (1.11)

and the null space of A is denoted and defined as

N (A) = {x ∈ ℝ^{n}|Ax = 0} .  (1.12)

Definition 1.10 (Rank and Nullity) The rank of a matrix A, rank(A), is the dimension of R(A) and the nullity of A, nullity(A), is the dimension of N (A).

Note 1.1 rank(A) = rank(A^{T}) is the maximal number of linearly independent columns (or rows) in A.

Note 1.2 Suppose A ∈ ℝ^{m×n} and m ≥ n, then

rank(A) + nullity(A) = n.

Note 1.3 Suppose A ∈ ℝ^{n×n}. Then

A is non-singular (invertible, A^{−1} exists)

⟺ Ax = 0 ⇒ x = 0
⟺ Ax = b has a unique solution
⟺ rank(A) = n
⟺ det(A) ≠ 0

1.2.3 Orthogonality

Definition 1.11 (Orthogonal and Orthonormal) A set of vectors x_{1}, …, x_{m} ∈ ℝ^{n} are said to be orthogonal if

x_{i}^{T} x_{j} = 0, ∀ i ≠ j,

and orthonormal if

x_{i}^{T} x_{j} = δ_{ij} = \{ \begin{matrix}0 & i ≠ j \\ 1 & i = j \end{matrix}

Definition 1.12 Two subspaces S_{1} and S_{2} of ℝ^{n} are said to be orthogonal if x^{T} y = 0 for all x ∈ S_{1} and y ∈ S_{2}.

\underline{1.3 Norms 11}

Definition 1.13 (Orthogonal Complement) Suppose S ⊆ ℝ^{n} is a subspace. The orthogonal complement of S is defined as

S^{⊥} = {y ∈ ℝ^{n} | y^{T} x = 0, ∀ x ∈ S}

Definition 1.14 (Direct Sum) Suppose U and V are subspaces of a vector space S. If each s ∈ S can be written uniquely as a sum u + v, where u ∈ U and v ∈ V, then we say that S is a direct sum of U and V and we write S = U ⊕ V.

Theorem 1.13 Suppose A ∈ ℝ^{m×n}. Then

1. R(A)^{⊥} = N (A^{T});

2. N (A) = R(A^{T})^{⊥};

3. ℝ^{n} = R(A^{T}) ⊕ N (A).

Proof: For any y ∈ R(A)^{⊥}, y^{T} x = 0, ∀ x ∈ R(A). Since x is in R(A), there exists z ∈ ℝ^{n} such that x = Az. Then

0 = y^{T} x = y^{T} (Az) = (A^{T} y)^{T} z.

Since z = A^{T} y is arbitrary, it must be A^{T} y = 0. That is, y ∈ N (A^{T}). Hence

R(A)^{⊥} ⊆ N (A^{T}).

Conversely, suppose y ∈ N (A^{T}). Then A^{T} y = 0 and hence (A^{T} y)^{T} x = y^{T} (Ax) = 0 for any x ∈ ℝ^{n}. This means y ∈ R(A)^{⊥}. Therefore

N (A^{T}) ⊆ R(A)^{⊥}.

1.3 Norms

1.3.1 Vector Norm Definition and Properties

Definition 1.15 A vector norm is a function ∥ · ∥ : ℝ^{n} → ℝ satisfying the following conditions for all x, y ∈ ℝ^{n} and α ∈ ℝ.

1. ∥x∥ ≥ 0 (∥x∥ = 0 ⇔ x = 0);

2. ∥x + y∥ ≤ ∥x∥ + ∥y∥;

3. ∥αx∥ = |α|∥x∥.

Definition 1.16 For x ∈ ℝ^{n}, some of the most frequently used vector norms are • 1-norm:

∥x∥_{1} = ∑_{i=1}^{n} |x_{i}| = |x_{1}| + |x_{2}| + … + |x_{n}|.  (1.13)

• 2-norm:

∥x∥_{2} = \sqrt{∑_{i=1}^{n} |x_{i}|^{2}} = (|x_{1}|^{2} + |x_{2}|^{2} + … + |x_{n}|^{2})^{1/2} = \sqrt{x^{T} x}.  (1.14)

• ∞-norm:

∥x∥_{∞} = max_{1≤i≤n} |x_{i}|.  (1.15)

• p-norm:

∥x∥_{p} = (∑_{i=1}^{n} |x_{i}|^{p})^{1/p} = (|x_{1}|^{p} + |x_{2}|^{p} + … + |x_{n}|^{p})^{1/p} .  (1.16)

Definition 1.17 (unit vector) x ∈ ℝ^{n} is called a unit vector if ∥x∥ = 1 with respect to some vector norm.

Property 1.2 For any x, y ∈ ℝ^{n}, the following two inequalities hold.

• Hölder inequality:

|x^{T} y| ≤ ∥x∥_{p}∥y∥_{q}, where \frac{1}{p} + \frac{1}{q} = 1.

• Cauchy-Schwartz inequality:

|x^{T} y| ≤ ∥x∥_{2}∥y∥_{2}.

Definition 1.18 Two vector norms ∥ · ∥_{α} and ∥ · ∥_{β} are equivalent if there exist constants c_{1}, c_{2} ∈ ℝ such that

c_{1}∥x∥_{α} ≤ ∥x∥_{β} ≤ c_{2}∥x∥_{α}

for any x ∈ ℝ^{n}.

In fact, all vector norms on ℝ^{n} are equivalent, and it is easy to show the following equivalence properties for vector norms.

Property 1.3 For all x ∈ ℝ^{n},

∥x∥_{2} ≤ ∥x∥_{1} ≤ \sqrt{n}∥x∥_{2}.  (1.17)
∥x∥_{∞} ≤ ∥x∥_{2} ≤\sqrt{n}∥x∥_{∞}.  (1.18)
∥x∥_{∞} ≤ ∥x∥_{1} ≤ n∥x∥_{∞}.  (1.19)

Definition 1.19 (absolute error and relative error) Suppose x ∈ ℝ^{n} is the exact solution of some problem and x̃ is an approximation to x. We define

absolute error = ∥x − x̃∥  (1.20)

and

relative error = \frac{∥x − x̃∥}{∥x∥}, if x ≠ 0.  (1.21)

Definition 1.20 (convergence) Suppose {x_{k}} is a sequence of vectors in ℝ^{n}. We say x_{k} converges to x^{⋆}

x_{k} ⟶ x^{⋆} iff lim_{k→∞} ∥x_{k} − x^{⋆}∥ = 0,

for some vector norm.

1.3.2 Matrix Norm Definition and Properties

Definition 1.21 A matrix norm is a function ∥ · ∥ : ℝ^{m×n} → ℝ satisfying the following conditions for all A, B ∈ ℝ^{m×n} and α ∈ ℝ.

1. ∥A∥ ≥ 0 (∥A∥ = 0 ⇔ A = 0);

2. ∥A + B∥ ≤ ∥A∥ + ∥B∥;

3. ∥αA∥ = |α|∥A∥.

Definition 1.22 Some of the most frequently used matrix norms are

• Frobenius norm:

∥A∥_{F} = \sqrt{∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|^{2}}.  (1.22)

• 2-norm:

∥A∥_{2} = max_{{x∈ℝ^{n} \atop x≠0}} \frac{∥Ax∥_{2}}{∥x∥_{2}} = max_{∥x∥_{2}=1} ∥Ax∥_{2}.  (1.23)

• 1-norm:

∥A∥_{1} = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|.  (1.24)

• ∞-norm:

∥A∥_{∞} = max_{1≤i≤m} ∑_{j=1}^{n} |a_{ij}|.  (1.25)

• p-norm:

∥A∥_{p} = max_{{x∈ℝ^{n} \atop x≠0}} \frac{∥Ax∥_{p}}{∥x∥_{p}} = max_{∥x∥_{p}=1} ∥Ax∥_{p}.  (1.26)

Remarks 1.6 Frobenius norm, 1-norm, and ∞-norm are easy to compute, but not the 2norm or general p-norm.

Theorem 1.14 Suppose A ∈ ℝ^{m×n}. Then there exists z ∈ ℝ^{n}, ∥z∥_{2} = 1, such that A^{T} Az = μ^{2}z, where μ = ∥A∥_{2}.

Proof: Let z ∈ ℝ^{n}, ∥z∥_{2} = 1, be a unit vector that satisfies ∥A∥_{2} = ∥Az∥_{2} = max_{∥x∥_{2}=1} ∥Ax∥_{2}.

Define

g(x) = \frac{1}{2} \frac{x^{T} A^{T} Ax}{x^{T} x} = \frac{1}{2} \frac{∥Ax∥_{2}^{2}}{∥x∥_{2}^{2}} = \frac{1}{2} (\frac{∥Ax∥_{2}}{∥x∥_{2}} )^{2} ,

for x ∈ ℝ^{n}. Then z is a maximizer of g(x) which impliese ▽g(z) = 0. Since

▽g(x) = \frac{(x^{T} x)(A^{T} Ax) − (x^{T} A^{T} Ax)(x)}{(x^{T} x)^{2}}.

Hence

▽g(z) = 0 ⇒ (z^{T} z)(A^{T} Az) − (z^{T} A^{T} Az)z = 0
⇒ ∥z∥_{2}^{2}(A^{T} Az) − ∥Az∥_{2}^{2}z = 0
⇒ A^{T} Az = ∥A∥_{2}^{2}z = μ^{2}z.

Remarks 1.7 ∥A∥_{2} is the square root of the largest eigenvalue of A^{T} A. When A is symmetric, ∥A∥_{2} is the absolute value of the largest eigenvalue in magnitude.

Definition 1.23 (submultiplicative property) A matrix norm ∥ · ∥ is said to have the submultiplicative property if for any matrices A ∈ ℝ^{m×p} and B ∈ ℝ^{p×n} such that

∥AB∥ ≤ ∥A∥∥B∥.

Not all matrix norms satisfy the submultiplicative property, e.g., define ∥A∥_{△} = max_{i,j} |a_{ij}|,

and let A = B = [ \begin{matrix}1 & 1 \\ 1 & 1 \end{matrix} ]. But we conly consider matrix norms that satisfy the submulti-

plicative property.

Property 1.4 p-norm satisfies the submultiplicative property.

∥Ax∥_{p} ≤ ∥A∥_{p}∥x∥_{p}  (1.27)
∥AB∥_{p} ≤ ∥A∥_{p}∥B∥_{p}  (1.28)

Property 1.5 All matrix norms satisfying the submultiplicative property are equivalent.

∥A∥_{2} ≤ ∥A∥_{F} ≤\sqrt{n}∥A∥_{2}.  (1.29)
\frac{1}{\sqrt{n}}∥A∥_{∞} ≤ ∥A∥_{2} ≤\sqrt{m}∥A∥_{∞}.  (1.30)
\frac{1}{\sqrt{m}}∥A∥_{1} ≤ ∥A∥_{2} ≤\sqrt{n}∥A∥_{1}.  (1.31)
max_{i,j} |a_{ij}| ≤ ∥A∥_{2} ≤\sqrt{mn} max_{i,j} |a_{ij}|.  (1.32)

Property 1.6 Vector 2-norm is invariant under orthogonal transformation, that is, if Q is an n-by-n orthogonal matrix, then

∥Qx∥_{2} = ∥x∥_{2}  ∀ x ∈ ℝ^{n}.  (1.33)

Proof: Since

∥Qx∥_{2}^{2} = (Qx)^{T} (Qx) = x^{T} Q^{T} Qx = x^{T} x = ∥x∥_{2}^{2}.

Property 1.7 Matrix 2-norm and Frobenius norm are invariant under orthogonal transformation, that is, if Q is an n-by-n orthogonal matrix, then

∥QA∥_{2} = ∥A∥_{2}  (1.34)
∥QA∥_{F} = ∥A∥_{F}  (1.35)

for all A ∈ ℝ^{n×m}.

Proof: Since

∥QA∥_{2} = max_{∥x∥_{2}=1} ∥QAx∥_{2} = max_{∥x∥_{2}=1} ∥Ax∥_{2} = ∥A∥_{2}.

Property 1.8 If A ∈ ℝ^{m×n}, then ∥A∥_{2} ≤ \sqrt{∥A∥_{1}∥A∥_{∞}}.

Proof: From Theorem 1.14, there exists a vector z ∈ ℝ^{n}, z ≠ 0, such that ∥A∥_{2}^{2}z = A^{T} Az. This implies

∥A∥_{2}^{2}∥z∥_{1} = ∥A^{T} Az∥_{1} ≤ ∥A^{T} ∥_{1}∥A∥_{1}∥z∥_{1} = ∥A∥_{∞}∥A∥_{1}∥z∥_{1}.

Hence

∥A∥_{2}^{2} ≤ ∥A∥_{∞}∥A∥_{1}.

Theorem 1.15 Suppose that A ∈ ℝ^{n×n} and ∥ · ∥ is a submultiplicative matrix norm. If ∥A∥ < 1, then I − A is nonsingular and

(I − A)^{−1} = ∑_{k=0}^{∞} A^{k}  (1.36)

with

∥(I − A)^{−1}∥ ≤ \frac{1}{1 − ∥A∥}.  (1.37)

Proof: Suppose I −A is singular. Then there exists an x ∈ ℝ^{n}, x ≠ 0 such that (I −A)x = 0. This implies

Ax = x ⟹ ∥Ax∥ = ∥x∥ ⟹ \frac{∥Ax∥}{∥x∥} = 1 ⟹ ∥A∥ = max_{y≠0} \frac{∥Ay∥}{∥y∥} ≥ 1.

This contradicts to the assumption ∥A∥ < 1. Hence I − A must be nonsingular. Moreover

(I − A) (∑_{k=0}^{N} A^{k}) = I − A^{N+1}.

Since ∥A∥ < 1 and ∥·∥ is a submultiplicative matrix norm, ∥A^{N}∥ ≤ ∥A∥^{N} and lim_{N→∞} A^{N} = 0. Therefore

(I − A) ( lim_{N→∞} ∑_{k=0}^{N} A^{k}) = I.

It follows that

(I − A)^{−1} = lim_{N→∞} ∑_{k=0}^{N} A^{k} = ∑_{k=0}^{∞} A^{k}

and

∥(I − A)^{−1}∥ = ∥∑_{k=0}^{∞} A^{k}∥ ≤ ∑_{k=0}^{∞} ∥A^{k}∥ ≤ ∑_{k=0}^{∞} ∥A∥^{k} = \frac{1}{1 − ∥A∥}.

Theorem 1.16 If A is nonsingular and ∥A^{−1}E∥ < 1, then A + E is nonsingular and

∥(A + E)^{−1} − A^{−1}∥ ≤ \frac{∥E∥∥A^{−1}∥^{2}}{1 − ∥A^{−1}E∥}.  (1.38)

Proof: Since A is nonsingular, A + E = A(I + A^{−1}E). Since ∥A^{−1}E∥ < 1 it follows from Theorem 1.15 that I + A^{−1}E is nonsingular and ∥(I + A^{−1}E)^{−1}∥ ≤ \frac{1}{1−∥A^{−1}E∥}. Hence A + E is nonsingular, (A + E)^{−1} = (I + A^{−1}E)^{−1}A^{−1}, and

∥(A + E)^{−1}∥ = ∥(I + A^{−1}E)^{−1}A^{−1}∥ ≤ ∥(I + A^{−1}E)^{−1}∥∥A^{−1}∥ ≤ \frac{∥A^{−1}∥}{1 − ∥A^{−1}E∥}.

Now

(A + E)^{−1} − A^{−1} = (I − A^{−1}(A + E)) (A + E)^{−1} = −A^{−1}E(A + E)^{−1},

and so by taking norms we find

∥(A + E)^{−1} − A^{−1}∥ ≤ ∥A^{−1}∥∥E∥∥(A + E)^{−1}∥ ≤ \frac{∥E∥∥A^{−1}∥^{2}}{1 − ∥A^{−1}E∥}.

1.4 SVD: The Singular Value Decomposition

The singular value decomposition (SVD) is a matrix factorization whose computation is a step in many algorithms. Many problems of linear algebra can be better understood by considering the SVD.

Theorem 1.17 (Existence of SVD) If A ∈ ℝ^{m×n}, then there exists orthogonal matrices

U = [u_{1}, u_{2}, …, u_{m}] ∈ ℝ^{m×m} and V = [v_{1}.v_{2}, …, v_{n}] ∈ ℝ^{n×n}

such that

A = UΣV^{T} ,  (1.39)

where

Σ = diag(σ_{1}, σ_{2}, …, σ_{p}), p = min(m, n),

with

σ_{1} ≥ σ_{2} ≥ … ≥ σ_{p} ≥ 0.

Proof: Let σ_{1} = ∥A∥_{2} ≠ 0. By the compactness arguement there exists v_{1} ∈ ℝ^{n}, ∥v_{1}∥_{2} = 1 such that ∥Av_{1}∥_{2} = σ_{1}. Let u_{1} = \frac{1}{σ_{1}} Av_{1}. Then Av_{1} = σ_{1}u_{1}, ∥u_{1}∥_{2} = 1, and u_{1}^{T} Av_{1} = σ_{1}.

Let V_{1} = [v_{1}, V̂_{1}] ∈ ℝ^{n×n} and U_{1} = [u_{1}, Û_{1}] ∈ ℝ^{m×m} be the orthogonal matrices using the extension of v_{1} and u_{1} to form orthonormal bases for ℝ^{n} and ℝ^{m}, respectively. Note that this can always be done by Gram-Schmidt process. Then

Û_{1}^{T} Av_{1} = Û_{1}^{T} (σ_{1}u_{1}) = σ_{1}(Û_{1}^{T} u_{1}) = 0

and

U_{1}^{T} AV_{1} = [ u_{1}^{T} ; Û_{1}^{T} ] A [v_{1}, V̂_{1}] = [ \begin{matrix}u_{1}^{T} Av_{1} & u_{1}^{T} AV̂_{1} \\ Û_{1}^{T} Av_{1} & Û_{1}^{T} AV̂_{1} \end{matrix} ] ≡ [ \begin{matrix}σ_{1} & w^{T} \\ 0 & B \end{matrix} ] ≡ A_{1}.

Next we show that w^{T} ≡ u_{1}^{T} AV̂_{1} = 0. Since

A_{1} [ \begin{matrix}σ \\ w \end{matrix} ] = [ \begin{matrix}σ & w^{T} \\ 0 & B \end{matrix} ] [ \begin{matrix}σ \\ w \end{matrix} ] = [ \begin{matrix}σ^{2} + w^{T} w \\ Bw \end{matrix} ] ,

hence

∥A_{1}∥_{2} = max \frac{∥A_{1}x∥_{2}}{∥x∥_{2}} ≥ \begin{matrix}∥A_{1} [ \begin{matrix}σ \\ w \end{matrix} ]∥ & 2 \\ ∥[ \begin{matrix}σ \\ w \end{matrix} ]∥ & 2 \end{matrix} = \begin{matrix}∥[ \begin{matrix}σ^{2} + w^{T} w \\ Bw \end{matrix} ]∥ & 2 \\ ∥[ \begin{matrix}σ \\ w \end{matrix} ]∥ & 2 \end{matrix}

= \frac{((σ_{1}^{2} + w^{T} w)^{2} + w^{T} B^{T} Bw)^{1/2}}{(σ_{1}^{2} + w^{T} w)^{1/2}}
≥ \frac{(σ_{1}^{2} + w^{T} w)}{(σ_{1}^{2} + w^{T} w)^{1/2}} = (σ_{1}^{2} + w^{T} w)^{1/2} ≥ σ_{1}.

But

∥A_{1}∥_{2} = ∥U_{1}^{T} AV_{1}∥_{2} = ∥A∥_{2} = σ_{1}.

Hence

w^{T} w = 0 ⇒ w = 0.

Therefore

U_{1}^{T} AV_{1} = [ \begin{matrix}σ_{1} & 0 \\ 0 & B \end{matrix} ] ⇒ A = U_{1} [ \begin{matrix}σ_{1} & 0 \\ 0 & B \end{matrix} ] V_{1}^{T} .

Using induction argument, there exist orthogonal matrices U_{2} ∈ ℝ^{(m−1)×(m−1)} and V_{2} ∈ ℝ^{(n−1)×(n−1)} such that B = U_{2}Σ_{2}V_{2}^{T}, where Σ_{2} ∈ ℝ^{(m−1)×(n−1)} is diagonal. Then

A = U_{1} [ \begin{matrix}σ_{1} & 0 \\ 0 & B \end{matrix} ] V_{1}^{T}
= U_{1} [ \begin{matrix}σ_{1} & 0 \\ 0 & U_{2}Σ_{2}V_{2}^{T} \end{matrix} ] V_{1}^{T}
= U_{1} [ \begin{matrix}1 & 0 \\ 0 & U_{2} \end{matrix} ] [ \begin{matrix}σ_{1} & 0 \\ 0 & Σ_{2} \end{matrix} ] [ \begin{matrix}1 & 0 \\ 0 & V_{2} \end{matrix} ]^{T} V_{1}^{T}
≡ UΣV^{T} ,

where

U = U_{1} [ \begin{matrix}1 & 0 \\ 0 & U_{2} \end{matrix} ] , V = V_{1} [ \begin{matrix}1 & 0 \\ 0 & V_{2} \end{matrix} ] , Σ = [ \begin{matrix}σ_{1} & 0 \\ 0 & Σ_{2} \end{matrix} ] .

This proves the theorem.

The σ_{i} are called the singular values of A and the vectors u_{i} and v_{i} the i-th left singular vector and the i-th right singular vector, respectively. We usually use σ_{max}(A) to denote the largest singular value of A and σ_{min}(A) the smallest singular value of A.

Suppose A ∈ ℝ^{m×n} and rank(A) = r. Then

σ_{1} ≥ σ_{2} ≥ ⋯ ≥ σ_{r} > σ_{r+1} = σ_{p} = 0,

and A can be expressed as the summation of r rank-one matrices

A = ∑_{i=1}^{r} σ_{i}u_{i}v_{i}^{T} .  (1.40)

The SVD gives complete information about the four fundamental subspaces associated with A:

R(A) = span {u_{1}, u_{2}, …, u_{r}} ,  (1.41)
N (A) = span {v_{r+1}, v_{r+2}, …, v_{n}} ,  (1.42)
R(A^{T}) = span {v_{1}, v_{2}, …, v_{r}} ,  (1.43)
N (A^{T}) = span {u_{r+1}, u_{r+2}, …, u_{m}} .  (1.44)

The matrix 2-norm and Frobenius norm can be characterized in terms of singular values.

∥A∥_{2} = σ_{1},  (1.45)
∥A∥_{F} = \sqrt{σ_{1}^{2} + ⋯ + σ_{r}^{2}}.  (1.46)

If we write

A = UΣV^{T} = U [ \begin{matrix}Σ_{r} & 0 \\ 0 & 0 \end{matrix} ] V^{T} ,

where

Σ_{r} = [ \begin{matrix}σ_{1} \\ & . & . & . \\ & & & & σ_{r} \end{matrix} ] ∈ ℝ^{r×r},

then

A^{T} A = V Σ^{T} U^{T} UΣV^{T} = V Σ^{T} ΣV^{T} = V [ \begin{matrix}Σ_{r}^{2} & 0 \\ 0 & 0 \end{matrix} ] V^{T}

and

AA^{T} = UΣV^{T} V Σ^{T} U^{T} = UΣΣ^{T} U^{T} = U [ \begin{matrix}Σ_{r}^{2} & 0 \\ 0 & 0 \end{matrix} ] U^{T} .

Thus σ_{1}^{2}, …, σ_{r}^{2} are the nonzero eigenvalues of A^{T} A and AA^{T}, and v_{j} and u_{j} are the corresponding eigenvectors. When A is symmetric, A^{T} A = AA^{T} = A^{2}, and has real eigenvalues λ_{i}, then the singular values are given by σ_{i} = |λ_{i}| for i = 1, …, r. In principal, the SVD can be found from the eigenvalue decomposition of A^{T} A and AA^{T}. However, this does not lead to a stable algorithm for computing the SVD. Finally, when A is n × n and square, then | det(A)| = σ_{1} ⋯ σ_{n}.

20  Chapter 1. Mathematical Preliminaries

Computer Arithmetic

In this chapter, we explain the floating-point number system and develop basic facts about roundoff errors. Some topics such as loss of significance, stability of numerical algorithms, and condition of problems will be introduced with examples. Error analysis of computer arithmetic will be brifely discussed.

2.1 Floating-Point Number and Roundoff Error

In general, a nonzero real number x in the decimal number system can be written as

x = ±r × 10^{n},

where

\frac{1}{10} ≤ r < 1,

and n is an integer (positive, negative, or zero). This representation is the called normalized scientific notation, r is called the mantissa and n is the exponent. Note that the leading digit in the fraction is not zero (except when the number involved is zero). For example,

42.965 = 4 × 10^{1} + 2 × 10^{0} + 9 × 10^{−1} + 6 × 10^{−2} + 5 × 10^{−3}
= 0.42965 × 10^{2},
−0.00234 = −0.234 × 10^{−2}.

Likewise, we can use the scientific notation for binary number system to express the number x as

x = ±q × 2^{m}

with

\frac{1}{2} ≤ q < 1,

and some integer m. Both q and m will be expressed in terms of binary numbers. For example,

1001.1101 = 1 × 2^{3} + 1 × 2^{0} + 1 × 2^{−1} + 1 × 2^{−2} + 1 × 2^{−4}
= 0.10011101 × 2^{100}
= (9.8125)_{10}

Allmost all microcomputers uses binary number system, but not necessary mainframe computers.

Example 2.1 What is the binary representation of \frac{2}{3}?

Sol: To determine the binary representation for \frac{2}{3}, we write

\frac{2}{3} = (0.a_{1}a_{2}a_{3} …)_{2}.

Multiply by 2 to obtain

\frac{4}{3} = (a_{1}.a_{2}a_{3} …)_{2}.

Therefore, we get a_{1} = 1 by taking the integer part of both sides. Substracting 1, we have

\frac{1}{3} = (0.a_{2}a_{3}a_{4} …)_{2}.

Repeating the previous step, we arrive at

\frac{2}{3} = (0.101010 …)_{2}.

Most computers work internally in binary system and output decimal system for human users. A number that has a terminating exapnsion in one base may have a nonterminating exapnsion in another. The conversion procedure frequently involves a small error as shown in the following example.

Example 2.2 What is the binary representation of \frac{1}{10}?

Sol:

\frac{1}{10} = (0.1)_{10} = (0.0001 1001 1001 1001 …)_{2}.

Typically, only a relatively small subset of the real number system is used for the representation of all the real numbers. This subset, which are called the floating-point numbers, contains only rational numbers, both positive and negative. When a number can not be represented exactly with the fixed finite number of digits in a computer, a near-by floating-point number is chosed for approximate representation.

For any real number x, let

x = ±0.a_{1}a_{2} ⋯ a_{t}a_{t+1}a_{t+2} ⋯ × 2^{m}, a_{1} ≠ 0,  (2.1)

denote the normalized scientific binary representation of x. Note that the leading digit in mantissa a_{1} ≠ 0, hence a_{1} = 1. If x is within the numerical range of the machine, the floating-point form of x, denoted fl(x), is obtained by terminating the mantissa of x at t digits for some integer t. There are two ways of preforming this termination.

1. chopping: simply discard the excess bits a_{t+1}, a_{t+2}, … to obtain

fl(x) = ±0.a_{1}a_{2} ⋯ a_{t} × 2^{m}.  (2.2)

2. rounding up: add 2^{−(t+1)} × 2^{m} to x and then chop the excess bits to obtain a number

of the form

fl(x) = ±0.δ_{1}δ_{2} ⋯ δ_{t} × 2^{m}.  (2.3)

In this method, if a_{t+1} = 1, we add 1 to a_{t} to obtain fl(x), and if a_{t+1} = 0, we merely chop off all but the first t digits.

Since computers can only store real numbers using fixed finite numbers of digits, this places a restriction on the precision with which real numbers can be represented. The error results from replacing a number with its floating-point form is called roundoff error or rounding error, regardless of whether the rounding or chopping method is used. The following definition describes two methods for measuring approximation errors.

Definition 2.1 (Absolute Error and Relative Error) If x is an approximation to the exact value x^{⋆}, the absolute error is |x^{⋆} − x| and the relative error is \frac{|x^{⋆}−x|}{|x^{⋆}|}, provided that x^{⋆} ≠ 0.

Remark 2.1 As a measure of accuracy, the absolute error may be misleading and the relative error more meaningful.

If the floating-point representation fl(x) for the number x is obtained by using t digits and chopping procedure, then the relative error is

\frac{|x − fl(x)|}{|x|} = \frac{|0.00 ⋯ 0a_{t+1}a_{t+2} ⋯ × 2^{m}|}{|0.a_{1}a_{2} ⋯ a_{t}a_{t+1}a_{t+2} ⋯ × 2^{m}|} = \frac{|0.a_{t+1}a_{t+2} ⋯ |}{|0.a_{1}a_{2} ⋯ a_{t}a_{t+1}a_{t+2} ⋯ |} × 2^{−t}.

Since a_{1} ≠ 0, the minimal value of the denominator is \frac{1}{2}. The numerator is bounded above by 1. As a consequence

|\frac{x − fl(x)}{x}| ≤ 2^{−t+1}.  (2.4)

In a similar manner, if t-digit rounding arithmetic is used and a_{t+1} = 0, then fl(x) = ±0.a_{1}a_{2} ⋯ a_{t} × 2^{m}. A bound for the relative error is

\frac{|x − fl(x)|}{|x|} = \frac{|0.a_{t+1}a_{t+2} ⋯ |}{|0.a_{1}a_{2} ⋯ a_{t}a_{t+1}a_{t+2} ⋯ |} × 2^{−t} ≤ 2^{−t}.

since the numerator is bounded above by \frac{1}{2}. But if a_{t+1} = 1, then fl(x) = ±(0.a_{1}a_{2} ⋯ a_{t} + 2^{−t}) × 2^{m}, the upper bound for relative error becomes

\frac{|x − fl(x)|}{|x|} = \frac{|1 − 0.a_{t+1}a_{t+2} ⋯ |}{|0.a_{1}a_{2} ⋯ a_{t}a_{t+1}a_{t+2} ⋯ |} × 2^{−t} ≤ 2^{−t}.

Figure 2.1: 32-bit single precision.

since the numerator is bounded by \frac{1}{2} due to a_{t+1} = 1. Therefore the relative error for rounding arithmetic is

|\frac{x − fl(x)}{x}| ≤ 2^{−t} = \frac{1}{2} × 2^{−t+1}.  (2.5)

The number ϵ_{M} ≡ 2^{−t+1} is referred to as the unit roundoff error or machine epsilon. The floating-point representation, fl(x), of x can be expressed as

fl(x) = x(1 + δ),  |δ| ≤ ε_{M}.  (2.6)

In 1985, the IEEE (Institute for Electrical and Electronic Engineers) published a report called Binary Floating Point Arithmetic Standard 754-1985. In this report, formats were specified for single, double, and extended precisions, and these standards are generally followed by microcomputer manufactures using floating-point hardware.

The single precision IEEE standard floating-point format allocates 32 bits for the normalized floating-point number ±q × 2^{m} as shown in Figure 2.1. The first bit is a sign indicator, denoted s. This is followed by an 8-bit exponent c and a 23-bit mantissa f. The base for the exponent and mantissa is 2, and the actual exponent is c − 127. The value of c is restricted by the inequality 0 < c < 255. The values 0 and 255 are reserved for special cases including ±0 and ±∞, respectively. Hence the actual exponent of the number is restricted by the inequality −126 ≤ c − 127 ≤ 127. In addition, a normalization is imposed that requires that the leading digit in fraction be 1, and this digit is not stored as part of the 23-bit mantissa. Using this format gives a floating-point number of the form

(−1)^{s} × (1.f)_{2} × 2^{c−127}.

Since the mantissa f actually corresponds to 24 binary digits (i.e., precision t = 24), the machine epsilon is

ϵ_{M} = 2^{−24+1} = 2^{−23} ≈ 1.192 × 10^{−7}.  (2.7)

This approximately corresponds to 6 accurate decimal digits. And the first single precision floating-point number greater than 1 is 1 + 2^{−23}.

The largest number that can be represented by the single precision format is approximately 2^{128} ≈ 3.403 × 10^{38}, and the smallest positive number is 2^{−126} ≈ 1.175 × 10^{−38}. This range is in general not sufficient for some scientific calculation. For this and other reason, we need “double precision” arithmetic.

A floating point number in double precision IEEE standard format uses two words (64 bits) to store the number as shown in Figure 2.2. The first bit is a sign indicator, denoted Figure 2.2: 64-bit double precision.

single precision  double precision
ϵ_{M}  2^{−23} ≈ 1.192 × 10^{−7} 2^{−52} ≈ 2.220 × 10^{−16}
smallest positive number 2^{−126} ≈ 1.175 × 10^{−38} 2^{−1022} ≈ 2.225 × 10^{−308}
largest number  2^{128} ≈ 3.403 × 10^{38}  2^{1024} ≈ 1.798 × 10^{308}
decimal precision  6  15

Table 2.1: Some characteristics of IEEE standard floating-point numbers

s, same as the single precision format. This is followed by an 11-bit exponent c and a 52-bit mantissa f. The actual exponent is c−1023. Using this format gives a floating-point number of the form

(−1)^{s} × (1.f)_{2} × 2^{c−1023}.

which provides between 15 and 16 decimal digits of accruacy, since the machine epsilon

ϵ_{M} = 2^{−52} ≈ 2.220 × 10^{−16},

and a range of approximately 2^{−1022} ≈ 2.225 × 10^{−308} to 2^{1024} ≈ 1.798 × 10^{308}. The first double precision floating-point number greater than 1 is 1 + 2^{−52}.

Table 2.1 summarizes some characteristics of IEEE standard floating-point representations. Clearly, for the most accuracy, computations should be done using double precision floating-point numbers, however, the execution time is much higher.

In the IEEE floating-point standard, the round to nearest or correctly rounded value of the real number x, denoted round(x), is defined as follows. First, let x_{+} be the closest foating-point number greater than x and x_{−} be the closest one less than x. Then round(x) is either x_{+} or x_{−}, whichever is nearer to x. If x is a floating-point number, then round(x) = x. This method is almost always used since it is the most useful and gives the floating-point number closest to x. Other rounding modes could be (1) round towards 0 : round(x) is either x_{+} or x_{−}, whichever is between 0 and x; (2) round towards −∞/round dwon: round(x) = x_{−};

(3) round towards ∞/round up: round(x) = x_{+}.

If a number x = ±q × 2^{m} with m outside the computer’s possible range (too large or two small), then we say that an overflow or an underflow has occurred. Generally, an overflow results in a fatal error (or exception), and the normal excution of the program stops. An underflow, however, is usually treated automatically by setting x to zero without any interruption of the program but with a warning message in most computers.

There are some useful special numbers in the IEEE standard. For example, instead of terminating with an overflow when dividing a nonzero number by 0, the machine representation for ∞, Inf, is stored, which is the mathematically sensible thing to do. There are two different representations, +Inf and −Inf, that correspond to two quite different numbers, +∞ and −∞. A NaN stands for Not a Number and is an error pattern rather than a number. Table 2.2 lists the IEEE exception handling standard.

big*big  ± Inf  overflow
number/0.0 ± Inf  division
0.0/0.0 NaN invalid
small/big subnormal number underflow
2.0/3.0  rounded

Table 2.2: IEEE exception handling.

Because of the hidden bit representation, a special technique for storing zero is necessary. Note that all zero bits in the mantissa field represents the significant 1.0 rather than 0.0. Moreover, there are two different representations for the same number zero; namely +0 and −0.

For single precision integers, 31 bits are allocated for the integer, because onley 1 bit is needed for the sign. Consequently, the range for integers is from −(2^{31} − 1) to (2^{31} − 1) = 2147483647. Double precision integers, on the other hand, use 63-bit storage and 1 bit for sign. Pure integer calculation is not common in numerical algorithms.

2.2 Loss of Significance, Stability, and Conditioning

Roundoff errors are inevitable and difficult to control. Other types of errors which occur in computation may be under our control. The subject of numerical analysis is largely preoccupied with understanding and controlling errors of various kinds. Here in this section we examine some of them.

2.2.1 Loss of Significance

Suppose that x is a real number expressed in normalized scientific notation in the decimal system

x = ±a_{1}a_{2}a_{3} ⋯ × 10^{n} a_{1} ≠ 0.

The digits a_{1}, a_{2}, a_{3}, …, used to express the fraction part of x do not all have the same significance because they represent different power of 10. Thus, we say that a_{1} is the most significant digit, and the significance of the digits diminishes from left to right.

One of the most common error-producing calculations involves the cancellation of significant digits due to the subtraction of nearly equal numbers (or the addition of one very large number and one very small number). Assume that two nearly equal numbers x and y, with x > y, have the t-digit floating-point representations

fl(x) = 0.d_{1}d_{2} ⋯ d_{p}α_{p+1}α_{p+2} ⋯ α_{t} × 10^{n},

and

fl(x) = 0.d_{1}d_{2} ⋯ d_{p}β_{p+1}β_{p+2} ⋯ β_{t} × 10^{n}.

Then the floating-point form of x − y is

fl(fl(x) − fl(y)) = 0.σ_{p+1}σ_{p+2} ⋯ σ_{t} × 10^{n},

where

0.σ_{p+1}σ_{p+2} ⋯ σ_{t} = 0.α_{p+1}α_{p+2} ⋯ α_{t} − 0.β_{p+1}β_{p+2} ⋯ β_{t}.

As a result, the floating-point number used to represent x − y has at most t − p digits of significance. However, in most computers, x−y will be assigned t digits, with the last p digits being either zero or randomly assigned. Any further calculations involving the results of x−y retain the problem of having only t − p digits of significance, since a chain of calculations can not be expected to be more accurate than its weakest portion. The phenomenon can be illustrated with the following example.

Example 2.3 If x = 0.3721478693 and y = 0.3720230572, what is the relative error in the computation of x − y using five decimal digits of accuracy?

Sol: In exact computation using ten decimal digits of accuracy,

x − y = 0.0001248121.

But both x and y will be rounded to five decimal digits before subtraction. Thus

fl(x) = 0.37215
fl(y) = 0.37202
fl(x) − fl(y) = 0.00013 = 0.13000 × 10^{−3}

Therefore the relative error is

\frac{(x − y) − (fl(x) − fl(y))}{x − y} ≈ 0.04 = 4%.

An interesting question arised immediately would be how many significant binary bits are lost in the subtraction when x is close to y?

Theorem 2.1 If x ≥ 0 and y ≥ 0 are normalized floating-point binary numbers such that x > y and

2^{−q} ≤ 1 − \frac{y}{x} ≤ 2^{−p},

then at most q and at least p significant binary digits are lost in the subtraction x − y.

Proof: Write

x = r × 2^{n}, \frac{1}{2} ≤ r < 1

and

y = s × 2^{m}, \frac{1}{2} ≤ s < 1.

Since x > y, we must sift the decimal digits of y to the right

y = (s × 2^{m−n}) × 2^{n}.

Then

x − y = (r − s × 2^{m−n}) × 2^{n} = r (1 − \frac{s × 2^{m}}{r × 2^{n}} ) × 2^{n} = r (1 − \frac{y}{x}) × 2^{n}.

By assumption 2^{−q} ≤ 1 − \frac{y}{x} ≤ 2^{−p}, hence

r (1 − \frac{y}{x}) < 1 · 2^{−p} = 2^{−p}.

This means that to normalize the result x − y, a shift of at least p bits to the left is required. Similarly,

r (1 − \frac{y}{x}) ≥ \frac{1}{2} · 2^{−q} = 2^{−(q+1)},

and a shift of at most q bits to the right is required.

Sometimes, loss of significance can be avoided by using double precision. This at least doubles the number of bits in mantissa. But when both operands are already in double precision, this means quadruple precision would be needed, and this is not available in many machines and languages, or at least not cheaply. In some cases, rewriting the mathematical formula is an alternative and can make the difference.

Example 2.4 Consider the two equivalent functions

f(x) = x(\sqrt{x + 1} − \sqrt{x}) and g(x) = \frac{x}{\sqrt{x + 1} + \sqrt{x}}.

Compare the function evaluation of f(500) and g(500) using 6 digits and rounding.

Sol:

f(500) = 0.500000 × 10^{3} × (\sqrt{501} − \sqrt{500})
= 0.500000 × 10^{3} × (0.223830 × 10^{2} − 0.223607 × 10^{2})
= 0.500000 × 10^{3} × 0.223000
= 0.111500 × 10^{3}

and

g(500) = \frac{500}{\sqrt{501} + \sqrt{500}}
= \frac{0.500000 × 10^{3}}{0.223830 × 10^{2} + 0.223607 × 10^{2}}
= \frac{0.500000 × 10^{3}}{0.447437 × 10^{2}}
= 0.111748 × 10^{2}

If more digits are used, we can calculated

f(500) = 500 × (\sqrt{501} − \sqrt{500})
= 500 × (22.38302929 − 22.36067977)
= 500 × 0.022349516
= 11.1747553

Hence it can be argued that the formulation g(x) is better.

Example 2.5 The quadratic formulas for computing the roots of ax^{2} + bx + c = 0, when a ≠ 0, are

x_{1} = \frac{−b + \sqrt{b^{2} − 4ac}}{2a}  and  x_{2} = \frac{−b − \sqrt{b^{2} − 4ac}}{2a}.

Consider the quadratic equation x^{2} + 62.10x + 1 = 0 and discuss the numerical results.

Sol: Using the quadratic formula and 8-digit rounding arithmetic, one can obtain the roots

x_{1} = −0.01610723 and x_{2} = −62.08390.

We will use these values as “exact solutions”. Now we perform the calculations with 4-digit rounding arithmethic. First we have

\sqrt{b^{2} − 4ac} = \sqrt{62.10^{2} − 4.000} = \sqrt{3856 − 4.000} = \sqrt{3852} = 62.06,

and

fl(x_{1}) = \frac{−62.10 + 62.06}{2.000} = \frac{−0.04000}{2.000} = −0.02000.

The relative error in computing x_{1} is

\frac{|fl(x_{1}) − x_{1}|}{|x_{1}|} = \frac{| − 0.02000 + 0.01610723|}{| − 0.01610723|} = \frac{0.00389277}{0.01610723} ≈ 0.2417.

In calculating x_{2},

fl(x_{2}) = \frac{−62.1062.06}{2.000} = \frac{−124.2}{2.000} = −62.10,

and the relative error in computing x_{2} is

\frac{|fl(x_{2}) − x_{2}|}{|x_{2}|} = \frac{| − 62.10 + 62.08390|}{| − 62.08390|} = \frac{0.0161}{62.08390} ≈ 0.259 × 10^{−3}.

In this equation, b^{2} = 62.10^{2} is much larger than 4ac = 4. Hence b and \sqrt{b^{2} − 4ac} become two nearly equal numbers. The calculation of x_{1} involves the subtraction of two nearly equal numbers, in contrast, the calculation of x_{2} involves the addition of the nearly equal numbers which will not cause serious loss of significance.

To obtain a more accurate 4-digit rounding approximation for x_{1}, we change the formulation by rationalizing the numerator, that is,

x_{1} = \frac{−2c}{b + \sqrt{b^{2} − 4ac}}.

Then

fl(x_{1}) = \frac{−2.000}{62.10 + 62.06} = \frac{−2.000}{124.2} = −0.01610.

The relative error in computing x_{1} is now reduced to 0.62 × 10^{−3}. However, if we rationalize the numberator in x_{2} to get

x_{2} = \frac{−2c}{b − \sqrt{b^{2} − 4ac}}.

The use of this formula results not only involve the subtraction of two nearly equal numbers but also the division by the small number. This would cause degrade in accuracy.

fl(x_{2}) = \frac{−2.000}{62.10 − 62.06} = \frac{−2.000}{0.04000} = −50.00.

The relative error in x_{2} becomes 0.19.

The above example also shows that if a finite-precision representation or calculation introduces an error, further enlargement of the error occurs when dividing by a number with small magnitude, or, equivalently, when multiplying by a number with large magnitude.

There is another situation in which a drastic loss of significant digits will occur. This is in the evaluation of polynomials and certain functions.

Example 2.6 Evaluate

f(x) = x^{3} − 6x^{2} + 3x − 0.149

at x = 4.71 using 3-digit arithmetic.

Sol: The following table gives the intermediate results in the calculation.

x  x^{2}  x^{3}  6x^{2}  3x
Exact  4.71 22.1841 104.487111 133.1046 14.13
3-digit chopping 4.71 22.1  104  132  14.1
3-digit rounding 4.71 22.2  105  133  14.1

The exact function value is

f(4.71) = −14.636489.

The result from 3-digit chopping arithmetic is

f̂(4.71) = −14.0,

and 3-digit rounding

f̃(4.71) = −14.0.

Hence relative error for both calculation is

\frac{| − 14.636489 + 14.0|}{| − 14.636489|} ≈ 0.4 × 10^{−1}.

Alternatively, if we rewrite the formulation of f(x) in nested manner as

f(x) = ((x − 6)x + 3)x − 0.149.

2.2 Loss of Significance, Stability, and Conditioning  31

Then the 3-digit chopping arithmetic gives

f̂(4.17) = −14.5

with relative error

\frac{| − 14.636489 + 14.5|}{| − 14.636489|} ≈ 0.93 × 10^{−2},

and 3-digit rounding

f̃(4.17) = −14.6

with relative error

\frac{| − 14.636489 + 14.6|}{| − 14.636489|} ≈ 0.25 × 10^{−2}.

Example 2.7 Let

p(x) = ((x^{3} − 3x^{2}) + 3x) − 1,
q(x) = ((x − 3)x + 3)x − 1.

Compare the function values at x = 2.19. Sol: Use 3-digit and rounding for p(2.19) and q(2.19).

p̂(2.19) = ((2.19^{3} − 3 × 2.19^{2}) + 3 × 2.19) − 1
= ((10.5 − 14.4) + 3 × 2.19) − 1
= (−3.9 + 6.57) − 1
= 2.67 − 1
= 1.67
q̂(2.19) = ((2.19 − 3) × 2.19 + 3) × 2.19 − 1
= (−0.81 × 2.19 + 3) × 2.19 − 1
= (−1.77 + 3) × 2.19 − 1
= 1.23 × 2.19 − 1
= 2.69 − 1
= 1.69

With more digits, one can have

p(2.19) = g(2.19) = 1.685159

Hence the absolute errors are |p(2.19) − p̂(2.19)| = 0.015159 and

|q(2.19) − q̂(2.19)| = 0.004841,

respectively. One can observe that the evaluation formula q(x) is better than p(x).

In fact, polynomials should always be expressed in nested form before performing an evaluation, since this form minimizes the number of required arithmetic calculations. One way to reduce roundoff error is to reduce the number of error-producing computations.

Example 2.8 How to evaluate

y = x − sin x

when x is small?

Sol: Since x ≈ sin x for small x, the computation will cause loss of significance. Alternatively, use Taylor series for sin x so that

y = x − (x − \frac{x^{3}}{3!} + \frac{x^{5}}{5!} − \frac{x^{7}}{7!} + \frac{x^{9}}{9!} − ⋯)
= \frac{x^{3}}{3!} − \frac{x^{5}}{5!} + \frac{x^{7}}{7!} − \frac{x^{9}}{9!} + ⋯
= \frac{x^{3}}{6} − \frac{x^{5}}{6 × 20} + \frac{x^{7}}{6 × 20 × 42} − \frac{x^{9}}{6 × 20 × 42 × 72} ⋯
= \frac{x^{3}}{6} (1 − \frac{x^{2}}{20} (1 − \frac{x^{2}}{42} (1 − \frac{x^{2}}{72} (⋯))))

Finally, floating-point arithmetic is not always associative. This can be illustrated with the following example.

Example 2.9 Performing the calculation with 3-decimal mantissa and chopping arithmetic will result

fl(fl(10^{−3} + 1) − 1) = 0

and

fl(fl(10^{−3} + fl(1 − 1))) = 10^{−3}.

2.2.2 Numerical Stability

Another theme that occurs repeatedly in numerical analysis is the distiction between numerical algorithms that are stable and those that are not. Informally speaking, a numerical process is unstable if small errors made at one stage of the process are magnified and propagated in subsequent stages and seriously degrade the accuracy of the overall calculation. Whether a process is stable or unstable should be decided on the basis of relative error.

Example 2.10 Consider the following recurrence algorithm

\{ \begin{matrix}x_{0} = 1,  x_{1} = \frac{1}{3} \\ x_{n+1} = \frac{13}{3}x_{n} − \frac{4}{3}x_{n−1} \end{matrix}

for computing the sequence of \{x_{n} = (\frac{1}{3})^{n}\}. This algorithm is unstable.

Sol: A computer implemention of the recurrence algorithm gives the following result.

\begin{matrix}n & x_{n} & n & x_{n} & n & x_{n} & n & x_{n} \\ 0 & 1.0000000 & 4 & 0.0123466 & 8 & 0.0003757 & 12 & 0.0571502 \\ 1 & 0.3333333 & 5 & 0.0041187 & 9 & 0.0009437 & 13 & 0.2285939 \\ 2 & 0.1111112 & 6 & 0.0013857 & 10 & 0.0035887 & 14 & 0.9143735 \\ 3 & 0.0370373 & 7 & 0.0005153 & 11 & 0.0142927 & 15 & 3.6574934 \end{matrix}

The error present in x_{n} is multiplied by \frac{13}{3} in computing x_{n+1}. For example, the error will be propagated with a factor of (\frac{13}{3})^{14} in computing x_{15}. Additional roundoff errors in computing x_{2}, x_{3}, … may also be propagated and added to that of x_{15}.

2.2.3 Conditioning

The words condition and conditioning are used informally to indicate how sensitive the solution of a problem may be to small relative changes in the input data. A problem is ill-conditioned if small changes in the data can produce large changes in the resluts. For certain types of problems, a condition number can be defined. If that number is large, it indicates an ill-conditioned problem. In contrast, if the number is modest, the problem is recognized as a well-conditioned problem.

For example, the condition number for a nonsingular square matrix A is defined as

κ(A) = ∥A∥∥A^{−1}∥,

with respect to some matrix norm. The matrix A is said to be ill-conditioned if κ(A) is large, and well-conditioned when κ(A) is modest. For a general rectangular matrix, the singular values are used to characterize the condition number in 2-norm

κ_{2}(A) = \frac{σ_{max}(A)}{σ_{min}(A)},

where σ_{max}(A) is the largest singular value and σ_{min}(A) the smallest singular value of A. A well-known ill-conditioned matrix is the Hilbert matrix

H_{n} = [h_{ij}] ∈ ℝ^{n×n}, where h_{ij} = \frac{1}{i + j − 1}.

In general, ill-conditioning is not easy to detect.

In solving a system of linear equations Ax = b in which A is ill-conditioned, small perturbation in b will cause large perturbation in x.

Example 2.11 Consider solving the following system of linear equations

[ \begin{matrix}20514 & 4424 & 987 & 224 \\ 4424 & 987 & 224 & 54 \\ 987 & 224 & 54 & 14 \\ 224 & 54 & 14 & 4 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}20514 \\ 4424 \\ 987 \\ 224 \end{matrix} ] .

Sol: The numerical solution obtained by using Matlab command A\b, where A is the coefficient matrix and b is the right-hand-side vector, is

x = [ \begin{matrix}1.00000000000002 \\ −0.00000000000003 \\ −0.00000000000055 \\ 0.00000000000125 \end{matrix} ] .

However, if b_{1} is changed from 20514 to 20515, then the numerical solution becomes

x = [ \begin{matrix}0.99672906602257 \\ −0.00528381642514 \\ 0.16978663445968 \\ −0.33974939613341 \end{matrix} ] .

One may check that the 2-norm condition number of the coefficient is κ_{2}(A) ≈ 6.1191 × 10^{5}.

2.3 Floating-Point Error Analysis

In addition to inaccurate representation of numbers, the arithmetic performed in a computer is not exact. The arithmetic involves manipulating binary digits by various shifting, or logical, operations. We now consider the errors that are produced in the course of elementary arithmetic operations.

Let ⊙ stand for any one of the four basic arithmetic operators +, −, ⋆, ÷. Whenever two machine numbers x and y are to be combined arithmetically, the computer will produce fl(x ⊙ y) instead of x ⊙ y. In the operation, x ⊙ y is first correctly formed, normalized, and then rounded to become a machine number. Under this assumption and (2.6), the relative error of fl(x ⊙ y) satisfies

fl(x ⊙ y) = (x ⊙ y)(1 + δ), δ ≤ ϵ_{M},  (2.8)

where ϵ_{M} is the unit roundoff. But if x, y are not machine numbers, then they must first rounded to floating-point format before the arithmetic operation and the resulting relative error becomes

fl(fl(x) ⊙ fl(y)) = (x(1 + δ_{1}) ⊙ y(1 + δ_{2}))(1 + δ_{3}), δ_{i} ≤ ϵ_{M}, i = 1, 2, 3.  (2.9)

Thus, the machine unit roundoff gives an upper bound for the relative error in any single basic arithmetic operation.

The analysis (2.8) can be extended to arithmetic operations on three floating-point numbers. For example,

fl(x(y + z)) = (x · fl(y + z))(1 + δ_{1})
= (x(y + z)(1 + δ_{2}))(1 + δ_{1})
= x(y + z)(1 + δ_{1} + δ_{2} + δ_{1}δ_{2})
≈ x(y + z)(1 + δ_{1} + δ_{2})
= x(y + z)(1 + δ_{3})

The generalization to the sum of n floating-point numbers is as follows. Theorem 2.2 Let x_{0}, x_{1}, …, x_{n} be positive floating-point numbers. Then the relative error in computing ∑_{i=0}^{n} x_{i} in the usual way is at most (1+ϵ_{M})^{n}−1 ≈ nϵ_{M}, where ϵ_{M} is the machine unit roundoff. Proof: Let S_{k} = x_{0} + x_{1} + ⋯ + x_{k} and S_{k}^{⋆} = fl(S_{k}), and be computed in the usual way

\{ \begin{matrix}S_{0} = x_{0} \\ S_{j+1} = S_{j} + x_{j+1} \end{matrix}  and  \{ \begin{matrix}S_{0}^{⋆} = fl(x_{0}) \\ S_{j+1}^{⋆} = fl(S_{j}^{⋆} + x_{j+1}) \end{matrix}

Define

ρ_{k} = \frac{S_{k}^{∗} − S_{k}}{S_{k}} and δ_{k} = \frac{S_{k+1}^{∗} − S_{k+1}}{S_{k}^{∗} + x_{k+1}} ,

where each |δ_{k}| ≤ ϵ_{M}. Then

ρ_{k+1} = \frac{S_{k+1}^{∗} − S_{k+1}}{S_{k+1}}
= \frac{(S_{k}^{∗} + x_{k+1})(1 + δ_{k}) − (S_{k} + x_{k+1})}{S_{k+1}}
= \frac{(S_{k}(1 + ρ_{k}) + x_{k+1})(1 + δ_{k}) − (S_{k} + x_{k+1})}{S_{k+1}}
= \frac{(S_{k} + ρ_{k}S_{k} + x_{k+1})(1 + δ_{k}) − (S_{k} + x_{k+1})}{S_{k+1}}
= \frac{S_{k} + ρ_{k}S_{k} + x_{k+1} + δ_{k}S_{k} + δ_{k}ρ_{k}S_{k} + δ_{k}x_{k+1} − S_{k} − x_{k+1}}{S_{k+1}}
= \frac{ρ_{k}S_{k} + δ_{k}S_{k} + δ_{k}ρ_{k}S_{k}δ_{k}x_{k+1}}{S_{k+1}}
= \frac{δ_{k}(S_{k} + x_{k+1}) + ρ_{k}(S_{k} + δ_{k}S_{k})}{S_{k+1}}
= \frac{δ_{k}S_{k+1} + ρ_{k}S_{k}(1 + δ_{k})}{S_{k+1}}
= δ_{k} + ρ_{k} (\frac{S_{k}}{S_{k+1}} ) (1 + δ_{k}).

Since x_{i} are all positive, S_{k} < S_{k+1}, and

|ρ_{k+1}| ≤ ϵ_{M} + |ρ_{k}|(1 + ϵ_{M}) = ϵ_{M} + θ|ρ_{k}|,

where θ = 1 + ϵ_{M}. Thus

|ρ_{0}| = 0
|ρ_{1}| ≤ ϵ_{M}
|ρ_{2}| ≤ ϵ_{M} + ϵ_{M}θ
|ρ_{3}| ≤ ϵ_{M} + ϵ_{M}θ + ϵθ^{2}
⋮

In general,

|ρ_{n}| ≤ ϵ_{M} + ϵ_{M}θ + ϵθ^{2} + ⋯ + ϵθ^{n−1}
= ϵ (1 + θ + θ^{2} + ⋯ + θ^{n−1})
= ϵ (\frac{θ^{n} − 1}{θ − 1})
= ϵ (\frac{(1 + ϵ_{M})^{n} − 1}{1 + ϵ_{M} − 1})
= (1 + ϵ_{M})^{n} − 1
= 1 + \binom{n}{1}ϵ_{M} + \binom{n}{2}ϵ_{M}^{2} + ⋯ − 1
≈ \binom{n}{1}ϵ_{M}
= nϵ_{M}.

Direct Methods for Solving Systems of Linear Equations

Systems of linear equations are associated with many problems in engineering and science, as well as with applications to the social sciences and the quantitative study of business and economic problems.

The principal objective of this chapter is to discuss the numerical aspects of solving linear systems of equations having the form

{a_{11}x_{1} + a_{12}x_{2} + ⋯ + a_{1n}x_{n} = b_{1} ; a_{21}x_{1} + a_{22}x_{2} + ⋯ + a_{2n}x_{n} = b_{2} ; ⋮ ; a_{n1}x_{1} + a_{n2}x_{2} + ⋯ + a_{nn}x_{n} = b_{n}  (3.1)

This is a linear system of n equations in n unknowns x_{1}, x_{2}, …, x_{n}. This system can simply be written in the matrix equation form

Ax = b,  (3.2)

where

A = [ \begin{matrix}a_{11} & a_{12} & ⋯ & a_{1n} \\ a_{21} & a_{22} & ⋯ & a_{2n} \\ ⋮ & ⋮ & ... & ⋮ \\ a_{n1} & a_{n2} & ⋯ & a_{nn} \end{matrix} ] , b = [ b_{1} ; b_{2} ; ⋮ ; b_{n} ] , x = [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] .  (3.3)

This equation has a unique solution x = A^{−1}b when the coefficient matrix A is nonsingular. Unless otherwise stated, we shall assume that this is the case. If A^{−1} is already available, then x = A^{−1}b provides a good method of computing the solution x. If A^{−1} is not available, then in general A^{−1} should not be computed solely for the purpose of obtaining x. More efficient numerical procedures will be developed.

Direct methods, which are techniques that give a solution in a fixed number of steps, subject only to round-off errors, are considered in this chapter. Guassian elimination is the principal tool in the direct solution of (3.2). An important technique is to use Gaussian elimination to factor the coefficient matrix into a product of matrices that are easier to manipulate. The factorization is called LU-factorization and has the form A = LU, where L is unit lower triangular and U is upper triangular. We shall construct a general-purpose algorithm, analyze the errors that are associated with the computer solution, and study methods for controlling and reducing the errors.

3.1 Triangular Systems

The Gaussian elimination and LU factorization methods for linear systems involve the conversion of the given square matrix to a triangular system that has the same solution. This section is about the solution of triangular systems.

3.1.1 Diagonal System

We begin by considering the case in which A has a diagonal structure. The easiest linear system of equations (3.2) is the one with diagonal coefficient matrix A, i.e., all the nonzero elements of A are on the main diagonal,

A = [ \begin{matrix}a_{11} & 0 & ⋯ & 0 \\ 0 & a_{22} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & ⋯ & a_{nn} \end{matrix} ] .

In this case, computing the solution x is trivial

x = [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] = [ b_{1}/a_{11} ; b_{2}/a_{22} ; ⋮ ; b_{n}/a_{nn} ] ,  (3.4)

provided that all a_{ii} ≠ 0. If a_{ii} = 0 and b_{i} = 0 for some index i, then x_{i} can be any real number. If a_{ii} = 0 but b_{i} ≠ 0, no solution of the system exists.

However, a linear system with diagonal coefficient matrix is not common and reducing a general matrix to diagonal form is not practical.

3.1.2 Forward Substitution

When a linear system Lx = b is lower triangular of the form

[ \begin{matrix}ℓ_{11} & 0 & ⋯ & 0 \\ ℓ_{21} & ℓ_{22} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ ℓ_{n1} & ℓ_{n2} & ⋯ & ℓ_{nn} \end{matrix} ] [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] = [ b_{1} ; b_{2} ; ⋮ ; b_{n} ] ,  (3.5)

where all diagonals ℓ_{ii} ≠ 0, x_{i} can be obtained by the following procedure

x_{1} = b_{1}/ℓ_{11}
x_{2} = (b_{2} − ℓ_{21}x_{1})/ℓ_{22}
x_{3} = (b_{3} − ℓ_{31}x_{1} − ℓ_{32}x_{2})/ℓ_{33}
⋮
x_{n} = (b_{n} − ℓ_{n1}x_{1} − ℓ_{n2}x_{2} − ⋯ − ℓ_{n,n−1}x_{n−1})/ℓ_{nn}

The general formulation for computing x_{i} is

x_{i} = (b_{i} − ∑_{j=1}^{i−1} ℓ_{ij}x_{j}) /ℓ_{ii},  i = 1, 2, …, n.  (3.6)

This procedure is known as the forward substitution and is used for solving a lower triangular linear system. Algorithms for forward substitution in row-oriented and column-oriented oder are presented.

Algorithm 3.1 (Forward Substitution: Row Version) Suppose that L ∈ ℝ^{n×n} is nonsingular lower triangular and b ∈ ℝ^{n}. This algorithm computes the solution of Lx = b using row-oriented procedure.

for i = 1, …, n do tmp = 0.0 for j = 1, …, i − 1 do

tmp = tmp + L(i, j) ∗ x(j) end for x(i) = (b(i) − tmp)/L(i, i)

end for

Algorithm 3.2 (Forward Substitution: Column version) Suppose that L ∈ ℝ^{n×n} is nonsingular lower triangular and b ∈ ℝ^{n}. This algorithm computes the solution of Lx = b using column-oriented procedure.

for i = 1, …, n do x(i) = b(i)

end for

for j = 1, …, n do x(j) = x(j)/L(j, j) for i = (j + 1), …, n do

x(i) = x(i) − L(i, j) ∗ x(j) end for end for

Since b_{i} is only involved in the formula for x_{i}, therefore the memory storage for b can be overwritten by x. The number of floating-point operations, flops, involved in the forward substitution are

∑_{i=1}^{n} [2(i − 1) + 2] = n^{2} + n.

Hence the forward substitution algorithm is an O(n^{2}) algorithm.

3.1.3 Back Substitution

The analogous algorithm for upper triangular system Ux = b of the form

[ \begin{matrix}u_{11} & u_{12} & ⋯ & u_{1n} \\ 0 & u_{22} & ⋯ & u_{2n} \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & ⋯ & u_{nn} \end{matrix} ] [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] = [ b_{1} ; b_{2} ; ⋮ ; b_{n} ]  (3.7)

is called back substitution. The solution x_{i} are computed in a reversed order by

x_{n} = b_{n}/u_{nn}
x_{n−1} = (b_{n−1} − u_{n−1,n}x_{n})/u_{n−1,n−1}
x_{n−2} = (b_{n−2} − u_{n−2,n−1}x_{n−1} − u_{n−2,n}x_{n})/u_{n−2,n−2}
⋮
x_{1} = (b_{1} − u_{12}x_{2} − u_{13}x_{3} − ⋯ − u_{1n}x_{n})/u_{11}

provided that all u_{ii} ≠ 0. The general formulation is

x_{i} = (b_{i} − ∑_{j=i+1}^{n} u_{ij}x_{j}) /u_{ii},  i = n, n − 1, …, 1.  (3.8)

Algorithm 3.3 (Back Substitution: Row Version) Suppose that U ∈ ℝ^{n×n} is nonsingular upper triangular and b ∈ ℝ^{n}. This algorithm computes the solution of Ux = b using row-oriented procedure.

x(n) = b(n)/U(n, n) for i = (n − 1), …, 1 do

tmp = 0.0

for j = i + 1 : n do

tmp = tmp + U(i, j) ∗ x(j) end for x(i) = (b(i) − tmp)/U(i, i)

end for Algorithm 3.4 (Back Substitution: Column Version) Suppose that U ∈ ℝ^{n×n} is nonsingular upper triangular and b ∈ ℝ^{n}. This algorithm computes the solution of Ux = b using row-oriented procedure.

for i = 1, …, n do x(i) = b(i)

end for

for j = n, …, 1 do x(j) = x(j)/U(j, j) for i = 1 : (j − 1) do

x(i) = x(i) − U(i, j) ∗ x(j) end for end for

Once again the memory storage for b can be overwritten by x. Back substitution requires n^{2} + O(n) flops.

3.2 Gaussian Elimination and LU Factorization

In dealing with systems of linear equations there is a concept of equivalence that is important. If two systems have precisely the same solutions, they are equivalent systems. Thus, to solve a system of linear equations, we can instead solve any equivalent system, no solutions are lost and no new ones appear.

As we have just seen in the previous section that triangular systems are “easy” to solve. The motivation at the heart of Gaussian elimination is to convert a given system Ax = b to an equivalent triangular system. The conversion is achieved by taking appropriate linear combination of the equations. In this section we will give a complete specification of this central procedure and derive an algorithm that computes a matrix factorization called LU factorization such that A = LU, where L is unit lower triangular and U is upper triangular. The solution to the original problem Ax = LUx = b is then found by a two-step triangular solve process:

Ly = b,  Ux = y.  (3.9)

3.2.1 Gaussian Elimination

Given a system of linear equations to be solved, it can be transformed into a simpler equivalent system by the following three types of elementary operations:

1. Interchange two equations in the system (or equivalently, interchange two rows in A):

E_{i} ↔ E_{j};

2. Multiply an equation by a non-zero constant (multiply one row of A by a non-zero

constant):

E_{i} ← λE_{i}.

3. Add to an equation a multiple of some other equation (add to a row a multiple of some

other row):

E_{i} ← E_{i} + λE_{j}.

Here E_{i} denotes the i-th equation in the system. If one system of equations is obtained from another by a finite sequence of elementary operations, then the two systems can be proved to be equivalent.

The first step in the Gaussian elimination process consists of performing, for each i = 2, 3, …, n, the elementary operations

E_{i} ← (E_{i} − m_{i,1}E_{1}),  where m_{i,1} = \frac{a_{i1}}{a_{11}} .  (3.10)

These operations transform the system into one in which all the entries in the first column below the diagonal are zero. Then the process is repeated on the resulting equations E_{2}, …, E_{n}, and so on.

The elementary operation can be carried out with matrix multiplications. An elementary matrix is defined to be an n×n matrix obtained by applying an elementary operation to the n × n identity matrix. Each elementary operation on A can be accomplished by multiplying A on the left by an elementary matrix.

To describe formally the process of Gaussian elimination, we interpret it as a succession of n − 1 major steps resulting in a sequence of matrices as follows:

A = A^{(1)} → A^{(2)} → ⋯ → A^{(n)},

where A^{(n)} is upper triangular. At the conclusion of step k−1, the matrix A^{(k)} is constructed from A^{(k−1)} by elementary operations similar to (3.10), and has the following form:

A^{(k)} = [ \begin{matrix}a_{11}^{(k)} & ⋯ & a_{1,k−1}^{(k)} & a_{1k}^{(k)} & ⋯ & a_{1j}^{(k)} & ⋯ & a_{1n}^{(k)} \\ ⋮ & ... & ⋮ & ⋮ & & ⋮ & & ⋮ \\ 0 & ⋯ & a_{k−1,k−1}^{(k)} & a_{k−1,k}^{(k)} & ⋯ & a_{k−1,j}^{(k)} & ⋯ & a_{k−1,n}^{(k)} \\ 0 & ⋯ & 0 & a_{kk}^{(k)} & ⋯ & a_{kj}^{(k)} & ⋯ & a_{kn}^{(k)} \\ ⋮ & & ⋮ & ⋮ & & ⋮ & & ⋮ \\ 0 & ⋯ & 0 & a_{ik}^{(k)} & ⋯ & a_{ij}^{(k)} & ⋯ & a_{in}^{(k)} \\ ⋮ & & ⋮ & ⋮ & & ⋮ & & ⋮ \\ 0 & ⋯ & 0 & a_{nk}^{(k)} & ⋯ & a_{nj}^{(k)} & ⋯ & a_{nn}^{(k)} \end{matrix} ]

In the k-th step, a_{kk}^{(k)} is used as a pivot element and the k-th row as pivot row, and elementary operations are applied to rows k + 1 through n so that zeros are produced in column k below the diagonal. That is, A^{(k+1)} is obtained from A^{(k)} in which a_{k+1,k}^{(k+1)}, ⋯, a_{nk}^{(k+1)} are zero, row k + 1 through n are modified but row 1 through row k are unchanged when compared to A^{(k)}. More precisely, the entries of A^{(k+1)} are produced by the formula

a_{ij}^{(k+1)} = \begin{cases}a_{ij}^{(k)}, & for i = 1, …, k, and j = 1, …, n; \\ 0, & for i = k + 1, …, n, and j = 1, …, k; \\ a_{ij}^{(k)} − \frac{a_{ik}^{(k)}}{a_{kk}^{(k)}} ∗ a_{kj}^{(k)}, & for i = k + 1, …, n, and j = k + 1, …, n. \end{cases}  (3.11)

If we collect all the multipliers

ℓ_{ik} = \begin{cases}0, & if i < k; \\ 1, & if i = k; \\ \frac{a_{ik}^{(k)}}{a_{kk}^{(k)}} , & if i > k, \end{cases}  (3.12)

and let L = [ℓ_{ik}] and U = A^{(n)}, then L is unit lower triangular, U is upper triangular, and later we shall show that matrix A has the factorization A = LU.

An algorithm to carry out the basic Gaussian elimination process just described is as follows.

Algorithm 3.5 (Gaussian elimination) Given A ∈ ℝ^{n×n} and b ∈ ℝ^{n}, this algorithm implements the Gaussian elimination procedure to reduce A to upper triangular and modify the entries of b accordingly.

for k = 1, …, n − 1 do

for i = k + 1, …, n do t = A(i, k)/A(k, k) A(i, k) = 0 b(i) = b(i) − t ∗ b(k) for j = k + 1, …, n do

A(i, j) = A(i, j) − t ∗ A(k, j) end for end for end for

In the algorithm, it is assumed that all the pivot elements in the elimination process are nonzero. However, it is clear from (3.11) and (3.12) the algorithm will break down if a zero A(k, k) is encountered. A check for nonzero pivot should be included in practical implementation. After the reduction, A becomes upper triangular and the back substitution can be used to obtain the solution vector x.

Example 3.1 This example illustrates the application of Gaussian elimination in solving system of linear equations.

[ \begin{matrix}6 & −2 & 2 & 4 \\ 12 & −8 & 6 & 10 \\ 3 & −13 & 9 & 3 \\ −6 & 4 & 1 & −18 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}12 \\ 34 \\ 27 \\ −38 \end{matrix} ]

Sol:

1^{st} step Use 6 as pivot element, the first row as pivot row, and multipliers 2, \frac{1}{2}, −1 are produced to reduce the system to

[ \begin{matrix}6 & −2 & 2 & 4 \\ 0 & −4 & 2 & 2 \\ 0 & −12 & 8 & 1 \\ 0 & 2 & 3 & −14 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}12 \\ 10 \\ 21 \\ −26 \end{matrix} ]

2^{nd} step Use −4 as pivot element, the second row as pivot row, and multipliers 3, −\frac{1}{2} are computed to reduce the system to

[ \begin{matrix}6 & −2 & 2 & 4 \\ 0 & −4 & 2 & 2 \\ 0 & 0 & 2 & −5 \\ 0 & 0 & 4 & −13 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}12 \\ 10 \\ −9 \\ −21 \end{matrix} ]

3^{rd} step Use 2 as pivot element, the third row as pivot row, and multipliers 2 is found to reduce the system to

[ \begin{matrix}6 & −2 & 2 & 4 \\ 0 & −4 & 2 & 2 \\ 0 & 0 & 2 & −5 \\ 0 & 0 & 0 & −3 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}12 \\ 10 \\ −9 \\ −3 \end{matrix} ]

Now the solution can be obtained by solving this triangular system with back substitution. Note that if we collect all the multipliers and let

L = [ \begin{matrix}1 & 0 & 0 & 0 \\ 2 & 1 & 0 & 0 \\ \frac{1}{2} & 3 & 1 & 0 \\ −1 & −\frac{1}{2} & 2 & 1 \end{matrix} ] and U = [ \begin{matrix}6 & −2 & 2 & 4 \\ 0 & −4 & 2 & 2 \\ 0 & 0 & 2 & −5 \\ 0 & 0 & 0 & −3 \end{matrix} ] ,

then one can verify that LU = A.

3.2.2 Gaussian Transformation and LU Factorization

To obtain a matrix factorization description of Gaussian elimination, it is easier to use a matrix description of the zeroing process. For a given vector v ∈ ℝ^{n} with v_{k} ≠ 0 for some 1 ≤ k ≤ n, let

ℓ_{ik} = \frac{v_{i}}{v_{k}} , i = k + 1, …, n, l_{k} = [ 0 ; ⋮ ; 0 ; ℓ_{k+1,k} ; ⋮ ; ℓ_{n,k} ] ,

and

M_{k} = I − l_{k}e_{k}^{T} = [ \begin{matrix}1 & ⋯ & 0 & 0 & ⋯ & 0 \\ ⋮ & ... & ⋮ & ⋮ & & ⋮ \\ 0 & ⋯ & 1 & 0 & ⋯ & 0 \\ 0 & ⋯ & −ℓ_{k+1,k} & 1 & ⋯ & 0 \\ ⋮ & & ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & −ℓ_{n,k} & 0 & ⋯ & 1 \end{matrix} ] .  (3.13)

Then one can verify that

M_{k}v = [ v_{1} ; ⋮ ; v_{k} ; 0 ; ⋮ ; 0 ] .

That is, M_{k} zeros v_{k+1}, …, v_{n} and keeps v_{1}, …, v_{k} unchanged. This unit lower triangular matrix M_{k} is called a Gaussian transformation, the vector l_{k} a Gauss vector, and the entries ℓ_{k+1}, …, ℓ_{n} the multipliers. Furthermore, one can verify that

M_{k}^{−1} = (I − l_{k}e_{k}^{T})^{−1} = I + l_{k}e_{k}^{T} = [ \begin{matrix}1 & ⋯ & 0 & 0 & ⋯ & 0 \\ ⋮ & ... & ⋮ & ⋮ & & ⋮ \\ 0 & ⋯ & 1 & 0 & ⋯ & 0 \\ 0 & ⋯ & ℓ_{k+1,k} & 1 & ⋯ & 0 \\ ⋮ & & ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & ℓ_{n,k} & 0 & ⋯ & 1 \end{matrix} ] .  (3.14)

Given a nonsingular matrix A ∈ ℝ^{n×n}, Gauss transformations M_{1}, M_{2}, …, M_{n−1} can usually be found such that M_{n−1} ⋯ M_{2}M_{1}A ≡ U is upper triangular. To see this, we denote A^{(1)} = [a_{ij}^{(1)}] = A. If a_{11}^{(1)} ≠ 0, then the Gauss transformation

M_{1} = I − l_{1}e_{1}^{T} , where l_{1} = [ 0 ; ℓ_{21} ; ⋮ ; ℓ_{n1} ] , ℓ_{i1} = \frac{a_{i1}^{(1)}}{a_{11}^{(1)}} , i = 2, …, n,

can be formed such that

A^{(2)} = M_{1}A^{(1)} = [ \begin{matrix}a_{11}^{(2)} & a_{12}^{(2)} & ⋯ & a_{1n}^{(2)} \\ 0 & a_{22}^{(2)} & ⋯ & a_{2n}^{(2)} \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & a_{n2}^{(2)} & ⋯ & a_{nn}^{(2)} \end{matrix} ] ,

where

a_{ij}^{(2)} = \begin{cases}a_{ij}^{(1)}, & for i = 1 and j = 1, …, n; \\ a_{ij}^{(1)} − ℓ_{i1} ∗ a_{1j}^{(1)}, & for i = 2, …, n and j = 2, …, n. \end{cases}

The procedure can proceed to zero the entries in the second column below the diagonal, and so on. In general, at the k-th step, we are confronted with a matrix

A^{(k)} = M_{k−1} ⋯ M_{2}M_{1}A^{(1)}

= [ \begin{matrix}a_{11}^{(k)} & a_{12}^{(k)} & ⋯ & a_{1,k−1}^{(k)} & a_{1k}^{(k)} & ⋯ & a_{1n}^{(k)} \\ 0 & a_{22}^{(k)} & ⋯ & a_{2,k−1}^{(k)} & a_{2k}^{(k)} & ⋯ & a_{2n}^{(k)} \\ ⋮ & ⋮ & ... & ⋮ & ⋮ & & ⋮ \\ 0 & 0 & ⋯ & a_{k−1,k−1}^{(k)} & a_{k−1,k}^{(k)} & ⋯ & a_{k−1,n}^{(k)} \\ 0 & 0 & ⋯ & 0 & a_{kk}^{(k)} & ⋯ & a_{kn}^{(k)} \\ ⋮ & ⋮ & & ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & ⋯ & 0 & a_{kn}^{(k)} & ⋯ & a_{nn}^{(k)} \end{matrix} ]

= [ \begin{matrix}a_{11}^{(1)} & a_{12}^{(1)} & ⋯ & a_{1,k−1}^{(1)} & a_{1k}^{(1)} & ⋯ & a_{1n}^{(1)} \\ 0 & a_{22}^{(2)} & ⋯ & a_{2,k−1}^{(2)} & a_{2k}^{(2)} & ⋯ & a_{2n}^{(2)} \\ ⋮ & ⋮ & ... & ⋮ & ⋮ & & ⋮ \\ 0 & 0 & ⋯ & a_{k−1,k−1}^{(k)} & a_{k−1,k}^{(k)} & ⋯ & a_{k−1,n}^{(k)} \\ 0 & 0 & ⋯ & 0 & a_{kk}^{(k)} & ⋯ & a_{kn}^{(k)} \\ ⋮ & ⋮ & & ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & ⋯ & 0 & a_{kn}^{(k)} & ⋯ & a_{nn}^{(k)} \end{matrix} ]

If the pivot a_{kk}^{(k)} ≠ 0, then the multipliers

ℓ_{ik} = \frac{a_{ik}^{(k)}}{a_{kk}^{(k)}} , i = k + 1, …, n,  (3.15)

can be computed and the Gaussian transformation

M_{k} = I − l_{k}e_{k}^{T} , where l_{k} = [ 0 ; ⋮ ; 0 ; ℓ_{k+1,k} ; ⋮ ; ℓ_{nk} ] ,

can be applied to the left of A^{(k)} to obtain

A^{(k+1)} = M_{k}A^{(k)} = [ \begin{matrix}a_{11}^{(k+1)} & a_{12}^{(k+1)} & ⋯ & a_{1,k−1}^{(k+1)} & a_{1k}^{(k+1)} & ⋯ & ⋯ & a_{1n}^{(k+1)} \\ 0 & a_{22}^{(k+1)} & ⋯ & a_{2,k−1}^{(k+1)} & a_{2k}^{(k+1)} & ⋯ & ⋯ & a_{2n}^{(k+1)} \\ ⋮ & ⋮ & ... & ⋮ & ⋮ & & ⋯ & ⋮ \\ 0 & 0 & ⋯ & a_{k−1,k−1}^{(k+1)} & a_{k−1,k}^{(k+1)} & ⋯ & ⋯ & a_{k−1,n}^{(k+1)} \\ 0 & 0 & ⋯ & 0 & a_{kk}^{(k+1)} & ⋯ & ⋯ & a_{kn}^{(k+1)} \\ ⋮ & ⋮ & & ⋮ & 0 & ... & & ⋮ \\ ⋮ & ⋮ & & ⋮ & ⋮ & & ... & ⋮ \\ 0 & 0 & ⋯ & 0 & 0 & ⋯ & ⋯ & a_{nn}^{(k+1)} \end{matrix} ] ,

in which

a_{ij}^{(k+1)} = \begin{cases}a_{ij}^{(k)}, & for i = 1, …, k, j = 1, …, n; \\ 0, & for i = k + 1, …, n, j = k; \\ a_{ij}^{(k)} − ℓ_{ik}a_{kj}^{(k)}, & for i = k + 1, …, n, j = k + 1, …, n. \end{cases}  (3.16)

Upon the completion,

U ≡ A^{(n)} = M_{n−1} ⋯ M_{2}M_{1}A

is upper triangular. Hence

A = M_{1}^{−1}M_{2}^{−1} ⋯ M_{n−1}^{−1}U ≡ LU,  (3.17)

where

L ≡ M_{1}^{−1}M_{2}^{−1} ⋯ M_{n−1}^{−1} = (I − l_{1}e_{1}^{T})^{−1}(I − l_{2}e_{2}^{T})^{−1} ⋯ (I − l_{n−1}e_{n−1}^{T})^{−1}
= (I + l_{1}e_{1}^{T})(I + l_{2}e_{2}^{T}) ⋯ (I + l_{n−1}e_{n−1}^{T})
= I + l_{1}e_{1}^{T} + l_{2}e_{2}^{T} + ⋯ + l_{n−1}e_{n−1}^{T}

= [ \begin{matrix}1 & 0 & 0 & ⋯ & 0 \\ ℓ_{21} & 1 & 0 & ⋯ & 0 \\ ℓ_{31} & ℓ_{32} & 1 & ⋯ & 0 \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ ℓ_{n1} & ℓ_{n2} & ℓ_{n3} & ⋯ & 1 \end{matrix} ]  (3.18)

is unit lower triangular. This matrix factorization is called the LU-factorization of A.

Equations (3.15) and (3.16) provide formulations for computing the LU-factorization. Alternatively, the algorithm can be derived directly by componentwise comparison in A = LU. Suppose that the factors L and U are written as

L = [ \begin{matrix}1 & 0 & 0 & ⋯ & 0 \\ ℓ_{21} & 1 & 0 & ⋯ & 0 \\ ℓ_{31} & ℓ_{32} & 1 & ⋯ & 0 \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ ℓ_{n1} & ℓ_{n2} & ℓ_{n3} & ⋯ & 1 \end{matrix} ] , U = [ \begin{matrix}u_{11} & u_{12} & u_{13} & ⋯ & u_{1n} \\ 0 & u_{22} & u_{23} & ⋯ & u_{2n} \\ 0 & 0 & u_{33} & ⋯ & u_{3n} \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & 0 & ⋯ & u_{nn} \end{matrix} ] .

By comparing the components in A = LU as

[ \begin{matrix}a_{11} & a_{12} & a_{13} & ⋯ & a_{1n} \\ a_{21} & a_{22} & a_{23} & ⋯ & a_{2n} \\ a_{31} & a_{32} & a_{33} & ⋯ & a_{3n} \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ a_{n1} & a_{n2} & a_{n3} & ⋯ & a_{nn} \end{matrix} ]

= [ \begin{matrix}u_{11} & u_{12} & u_{13} & ⋯ & u_{1n} \\ ℓ_{21}u_{11} & ℓ_{21}u_{12} + u_{22} & ℓ_{21}u_{13} + ℓ_{22}u_{23} & ⋯ & l_{21}u_{1n} + u_{2n} \\ ℓ_{31}u_{11} & ℓ_{31}u_{12} + ℓ_{32}u_{22} & ℓ_{31}u_{13} + ℓ_{32}u_{23} + u_{33} & ⋯ & ⋮ \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ ℓ_{n1}u_{11} & ℓ_{n1}u_{12} + ℓ_{n2}u_{2n} & ℓ_{n1}u_{13} + ℓ_{n2}u_{23} + ℓ_{n3}u_{33} & ⋯ & ℓ_{n1}u_{1n} + ⋯ + u_{nn} \end{matrix} ] ,

we see that u_{11} = a_{11}, u_{12} = a_{12}, …, u_{1n} = a_{1n}. Once u_{11} is determined, ℓ_{21} = a_{21}/u_{11}, ℓ_{31} = a_{31}/u_{11}, …, ℓ_{n1} = a_{n1}/u_{11} can be computed. Next, a_{22} = ℓ_{21}u_{12} + u_{22}. Since ℓ_{21} and u_{12} are known at this stage, u_{22} = a_{22} − ℓ_{21}u_{12} is computed. Once u_{22} is determined, u_{22}, …, u_{2n}, and ℓ_{22}, …, ℓ_{n2} can be obtained. Then we proceed to compute the third row of U and the third column of L, and so on.

The principal of the procedure follows from the identity

a_{ij} = ∑_{p=1}^{min(i,j)} ℓ_{ip}u_{pj}.  (3.19)

Suppose that after k − 1 steps we have computed the first k − 1 columns of L and the first k − 1 rows of U. At the k-th step, we can first determine u_{kk} since

a_{kk} = ∑_{p=1}^{k} ℓ_{kp}u_{pk} = ∑_{p=1}^{k−1} ℓ_{kp}u_{pk} + u_{kk},

and ℓ_{kp}, u_{pk}, p = 1, 2, …, k − 1, are known. The computation is done by performing

u_{kk} = a_{kk} − ∑_{p=1}^{k−1} ℓ_{kp}u_{pk}.  (3.20)

Once u_{kk} is determined and u_{kk} ≠ 0, we can compute the k-th column of L and the k-th row of U. By using the identities

a_{ik} = ∑_{p=1}^{k−1} l_{ip}u_{pk} + l_{ik}u_{kk}, i = k + 1, …, n,

and

a_{kj} = ∑_{p=1}^{k−1} l_{kp}u_{pj} + u_{kj}, j = k + 1, …, n,

we can compute

ℓ_{ik} = (a_{ik} − ∑_{p=1}^{k−1} ℓ_{ip}u_{pk}) /u_{kk},  i = k + 1, …, n,  (3.21)

and

u_{kj} = a_{kj} − ∑_{p=1}^{k−1} ℓ_{kp}u_{pj},  j = k + 1, …, n.  (3.22)

Formulations (3.20), (3.21), and (3.22) together give the Doolittle LU-factorization algorithm.

Algorithm 3.6 (Doolittle LU-Factorization) Given a nonsingular square matrix A ∈ ℝ^{n×n}, this algorithm computes a unit lower triangular matrix L and an upper triangular matrix U such that A = LU.

Initialize L = 0, U = 0 for k = 1, …, n do

L(k, k) = 1 t = 0 for p = 1, …, k − 1 do

t = t + L(k, p) ∗ U(p.k) end for U(k, k) = A(k, k) − t for i = k + 1, …, n do

t = 0 for p = 1, …, k − 1 do

t = t + L(i, p) ∗ U(p, k) end for L(i, k) = (A(i, k) − t)/U(k, k)

end for for j = k + 1, …, n do

t = 0 for p = 1, …, k − 1 do

t = t + L(k, p) ∗ U(p, j) end for U(k, j) = A(k, j) − t

end for end for

For practical implementation, the storage of A can be overwritten by the entries of L and U. The upper triangular components of U overwrite upper triangular part of A, and the multipliers (the strictly lower triangular part of L) can be stored in the strictly lower triangualr part of A. The diagonal part of L needs not to be stored since they are all 1. An outer-product version of the LU-factorization is as follows.

Algorithm 3.7 (LU Factorization) Given a nonsingular square matrix A ∈ ℝ^{n×n}, this algorithm computes a unit lower triangular matrix L and an upper triangular matrix U such that A = LU. The matrix A is overwritten by L and U.

for k = 1, …, n − 1 do

for i = k + 1, …, n do A(i, k) = A(i, k)/A(k, k) for j = k + 1, …, n do

A(i, j) = A(i, j) − A(i, k) ⋆ A(k, j) end for

end for end for

This algorithm requires

∑_{k=1}^{n−1} ∑_{i=k+1}^{n} 2(n − k) = \frac{2}{3}n^{3} − \frac{1}{2}n^{2} + \frac{1}{3}n

flops.

The LU-factorization together with forward substitution and back substitution is a standard procedure for solving the linear system of equations. This approach requires \frac{2}{3}n^{3}+O(n^{2}) flops. The computation of the factorization dominates the cost.

3.2.3 Existence and Uniqueness of LU Factorization

Definition 3.1 (Leading principal minor) Let A be an n × n matrix. The upper left k × k submatrix, denoted as

A_{k} = [ \begin{matrix}a_{11} & a_{12} & ⋯ & a_{1k} \\ a_{21} & a_{22} & ⋯ & a_{2k} \\ ⋮ & ⋮ & ... & ⋮ \\ a_{k1} & a_{k2} & ⋯ & a_{kk} \end{matrix} ] ,

is called the leading k × k principal submtrix, and the determinant of A_{k}, det(A_{k}), is called the leading principal minor.

Theorem 3.1 If all leading principal minor of A ∈ ℝ^{n×n} are nonzero, that is, all leading principal submatrices are nonsingular, then A has an LU-factorization.

Proof: Proof by mathematical induction. When n = 1, A_{1} = [a_{11}] is nonsingular, then a_{11} ≠ 0. Let L_{1} = [1] and U_{1} = [a_{11}]. Then A_{1} = L_{1}U_{1}. The theorem holds.

Assume that the leading principal submatrices A_{1}, …, A_{k} are nonsingular and A_{k} has an LU-factorization A_{k} = L_{k}U_{k}, where L_{k} is unit lower triangular and U_{k} is upper triangular. We shall show that there exist an unit lower triangular matrix L_{k+1} and an upper triangular matrix U_{k+1} such that A_{k+1} = L_{k+1}U_{k+1}.

Write

A_{k+1} = [ \begin{matrix}A_{k} & v_{k} \\ w_{k}^{T} & a_{k+1,k+1} \end{matrix} ] , where v_{k} = [ a_{1,k+1} ; a_{2,k+1} ; ⋮ ; a_{k,k+1} ] and w_{k} = [ a_{k+1,1} ; a_{k+1,2} ; ⋮ ; a_{k+1,k} ] .

Since A_{k} is nonsingular, both L_{k} and U_{k} are nonsingular. Therefore the system L_{k}y_{k} = v_{k} has a unique solution y_{k} ∈ ℝ^{k}, and z^{t}U_{k} = w_{k}^{T} has a unique solution z_{k} ∈ ℝ^{k}. Let

L_{k+1} = [ \begin{matrix}L_{k} & 0 \\ z_{k}^{T} & 1 \end{matrix} ] and U_{k+1} = [ \begin{matrix}U_{k} & y_{k} \\ 0 & a_{k+1,k+1} − z_{k}^{T} y_{k} \end{matrix} ] .

Then L_{k+1} is unit lower triangular, U_{k+1} is upper triangular, and

L_{k+1}U_{k+1} = [ \begin{matrix}L_{k}U_{k} & L_{k}y_{k} \\ z_{k}^{T} U_{k} & z_{k}^{T} y_{k} + a_{k+1,k+1} − z_{k}^{T} y_{k} \end{matrix} ] = [ \begin{matrix}A_{k} & v_{k} \\ w_{k}^{T} & a_{k+1,k+1} \end{matrix} ] = A_{k+1}.

This proves the theorem.

Theorem 3.2 If A is nonsingular and the LU factorization exists, then the LU factorization is unique and det(A) = u_{11} ⋯ u_{nn}.

Proof: Suppose both A = L_{1}U_{1} and A = L_{2}U_{2} are LU factorizations. Since A is nonsingular, L_{1}, U_{1}, L_{2}, U_{2} are all nonsingular, and

A = L_{1}U_{1} = L_{2}U_{2} ⟹ L_{2}^{−1}L_{1} = U_{2}U_{1}^{−1}.

Since L_{1} and L_{2} are unit lower triangular which implies L_{2}^{−1}L_{1} is unit lower triangular, and U_{1} and U_{2} are upper triangular which implies U_{2}U_{1}^{−1} is upper triangular, it follows that L_{2}^{−1}L_{1} = I = U_{2}U_{1}^{−1}. Therefore L_{1} = L_{2} and U_{1} = U_{2}.

3.3 Pivoting

It is clear from the formulations, e.g., (3.20), (3.21), and (3.22), presented in the previous section that Gaussian elimination may fail or may not give satisfactory results if a zero or small pivot entry encountered during the procedure. Therefore good algorithms for solving systems of linear equations must incorporate the interchanging of equations in a system when the circumstances require it. In this section we shall illustrate some of these situations and present the most commonly used pivoting strategies.

3.3.1 The Need for Pivoting

The Gaussian elimination algorithm 3.5 could fail on systems that are in fact easy to solve. For example, the algorithm would fail at the first step on the input system

[ \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}1 \\ 1 \end{matrix} ]

since the first pivot element is zero. But if we interchange the rows, the system

[ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] [ \begin{matrix}x_{2} \\ x_{1} \end{matrix} ] = [ \begin{matrix}1 \\ 1 \end{matrix} ]

becomes trivial to solve.

Another difficulty will arise when a pivot element encountered is a small number ε different from zero. For example, the simple Gaussian elimination algorithm would produce relatively large error on the system

[ \begin{matrix}ε & 1 \\ 1 & 1 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}1 \\ 2 \end{matrix} ] ,

where ε < ϵ_{M}. Algorithm 3.5 would compute

[ \begin{matrix}ε & 1 \\ 0 & 1 − \frac{1}{ε} \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}1 \\ 2 − \frac{1}{ε} \end{matrix} ] ⟹ [ \begin{matrix}ε & 1 \\ 0 & −\frac{1}{ε} \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}1 \\ −\frac{1}{ε} \end{matrix} ] ,

since in the computer, if ε is small enough, 1 − \frac{1}{ε} will be computed to be the same as −\frac{1}{ε}. Likewise, 2 − \frac{1}{ε} will be computed to be the same as −\frac{1}{ε}. Hence, under these circumstances, back substitution would produce

x_{2} = \frac{−\frac{1}{ε}}{−\frac{1}{ε}} = 1 and x_{1} = \frac{1 − 1}{ε} = 0.

The computed solution is accurate for x_{2} but is extremely inaccurate for x_{1} since

[ \begin{matrix}ε & 1 \\ 1 & 1 \end{matrix} ] [ \begin{matrix}0 \\ 1 \end{matrix} ] = [ \begin{matrix}1 \\ 1 \end{matrix} ] .

But actually x_{1} = x_{2} = 1 would be a much better solution since

[ \begin{matrix}ε & 1 \\ 1 & 1 \end{matrix} ] [ \begin{matrix}1 \\ 1 \end{matrix} ] = [ \begin{matrix}1 + ε \\ 2 \end{matrix} ] ≈ [ \begin{matrix}1 \\ 2 \end{matrix} ] .

On the other hand, it would produce a much better result if we interchange the rows since Gaussian elimination would compute

[ \begin{matrix}1 & 1 \\ ε & 1 \end{matrix} ] [ \begin{matrix}x_{2} \\ x_{1} \end{matrix} ] = [ \begin{matrix}2 \\ 1 \end{matrix} ] ⟹ [ \begin{matrix}1 & 1 \\ 0 & 1 − ε \end{matrix} ] [ \begin{matrix}x_{2} \\ x_{1} \end{matrix} ] = [ \begin{matrix}2 \\ 1 − 2ε \end{matrix} ] ,

and the back substitution will give

x_{1} = \frac{1 − 2ϵ}{1 − ε} ≈ 1 and x_{2} = 2 − x_{1} ≈ 2 − 1 = 1.

The strategy of interchange rows/columns as described above is called “pivoting”. And we have seen that pivoting is necessary for some systems.

3.3.2 Partial Pivoting and Complete Pivoting

As we have observed in the previous subsection that if a_{kk}^{(k)} is small in magnitude compared to a_{ik}^{(k)}, i = k + 1, …, n, then the multipliers

ℓ_{ik} = \frac{a_{ik}^{(k)}}{a_{kk}^{(k)}}

will have magnitude much larger than 1. Roundoff introduced in computing

a_{ij}^{(k+1)} = a_{ij}^{(k)} − ℓ_{ik} ∗ a_{kj}^{(k)}, i = k + 1, …, n, j = k + 1, …, n,

will be large. Also when performing the back substitution for

x_{k} = (b̃_{k} − ∑_{j=k+1}^{n} a_{kj}^{(k)}) /a_{kk}^{(k)},

where b̃_{k} is the modified b_{k} by algorithm 3.5, any error in the numerator will be dramatically increased when dividing by a small a_{kk}^{(k)}.

To ensure that no large entries appear in the computed triangular factors, one can choose a pivot element to be the largest entry among |a_{kk}^{(k)}|, …, |a_{nk}^{(k)}|. Then permute this row with the k-th row. The multipliers ℓ_{ik} are computed after the permutation.

To explain the pivoting techniques in matrix language, let P_{1}, …, P_{k−1} be the permutations chosen and M_{1}, … M_{k−1} denote the Gaussian transformations performed in the first k − 1 steps. At the k-th step, a permutation matrix P_{k} is chosen so that

|(P_{k}M_{k−1} ⋯ M_{1}P_{1}A)_{kk}| = max_{k≤i≤n} |(M_{k−1} ⋯ M_{1}P_{1}A)_{ik}| .

This row interchange strategy is called partial pivoting. As a consequence, no multipliers is greater than one in magnitude, that is, |ℓ_{ij}| ≤ 1 for i = 1, …, n, j = 1, …, i. Upon completion, we obtain an upper triangular matrix

U ≡ M_{n−1}P_{n−1} ⋯ M_{1}P_{1}A.  (3.23)

Since any permutation matrix P_{k} is symmetric and P_{k}^{T} P_{k} = P_{k}^{2} = I, a manipulation of (3.23) reveals that

M_{n−1}P_{n−1} ⋯ M_{2}P_{2}M_{1}P_{2} ⋯ P_{n−1}P_{n−1} ⋯ P_{1}A = U,

therefore,

P_{n−1} ⋯ P_{1}A = (M_{n−1}P_{n−1} ⋯ M_{2}P_{2}M_{1}P_{2} ⋯ P_{n−1})^{−1}U.

In summary, Gaussian elimination with partial pivoting leads to the LU factorization

P A = LU,  (3.24)

where

P = P_{n−1} ⋯ P_{1}

is a permutation matrix, and

L ≡ (M_{n−1}P_{n−1} ⋯ M_{2}P_{2}M_{1}P_{2} ⋯ P_{n−1})^{−1}

is unit lower triangular.

While in the partial pivoting algorithm, the k-th pivot is determined by scanning the current k-th subcolumn A^{(k)}(k : n, k), another pivoting strategy called complete pivoting searches for the largest entry in magnitude in the current submatrix A^{(k)}(k : n, k : n) and permutes to the (k, k) position. That is, at the k-th step two permutation matrices P_{k} and Q_{k} are determined so that

|(P_{k}A^{(k)}Q_{k})_{kk}| = max_{k≤i,j≤n} |(A^{(k)})_{ij}| .

Gaussian elimination with complete pivoting leads to the LU factorization

P AQ = LU,  (3.25)

where P, Q are permutation matrices, L is unit lower triangular, and U is upper triangular.

In practical implementation, no permutation matrix should ever be explicitly formed. The permutations can be efficiently represented by an integer n-vector. For example, an integer array p is usually used for partial pivoting to keep the indices of rows such that p(k) stores the row index which interchanges with row k. In other words, (3.11) becomes

a_{p(i),j}^{(k+1)} = \begin{cases}a_{p(i),j}^{(k)}, & for i = 1, …, k, and j = 1, …, n; \\ 0, & for i = k + 1, …, n, and j = k; \\ a_{p(i),j}^{(k)} − \frac{a_{p(i),k}^{(k)}}{a_{p(k),k}^{(k)}} ∗ a_{p(k),j}^{(k)}, & for i = k + 1, …, n, and j = k + 1, …, n. \end{cases}  (3.26)

Two integer arrays will be required to store the permutation information for complete pivoting.

It should be noted that although there is no floating-point arithmetic involved when a permutation is applied to a matrix, only row/columns exchanged, it often causes irregular data movement and significant computational overhead. The LU factorization algorithm 3.7 can be modified to incorporate with partial pivoting as shown in the following. The one for complete pivoting is similar.

Algorithm 3.8 (LU-factorization with Partial Pivoting) Given a nonsingular square matrix A ∈ ℝ^{n×n}, this algorithm finds an appropriate permutation matrix P, and computes a unit lower triangular matrix L and an upper triangular matrix U such that P A = LU. The matrix A is overwritten by L and U, and the matrix P is not formed. An integer array p is instead used for storing the row/column indices.

for i = 1, …, n do

p(i) = i % initialize the pivoting vector end for for k = 1, …, n − 1 do

m = k

for i = k + 1, …, n do

if |A(p(m), k)| < |A(p(i), k)| then

m = i end if end for ℓ = p(k) p(k) = p(m) p(m) = ℓ for i = k + 1, …, n do

A(p(i), k) = A(p(i), k)/A(p(k), k)

for j = k + 1, …, n do

A(p(i), j) = A(p(i), j) − A(p(i), k) ∗ A(p(k), j) end for end for end for

Since the Gaussian elimination with partial pivoting produces the factorization (3.24), the linear system problem should comply accordingly.

Ax = b ⟹ P Ax = P b ⟹ LUx = P b.

Hence the permutation needs to be applied to the right-hand-side vetor b, and the forward substitution should be modified to

Algorithm 3.9 (Permuted Forward Substitution) Given an unit lower triangular matrix L, a permutation matrix P, and an n-vector b, this algorithm solves the triangular system Ly = P b. The matrix L is stored in the strictly lower triangular part of matrix A, and the premutation matrix P is represented by an integer array p.

y(p(1)) = b(p(1))/A(p(1), 1)

for i = 1, …, n do t = 0 for j = 1, …, i − 1 do

t = t + A(p(i), j) ∗ y(p(j)) end for y(p(i)) = b(p(i)) − t

end for

3.3.3 Scaled Row Pivoting

In the following example, one will see that it is not actually the smallness of the pivot element that is causing the trouble. Rather, it is the smallness of the pivot element relative to the other elements in its row. Consider

[ \begin{matrix}1 & \frac{1}{ε} \\ 1 & 1 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ 2^{\frac{1}{ε}} ] ,

where ε << ϵ_{M}. The algorithm of Gaussian elimination with or without pivoting would give

[ \begin{matrix}1 & \frac{1}{ε} \\ 0 & 1 − \frac{1}{ε} \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ 2 − \frac{1}{ε}^{\frac{1}{ε}} ] ⟹ [ \begin{matrix}1 & \frac{1}{ε} \\ 0 & −\frac{1}{ε} \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ −\frac{1}{ε}^{\frac{1}{ε}} ] ,

and the back substitution will result

x_{2} = \frac{\frac{−1}{ε}}{\frac{−1}{ε}} = 1, and x_{1} = \frac{1}{ε} − \frac{1}{ε} · x_{2} = 0.

Again, the solution is accurate for x_{2} but is extremely inaccurate for x_{1} since

[ \begin{matrix}1 & \frac{1}{ε} \\ 1 & 1 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}1 & \frac{1}{ε} \\ 1 & 1 \end{matrix} ] [ \begin{matrix}0 \\ 1 \end{matrix} ] = [ 1^{\frac{1}{ε}} ] .

What causes the problem? One observes that the pivot element, 1, is too small relative to the other entries, \frac{1}{ε}, in the pivot row in this example.

Another pivoting strategy called scaled row pivoting is introduced. We begin the factorization by computing the scale of each row

s_{i} = max_{1≤j≤n} |a_{ij}|, i = 1, …, n.  (3.27)

These values are stored in an array s in the algorithm. To keep track of the row indices, we begin by setting the permutation vector, also called pivoting vector, p = [1, 2, …, n]^{T}. In each Gaussian elimination step k, the algorithm scans the numbers

\frac{|a_{p(i),k}|}{s_{p(i)}}, i = k, …, n,

looking for a maximal one. Suppose m is the row index of the maximal ratio, we exchange indices p(k) and p(m), and use the new a_{p(k),k} as the pivot element and row p(k) as the pivot row to zero entries a_{p(k+1),k} throught a_{p(n),k}, and modify the lower right submatrix, that is,

a_{p(i),j} ← a_{p(i),j} − \frac{a_{p(i),k}}{a_{p(k),k}} ∗ a_{p(k),j} j = k + 1, …, n, i = k + 1, …, n.

This procedure is repeated for k = 1, …, n − 1.

We use the following example to illustrate how the Gaussian elimination with scaled row pivoting works.

Example 3.2 Consider solving the linear system of equations in Example 3.1 using Gaussian elimination with scaled row pivoting.

Sol: Initially, we have

A = [ \begin{matrix}6 & −2 & 2 & 4 \\ 12 & −8 & 6 & 10 \\ 3 & −13 & 9 & 3 \\ −6 & 4 & 1 & −18 \end{matrix} ] , p = [ \begin{matrix}1 \\ 2 \\ 3 \\ 4 \end{matrix} ] , s = [ \begin{matrix}6 \\ 12 \\ 13 \\ 18 \end{matrix} ] .

Step through the Gaussian elimination with scaled row pivoting we have

k = 1 : Scan \frac{6}{6}, \frac{12}{12}, \frac{3}{13}, \frac{6}{18}, choose row 1 as the pivot row, vector p unchanged.

A = [ \begin{matrix}6 & −2 & 2 & 4 \\ 2 & −4 & 2 & 2 \\ \frac{1}{2} & −12 & 8 & 1 \\ −1 & 2 & 3 & −14 \end{matrix} ] , p = [ \begin{matrix}1 \\ 2 \\ 3 \\ 4 \end{matrix} ] .

k = 2 : Scan \frac{4}{12}, \frac{12}{13}, \frac{2}{18}, choose row 3 as the pivot row.

A = [ \begin{matrix}6 & −2 & 2 & 4 \\ 2 & \frac{1}{3} & −\frac{2}{3} & \frac{5}{3} \\ \frac{1}{2} & −12 & 8 & 1 \\ −1 & −\frac{1}{6} & \frac{13}{3} & −\frac{83}{6} \end{matrix} ] , p = [ \begin{matrix}1 \\ 3 \\ 2 \\ 4 \end{matrix} ] .

k = 3 : Scan \frac{2/3}{12}, \frac{13/3}{18}, choose row 4 as the pivot row.

A = [ \begin{matrix}6 & −2 & 2 & 4 \\ 2 & \frac{1}{3} & −\frac{2}{13} & −\frac{18}{39} \\ \frac{1}{2} & −12 & 8 & 1 \\ −1 & −\frac{1}{6} & \frac{13}{3} & −\frac{83}{6} \end{matrix} ] , p = [ \begin{matrix}1 \\ 3 \\ 4 \\ 2 \end{matrix} ] .

Note that the entries of A are not actually moved and some positions of A are overwritten by the multipliers. If we let

P = [ \begin{matrix}1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{matrix} ]

denote the permutation matrix and

L = [ \begin{matrix}1 & 0 & 0 & 0 \\ \frac{1}{2} & 1 & 0 & 0 \\ −1 & −\frac{1}{6} & 1 & 0 \\ 2 & \frac{1}{3} & −\frac{2}{13} & 1 \end{matrix} ] , U = [ \begin{matrix}6 & −2 & 2 & 4 \\ 0 & −12 & 8 & 1 \\ 0 & 0 & \frac{13}{3} & −\frac{83}{6} \\ 0 & 0 & 0 & −\frac{18}{39} \end{matrix} ] ,

then one can verify that P A = LU.

Algorithm 3.10 (Gaussian Elimination with Scaled-Row Pivoting) Let A ∈ ℝ^{n×n} be a nonsingular matrix. This algorithm performs the Gaussian elimination with scaled row pivoting to produce the factorization P A = LU. The matrix A is modified and the multipliers are stored in A. An integer array p is used to store the permutation information.

for i = 1, …, n do p(i) = i

s(i) = max_{1≤j≤n} |a_{ij}|

end for

for k = 1, …, n − 1 do m = k for i = k + 1, …, n do

if |A(p(m), k)|/s(p(m)) < |A(p(i), k)|/s(p(i)) then

m = i end if end for ℓ = p(k) p(k) = p(m) p(m) = ℓ for i = k + 1, …, n do

A(p(i), k) = A(p(i), k)/A(p(k), k)

for j = k + 1, …, n do

A(p(i), j) = A(p(i), j) − A(p(i), k) ∗ A(p(k), j) end for end for end for

The number of arithmetic operations in the factorization can be approximately estimated as

∑_{k=1}^{n−1}[(n − k) + 2(n − k)(n − k + 1)] = 2 ∑_{k=1}^{n−1} k^{2} + 3 ∑_{k=1}^{n−1} k + (n − 1)

= \frac{2}{3}n^{3} + \frac{1}{2}n^{2} − \frac{1}{6}n − 1 ≈ \frac{2}{3}n^{3} + \frac{1}{2}n^{2}.

For large n, the term \frac{2}{3}n^{3} is the dominant one.

Notice that the multipliers are being stored in A at the locations where 0’s would have been created by the elimination process. That is, factors L and U overwrite A. The original matrix A should be saved in a separate array if it will be need again in later computation.

Once the factorization has been carried out on A, we apply the forward substitution algorithm using the final permutation array p and the multipliers that were determined by the factorization process and were stored in A. Finally we carry out the back substitution algorithm to solve for x.

Algorithm 3.11 (Updating & Back Substitution) Given b ∈ ℝ^{n}, an integer array p, and A ∈ ℝ^{n×n}, where A is the resulting matrix after Gaussian elimination with scaled row pivoting, this algorithm solves Ax = b for x.

for i = 1, …, n − 1 do for j = i + 1, …, n do

b(p(j)) = b(p(j)) − A(p(j), i) ∗ b(p(i)) end for end for for i = n, n − 1, …, 1 do

t = 0

for j = i + 1, …, n do t = t + A(p(i), j) ∗ x(j)

end for

x(i) = (b(p(i)) − t)/A(p(i), i) end for

The number of floating-point operations involved in the solution phase is about

∑_{i=1}^{n−1} 2(n − i) + ∑_{i=1}^{n}[2(n − i) + 2] = 2 ∑_{k=1}^{n−1} k + 2 ∑_{k=1}^{n} k = 2n^{2}.

In summary, it costs \frac{2}{3}n^{3} + O(n^{2}) flops to solve the linear system of equations Ax = b using Gaussian elimination with scaled row pivoting. Note that it would take \frac{4}{3}n^{3} + O(n^{2}) just to compute A^{−1}. Be advised that A^{−1} should not be computed when solving a linear system Ax = b, but rather x should be solved for directly.

3.4 Some Special Linear Systems

It is a basic tenet of numerical analysis that structure should be exploited whenever solving a problem. Algorithms for general matrix problems can be streamlined in the presence of such properties as symmetry, definiteness, and sparsity. This is the central theme of this section, where our principal aim is to discuss special algorithms for computing special variants of the LU factorization.

3.4.1 Symmetric Positive Definite System and Cholesky Factorization

An n × n matrix A is positive definite if x^{T} Ax > 0, for all x ∈ ℝ^{n}, x ≠ 0. If A is both symmetric and positive definite (spd), then we can derive a stable LU factorization called the Choleseky factorization.

Lemma 3.1 If A ∈ ℝ^{n×n} is positive definite, then A is nonsingular and a_{ii} > 0 for i = 1, …, n.

Proof: Suppose A is singular. Then there exists x ∈ ℝ^{n} and x ≠ 0 such that Ax = 0. This implies x^{T} Ax = 0, which contradicts the fact that A is positive definite. Therefore A is nonsingular. It is also easy to see that

a_{ii} = e_{i}^{T} Ae_{i} > 0,

where e_{i} is the i-th column of the n × n identify matrix.

Lemma 3.2 If A ∈ ℝ^{n×n} is positive definite, then all leading principal submatrices of A are nonsingular.

Proof: For 1 ≤ k ≤ n, let z_{k} = [x_{1}, …, x_{k}]^{T} ∈ ℝ^{k} and x = [x_{1}, …, x_{k}, 0, …, 0]^{T} ∈ ℝ^{n}, where x_{1}, …, x_{k} ∈ ℝ are not all zero. Since A is positive definite,

z_{k}^{T} A_{k}z_{k} = x^{T} Ax > 0,

where A_{k} is the k × k leading principal submatrix of A. This shows that A_{k} are also positive definite, hence A_{k} are nonsingular.

Theorem 3.3 If A ∈ ℝ^{n×n} is symmetric positive definite, then there exists a unique lower triangular matrix G ∈ ℝ^{n×n} with positive diagonal entries such that A has the factorization

A = GG^{T} .  (3.28)

Proof: From Lemma 3.2, all leading principal submatrices of a positive definite matrix are nonsingular, hence A has the LU factorization A = LU, where L is unit lower triangular and U is upper triangular. Since A is symmetric,

LU = A = A^{T} = U^{T} L^{T} ⟹ U(L^{T})^{−1} = L^{−1}U^{T} .

Note that U(L^{T})^{−1} is upper triangular and L^{−1}U^{T} is lower triangular, this forces U(L^{T})^{−1} to be a diagonal matrix, say, U(L^{T})^{−1} = D. Then U = DL^{T}. Hence

A = LDL^{T} .

Since A is positive definite,

x^{T} Ax > 0 ⟹ x^{T} LDL^{T} x = (L^{T} x)^{T} D(L^{T} x) > 0.

This means D is also positive definite, and hence d_{ii} > 0. Thus D^{1/2} is well-defined and we have

A = LDL^{T} = LD^{1/2}D^{1/2}L^{T} ≡ GG^{T} ,

where G ≡ LD^{1/2}. Since the LU factorization is unique, G is unique.

The factorization (3.28) is referred to as the Cholesky factorization and G is referred to as the Cholesky factor or Cholesky triangle, when A is both symmetric and positive definite.

To derive an algorithm for computing the Cholesky factorization (3.28) we assume that the first k − 1 columns of G have been determined after k − 1 steps. By componentwise comparison with equation (3.28), one has

a_{kk} = ∑_{j=1}^{k} g_{kj}^{2},

which gives

g_{kk}^{2} = a_{kk} − ∑_{j=1}^{k−1} g_{kj}^{2}.  (3.29)

Moreover,

a_{ik} = ∑_{j=1}^{k} g_{ij} ∗ g_{kj}, i = k + 1, …, n,

hence the k-th column of G can be computed by

g_{ik} = (a_{ik} − ∑_{j=1}^{k−1} g_{ij} ∗ g_{kj}) /g_{kk},  i = k + 1, …, n.  (3.30)

Equations (3.29) and (3.30) lay out the formulations for the Cholesky factorization algorithm.

Algorithm 3.12 (Cholesy Factorization) Given an n × n symmetric positive definite matrix A, this algorithm computes the Cholesky factorization A = GG^{T}.

Initialize G = 0 for k = 1, …, n do

G(k, k) = \sqrt{A(k, k) − ∑_{j=1}^{k−1} G(k, j) ∗ G(k, j)}

for i = k + 1, …, n do

G(i, k) = (A(i, k) − ∑_{j=1}^{k−1} G(i, j) ∗ G(k, j)) /G(k, k)

end for end for In addition to n square root operations, there are approximately

∑_{k=1}^{n} [2k − 1 + 2k(n − k)] = \frac{1}{3}n^{3} + n^{2} − \frac{1}{3}n

floating-point arithmetic required by the algorithm. In practical implementation, the lower triangular part of A can be overwritten by the Cholesy factor G.

It is clear from (3.29) that

a_{kk} = ∑_{j=1}^{k} g_{kj}^{2} ≥ g_{ki}^{2}, i = 1, …, k.

Hence

|g_{ki}| ≤ \sqrt{a_{kk}},  ∀ i = 1, …, k.  (3.31)

This means that the entries of G are nicely bounded and will not grow large relative to the diagonals of A. Therefore, pivoting will not be required, and the Cholesky factorization is a stable numerical algorithm.

3.4.2 Diagonally Dominant Systems

A matrix A ∈ ℝ^{n×n} is said to be diagonally dominant if

|a_{ii}| > ∑_{j=1,j≠i}^{n} |a_{ij}|.

We shall show that Guassian elimination without pivoting can be safely used for this class of matrices.

Lemma 3.3 If A ∈ ℝ^{n×n} is diagonally dominant, then A is nonsingular.

Proof: Suppose A is singular. Then there exists x ∈ ℝ^{n}, x ≠ 0 such that Ax = 0. Let k be the integer index such that

|x_{k}| = max_{1≤i≤n} |x_{i}| ⟹ \frac{|x_{i}|}{|x_{k}|} < 1, ∀ i ≠ k.

Since Ax = 0, for the fixed k, we have

∑_{j=1}^{n} a_{kj}x_{j} = 0 ⟹ a_{kk}x_{k} = − ∑_{j=1,j≠k}^{n} a_{kj}x_{j} ⟹ |a_{kk}||x_{k}| ≤ ∑_{j=1,j≠k}^{n} |a_{kj}||x_{j}|,

which implies

|a_{kk}| ≤ ∑_{j=1,j≠k}^{n} |a_{kj}|\frac{|x_{j}|}{|x_{k}|} < ∑_{j=1,j≠k}^{n} |a_{kj}|.

But this contradicts the assumption that A is diagonally dominant. Therefore A must be nonsingular.

Theorem 3.4 Gaussian elimination without pivoting preserve the diagonal dominance of a matrix.

Proof: Let A ∈ ℝ^{n×n} be a diagonally dominant matrix and A^{(2)} = [a_{ij}^{(2)}] is the result of applying one step of Gaussian elimination to A^{(1)} = A without any pivoting strategy.

After one step of Gaussian elimination, a_{i1}^{(2)} = 0 for i = 2, …, n, and the first row is unchanged. Therefore, the property

a_{11}^{(2)} > ∑_{j=2}^{n} |a_{1j}^{(2)}|

is preserved, and all we need to show is that

a_{ii}^{(2)} > ∑_{j=2,j≠i}^{n} |a_{ij}^{(2)}|, for i = 2, …, n.

Using the Gaussian elimination formula (3.11), we have

|a_{ii}^{(2)}| = |a_{ii}^{(1)} − \frac{a_{i1}^{(1)}}{a_{11}^{(1)}} ∗ a_{1i}^{(1)}| = |a_{ii} − \frac{a_{i1}}{a_{11}} ∗ a_{1i}|
≥ |a_{ii}| − \frac{|a_{i1}|}{|a_{11}|} ∗ |a_{1i}|
= |a_{ii}| − |a_{i1}| + |a_{i1}| − \frac{|a_{i1}|}{|a_{11}|} ∗ |a_{1i}|
= |a_{ii}| − |a_{i1}| + \frac{|a_{i1}|}{|a_{11}|} ∗ (|a_{11}| − |a_{1i}|)
> ∑_{j=2,j≠i}^{n} |a_{ij}| + \frac{|a_{i1}|}{|a_{11}|} ∑_{j=2,j≠i}^{n} |a_{1j}|

= ∑_{j=2,j≠i}^{n} |a_{ij}| + ∑_{j=2,j≠i}^{n} \frac{|a_{i1}|}{|a_{11}|}|a_{1j}|

≥ ∑_{j=2,j≠i}^{n} |a_{ij} − \frac{a_{i1}}{a_{11}} a_{1j}|

= ∑_{j=2,j≠i}^{n} |a_{ij}^{(2)}|

Thus A^{(2)} is still diagonally dominant. Since the subsequent steps of Gaussian elimination mimic the first, except for being applied to submatrices of smaller size, it suffices to conclude that Gaussian elimination without pivoting preserves the diagonal dominance of a matrix.

Theorem 3.5 If the Gaussian elimination with scaled row pivoting algorithm is applied to a diagonally dominant matrix A ∈ ℝ^{n×n}, then the pivot vector will be p = [1, 2, …, n]^{T}. That is, no permutation is needed.

Proof: By theorem 3.4, it is sufficient to prove that the first pivot row chosen by Algorithm 3.10 is row one. That is, it is enough to show that

\frac{|a_{11}|}{s_{1}} > \frac{|a_{i1}|}{s_{i}}, ∀ i = 2, …, n.

Because of the diagonal dominance,

|a_{ii}| > ∑_{j=1,j≠i}^{n} |a_{ij}| ⟹ |a_{ii}| > |a_{ij}|, 1 ≤ j ≤ n, j ≠ i ⟹ s_{i} = max_{1≤j≤n} |a_{ij}| = |a_{ii}|,

for i = 1, …, n, hence

\frac{|a_{11}|}{s_{1}} = \frac{|a_{11}|}{|a_{11}|} = 1, and \frac{|a_{i1}|}{s_{i}} = \frac{|a_{i1}|}{|a_{ii}|} < 1, ∀ i = 2, … n.

Thus the theorem is proved.

3.4.3 Tridiagonal System

A square matrix A = [a_{ij}] is said to be tridiagonal if a_{ij} = 0 for all i, j that satisfy |i−j| > 1, that is, A has the structure

A = [ \begin{matrix}a_{11} & a_{12} \\ a_{21} & a_{22} & a_{23} \\ & ... & ... & ... \\ & & ... & ... & a_{n−1,n} \\ & & & a_{n,n−1} & a_{n,n} \end{matrix} ] .

In the display, matrix elements not shown are zero’s. The factorization algorithm can be simplified considerably in the case of tridiagonal matrix if Guassian elimination can be applied safely without pivoting. This is true, for example, if the matrix is symmetric positive definite or diagonally dominant. The L and U factors would have the form

L = [ \begin{matrix}1 \\ ℓ_{21} & 1 \\ & ... & ... \\ & & ... & ... \\ & & & ℓ_{n,n−1} & 1 \end{matrix} ] and U = [ \begin{matrix}u_{11} & u_{12} \\ & u_{22} & u_{23} \\ & & ... & ... \\ & & & ... & u_{n−1,n} \\ & & & & u_{nn} \end{matrix} ] ,

and the entries are computed by the simple algorithm which only costs 3n flops.

Algorithm 3.13 (Tridiagonal LU Factorization) This algorithm computes the LU factorization for a tridiagonal matrix without using pivoting strategy.

U(1, 1) = A(1, 1) for i = 2, …, n do

U(i − 1, i) = A(i − 1, i)

L(i, i − 1) = A(i, i − 1)/U(i − 1, i − 1) U(i, i) = A(i, i) − L(i, i − 1) ∗ U(i − 1, i)

end for

For practical implementation, A can be overwritten by the entries of L and U. Moreover, a tridiagonal matrix can be stored in an n × 3 array. Alternatively we can arrange the notation like this

A = [ \begin{matrix}d_{1} & c_{1} \\ a_{1} & d_{2} & c_{2} \\ & ... & ... & ... \\ & & ... & ... & c_{n−1} \\ & & & a_{n−1} & d_{n} \end{matrix} ] ,  (3.32)

and use three n-vectors for storage. A tridiagonal linear system arises in many applications, such as finite difference discretization to second order linear boundary-value problem and the cubic spline approximations. Such a system can be solved efficiently by the following algorithm, which costs only 8n flops.

Algorithm 3.14 (Solving Tridiagonal System) This algorithm solves a tridiagonal linear sytem Ax = b of the form (3.32) using Gaussian elimination without pivoting.

for i = 2, …, n do t = a(i − 1)/d(i − 1) d(i) = d(i) − t ∗ c(i − 1) b(i) = b(i) − t ∗ b(i − 1)

end for

x(n) = b(n)/d(n)

for i = n − 1, …, 1 do

x(i) = (b(i) − c(i) ∗ x(i + 1))/d(i) end for

If partial pivoting is necessary, then, in addition to the pivoting vector, an extra storage should be arranged for the entries appear in the second superdiagonal because of pivoting.

3.4.4 General Banded Systems

In many applications that involve linear systems, the coefficient matrix is banded. Formally, we say that A = [a_{ij}] has upper bandwidth q if a_{ij} = 0 whenever j > i + q and lower bandwidth p if a_{ij} = 0 whenever i > j+p. Substantial economies can be realized when solving banded systems because the triangular factors in the LU factorization are also banded.

3.5 Perturbation Analysis

In this section, we develop some perturbation theory for the problem of solving linear systems Ax = b. If we solve such a system numerically, we obtain not the exact solution x but an approximate computed solution x̂. The difference

e = x − x̂

is called the error vector which is, however, not known. Instead one can test the accuracy of x̂ by forming Ax̂ to see whether it is close to b. Thus we have the definition for the residual vector.

Definition 3.2 Let x̂ be the computed solution to the linear system of equations Ax = b. Then the vector

r = b − Ax̂  (3.33)

is called the residual vector.

Then we can derive the residual equation

Ae = Ax − Ax̂ = b − Ax̂ = r  (3.34)

between the error vector and the residual vector.

Notice that x̂ is the exact solution of the linear system

Ax̂ = b̂,

which has a perturbed right-hand side

b̂ = b − r.

Then

∥x − x̂∥ = ∥A^{−1}b − A^{−1}b̂∥ = ∥A^{−1}(b − b̂)∥
≤ ∥A^{−1}∥∥b − b̂∥ = ∥A^{−1}∥∥b∥\frac{∥b − b̂∥}{∥b∥} = ∥A^{−1}∥∥Ax∥\frac{∥b − b̂∥}{∥b∥}
≤ ∥A^{−1}∥∥A∥∥x∥\frac{∥b − b̂∥}{∥b∥}

Therefore

\frac{∥x − x̂∥}{∥x∥} ≤ κ(A)\frac{∥b − b̂∥}{∥b∥} = κ(A)\frac{∥r∥}{∥b∥},  (3.35)

where

κ(A) = ∥A∥∥A^{−1}∥  (3.36)

is called the condition number of A.

On the other hand, by the residual vector, we have

∥r∥∥x∥ = ∥Ae∥∥A^{−1}b∥ ≤ ∥A∥∥A^{−1}∥∥e∥∥b∥ = κ(A)∥x − x̂∥∥b∥.

Hence

\frac{1}{κ(A)} \frac{∥r∥}{∥b∥} ≤ \frac{∥x − x̂∥}{∥x∥}.  (3.37)

Therefore we have established relationships between the relative error in x and b.

Theorem 3.6

\frac{1}{κ(A)} \frac{∥r∥}{∥b∥} ≤ \frac{∥x − x̂∥}{∥x∥} ≤ κ(A)\frac{∥r∥}{∥b∥}.  (3.38)

Next we further analyze the situation when there are both perturbations in the coefficient matrix and the right-hand-side vector.

Lemma 3.4 Suppose that x and x̃ satisfy

Ax = b  and  (A + △A)x̃ = b + △b,

where A ∈ ℝ^{n×n}, △A ∈ ℝ^{n×n}, 0 ≠ b ∈ ℝ^{n}, and △b ∈ ℝ^{n}, with

\frac{∥△A∥}{∥A∥} ≤ δ  and  \frac{∥△b∥}{∥b∥} ≤ δ.

If κ(A) · δ < 1, then A + △A is nonsingular and

\frac{∥x̃∥}{∥x∥} ≤ \frac{1 + κ(A)δ}{1 − κ(A)δ} .  (3.39)

Proof: Since ∥A^{−1}△A∥ ≤ ∥A^{−1}∥∥△A∥ ≤ δ∥A^{−1}∥∥A∥ = δκ(A) < 1, it follows from Theorem 1.16 that A + △A is nonsingular. Now (A + △A)x̃ = b + △b,

(I + A^{−1}△A)x̃ = A^{−1}b + A^{−1}△b = x + A^{−1}△b,

and so by taking norms and using Theorem 1.15 we find

∥x̃∥ ≤ ∥(I + A^{−1}△A)^{−1}∥ (∥x∥ + ∥A^{−1}∥∥△b∥)
≤ ∥(I + A^{−1}△A)^{−1}∥ (∥x∥ + δ∥A^{−1}∥∥b∥)
≤ \frac{1}{1 − ∥A^{−1}△A∥} (∥x∥ + δ∥A^{−1}∥∥b∥)
≤ \frac{1}{1 − δκ(A)} (∥x∥ + δ∥A^{−1}∥∥b∥)
= \frac{1}{1 − δκ(A)} (∥x∥ + δ∥A^{−1}∥∥Ax∥)
≤ \frac{1}{1 − δκ(A)} (∥x∥ + δ∥A^{−1}∥∥A∥∥x∥)
= \frac{1}{1 − δκ(A)} (∥x∥ + δκ(A)∥x∥)
= \frac{1}{1 − δκ(A)} (1 + δκ(A)) ∥x∥.

Therefore

\frac{∥x̃∥}{∥x∥} ≤ \frac{1 + δκ(A)}{1 − δκ(A)}.

Theorem 3.7 If the conditions of Lemma 3.4 hold then

\frac{∥x − x̃∥}{∥x∥} ≤ \frac{2δ}{1 − κ(A)δ} κ(A)  (3.40)

Proof: Since x̃ satisfies (A + △A)x̃ = b + △b, Ax̃ = b + △b − △Ax̃. Then we have

Ax̃ − Ax = △b + △Ax̃

and

x̃ − x = A^{−1} (△b + △Ax̃) .

Hence

∥x̃ − x∥ ≤ ∥A^{−1}∥ (∥△b∥ + ∥△A∥∥x̃∥)
≤ ∥A^{−1}∥ (δ∥b∥ + δ∥A∥∥x̃∥)
= δ∥A^{−1}∥ (∥Ax∥ + ∥A∥∥x̃∥)
≤ δ∥A∥∥A^{−1}∥ (∥x∥ + ∥x̃∥) ,

which gives

\frac{∥x̃ − x∥}{∥x∥} ≤ δκ(A) (1 + \frac{∥x̃∥}{∥x∥}) ≤ δκ(A) (1 + \frac{1 + κ(A)δ}{1 − κ(A)δ} ) = \frac{2δκ(A)}{1 − δκ(A)}.

68  Chapter 3. Direct Methods for Solving Systems of Linear Equations

Iterative Methods for Solving Systems of Linear Equations

In this chapter we present some iterative methods for solving a linear system of equations Ax = b.

4.1 Classic Iterative Methods

4.1.1 Basic Concept

First of all we give an example to illustrate the process of iterative methods for solving systems of linear equations.

Consider solving

[ \begin{matrix}3 & 2 \\ 1 & 4 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}5 \\ 5 \end{matrix} ] .

This system has the exact solution x_{1} = x_{2} = 1. Equivalently we can write the system as

\begin{cases}3x_{1} + 2x_{2} = 5 \\ x_{1} + 4x_{2} = 5 \end{cases}

This implies that

\begin{cases}x_{1} = \frac{1}{3}(5 − 2x_{2}) \\ x_{2} = \frac{1}{4}(5 − x_{1}) \end{cases}

A naieve idea is to solve the system by

\begin{cases}x_{1}^{(k)} = \frac{1}{3}(5 − 2x_{2}^{(k−1)}) \\ x_{2}^{(k)} = \frac{1}{4}(5 − x_{1}^{(k−1)}) \end{cases}

that is, to use the iterative formulation

[ \begin{matrix}x_{1}^{(k)} \\ x_{2}^{(k)} \end{matrix} ] = [ \begin{matrix}\frac{1}{3} & 0 \\ 0 & \frac{1}{4} \end{matrix} ] ([ \begin{matrix}5 \\ 5 \end{matrix} ] − [ \begin{matrix}0 & 2 \\ 1 & 0 \end{matrix} ] [ \begin{matrix}x_{1}^{(k−1)} \\ x_{2}^{(k−1)} \end{matrix} ])

If we choose the initial guess x_{1}^{(0)} = x_{2}^{(0)} = 0, we would otain

[ \begin{matrix}x_{1}^{(1)} \\ x_{2}^{(1)} \end{matrix} ] = [ \begin{matrix}\frac{1}{3} & 0 \\ 0 & \frac{1}{4} \end{matrix} ] ([ \begin{matrix}5 \\ 5 \end{matrix} ] − [ \begin{matrix}0 & 2 \\ 1 & 0 \end{matrix} ] [ \begin{matrix}0 \\ 0 \end{matrix} ]) = [ \begin{matrix}1.6667 \\ 1.2500 \end{matrix} ]

[ \begin{matrix}x_{1}^{(2)} \\ x_{2}^{(2)} \end{matrix} ] = [ \begin{matrix}\frac{1}{3} & 0 \\ 0 & \frac{1}{4} \end{matrix} ] ([ \begin{matrix}5 \\ 5 \end{matrix} ] − [ \begin{matrix}0 & 2 \\ 1 & 0 \end{matrix} ] [ \begin{matrix}1.6667 \\ 1.2500 \end{matrix} ]) = [ \begin{matrix}0.8333 \\ 0.8333 \end{matrix} ]

By repeating the process, we have the following table

\begin{matrix}k & 3 & 4 & 5 & 6 & 7 \\ x_{1}^{(k)} & 1.1111 & 0.9722 & 1.0185 & 0.9954 & 1.0031 \\ x_{2}^{(k)} & 1.0417 & 0.9722 & 1.0000 & 0.9954 & 1.0012 \end{matrix}

From this example, we observe that the basic idea is to split the coefficient matrix A into

A = M − (M − A),  (4.1)

for some matrix M, which is called the splitting matrix. Here we assume that A and M are both nonsingular. Then the original problem is rewritten in the equivalent form

Mx = (M − A)x + b.

This suggests an iterative process

x^{(k)} = (I − M^{−1}A)x^{(k−1)} + M^{−1}b ≡ T x^{(k−1)} + c,  (4.2)

where T is usually called the iteration matrix. The initial vector x^{(0)} can be arbitrary or be chosen according to certain conditions.

Two criteria for choosing the splitting matrix M are

1. x^{(k)} is easily computed. More precisely, the system Mx^{(k)} = y is easy to solve;

2. the sequence {x^{(k)}} converges rapidly to the exact solution.

Note that one way to achieve the second goal is to choose M so that M^{−1} approximate A^{−1},

In the following subsections, we will introduce some of the mostly commonly used classic iterative methods.

4.1.2 Richard’s Method

When we choose M = I such that A = I − (I − A), we obtain the iteration procedure

x^{(k)} = (I − A)x^{(k−1)} + b = x^{(k−1)} − Ax^{(k−1)} + b ≡ x^{(k−1)} + r^{(k−1)},  (4.3)

This algorithm is called the Richard’s method.

Algorithm 4.1 (Richard’s Method)

for k = 1, 2, … do for i = 1, 2, …, n do

r_{i}^{(k−1)} = b_{i} − ∑_{j=1}^{n} a_{ij}x_{j}^{(k−1)}

x_{i}^{(k)} = x_{i}^{(k−1)} + r_{i}^{(k−1)} end for end for

Two n-vectors are required to implement this algorithm.

4.1.3 Jacobi Method

If we decompose the coefficient matrix A as

A = L + D + U,  (4.4)

where D is the diagonal part, L is the strictly lower triangular part, and U is the strictly upper triangular part, of A, and choose M = D, then we derive the iterative formulation for Jacobi method:

x^{(k)} = −D^{−1}(L + U)x^{(k−1)} + D^{−1}b.  (4.5)

With this method, the iteration matrix T = −D^{−1}(L + U) and c = D^{−1}b. Each component x_{i}^{(k)} can be computed by

x_{i}^{(k)} = (b_{i} − ∑_{j=1}^{i−1} a_{ij}x_{j}^{(k−1)} − ∑_{j=i+1}^{n} a_{ij}x_{j}^{(k−1)}) /a_{ii}.  (4.6)

Algorithm 4.2 (Jacobi Method)

for k = 1, 2, … do for i = 1, 2, …, n do

x_{i}^{(k)} = (b_{i} − ∑_{j=1}^{i−1} a_{ij}x_{j}^{(k−1)} − ∑_{j=i+1}^{n} a_{ij}x_{j}^{(k−1)}) /a_{ii}

end for end for

Two n-vectors are required to implement this algorithm. A big advantage of Jacobi method is that x_{i}^{(k)}, i = 1, …, n, can be computed in parallel at each iteration k.

Another important observation is that in Jacobi method, only the components of x^{(k−1)} are used to compute x^{(k)}. When computing x_{i}^{(k)} for i > 1, x_{1}^{(k)}, …, x_{i−1}^{(k)} have already been computed and are likely to be better approximations to the exact x_{1}, …, x_{i−1} than x_{1}^{(k−1)}, …, x_{i−1}^{(k−1)}. It seems reasonable to compute x_{i}^{(k)} using these most recently computed values. This improvement induce the Gauss-Seidel method which is to be discussed in the next subsection.

4.1.4 Gauss-Seidel Method

The Gauss-Seidel method sets M = D + L and defines the iteration as

x^{(k)} = −(D + L)^{−1}Ux^{(k−1)} + (D + L)^{−1}b.  (4.7)

That is, Gauss-Seidel method uses T = −(D + L)^{−1}U as the iteration matrix. The formulation above can be rewritten as

x^{(k)} = −D^{−1} (Lx^{(k)} + Ux^{(k−1)} − b) .  (4.8)

Hence each component x_{i}^{(k)} can be computed by

x_{i}^{(k)} = (b_{i} − ∑_{j=1}^{i−1} a_{ij}x_{j}^{(k)} − ∑_{j=i+1}^{n} a_{ij}x_{j}^{(k−1)}) /a_{ii}.  (4.9)

Algorithm 4.3 (Gauss-Seidel Method)

for k = 1, 2, … do for i = 1, 2, …, n do

x_{i}^{(k)} = (b_{i} − ∑_{j=1}^{i−1} a_{ij}x_{j}^{(k)} − ∑_{j=i+1}^{n} a_{ij}x_{j}^{(k−1)}) /a_{ii}

end for end for

At each iteration, since x_{i}^{(k)} can not be computed until x_{1}^{(k)}, …, x_{i−1}^{(k)} are available, Gauss-Seidel method is not a parallel algorithm in nature. Moreover, only one n-vector is required for implementation in theory, but two are usually used in practice.

4.1.5 Successive Over Relaxation (SOR) Method

The successive over relaxation (SOR) method choose M = ω^{−1}(D + ωL), where 0 < ω < 2 is called the relaxation parameter, and defines the iteration

(D + ωL)x^{(k)} = [(1 − ω)D − ωU] x^{(k−1)} + ωb.  (4.10)

Hence the iteration matrix T = (D + ωL)^{−1}((1 − ω)D − ωU). Each component x_{i}^{(k)} can be computed by the formulation

x_{i}^{(k)} = ω (b_{i} − ∑_{j=1}^{i−1} a_{ij}x_{j}^{(k)} − ∑_{j=i+1}^{n} a_{ij}x_{j}^{(k−1)}) /a_{ii} + (1 − ω)x_{i}^{(k−1)}.  (4.11)

The question of choosing a good relaxation parameter ω is a very complex topic.

4.1.6 Symmetric Successive Over Relaxation (SSOR) Method

In theory the symmetric successive over relaxation (SSOR) method chooses the splitting matrix M = \frac{1}{ω(2−ω)}(D + ωL)D^{−1}(D + ωU) and iterates with the iteration matrix

T = (D + ωU)^{−1} ((1 − ω)D − ωL) (D + ωL)^{−1} ((1 − ω)D − ωU) .  (4.12)

The idea is in fact to implement the SOR formulation twice, one forward and one backward, at each iteration. That is, SSOR method defines

(D + ωL)x^{(k− \frac{1}{2} )} = ((1 − ω)D − ωU) x^{(k−1)} + ωb  (4.13)
(D + ωU)x^{(k)} = ((1 − ω)D − ωL) x^{(k− \frac{1}{2} )} + ωb  (4.14)

Each component x_{i}^{(k)} is obtained by first computing

x_{i}^{(k− \frac{1}{2} )} = ω (b_{i} − ∑_{j=1}^{i−1} a_{ij}x_{j}^{(k− \frac{1}{2} )} − ∑_{j=i+1}^{n} a_{ij}x_{j}^{(k−1)}) /a_{ii} + (1 − ω)x_{i}^{(k)}  (4.15)

followed by

x_{i}^{(k)} = ω (b_{i} − ∑_{j=1}^{i−1} a_{ij}x_{j}^{(k)} − ∑_{j=i+1}^{n} a_{ij}x_{j}^{(k− \frac{1}{2} )}) /a_{ii} + (1 − ω)x_{i}^{(k− \frac{1}{2} )}.  (4.16)

4.2 Convergence Analysis

Definition 4.1 (Eigenvalue) Suppose A is a square matrix. The number λ is called an eigenvalue of A if there exists a nonzero vector x such that Ax = λx. And x is called the corresponding eigenvector of A.

Definition 4.2 The polynomial defined by

p(λ) = det(λI − A)

is called the characteristic polynomial of A.

Theorem 4.1 λ is an eigenvalue of A if and only if λ is a root of the characteristic polynomial of A, that is, p(λ) = 0.

Definition 4.3 (Spectrum and Spectral Radius) The set of all eigenvalues of a matrix A is called the spectrum of A and is denoted by λ(A). The spectral radius of A is

ρ(λ) = max{|λ||λ ∈ λ(A)}.  (4.17)

Lemma 4.1 If A ∈ ℝ^{n×n}, then

1. ∥A∥_{2} = \sqrt{ρ(A^{T} A)};

2. ρ(A) ≤ ∥A∥ for any subordinate matrix norm.

Proof: Proof for the second part. Suppose λ is an eigenvalue of A and x ≠ 0 is a corresponding eigenvector such that Ax = λx and ∥x∥ = 1. Then

|λ| = |λ|∥x∥ = ∥λx∥ = ∥Ax∥ ≤ ∥A∥∥x∥ = ∥A∥,

that is, |λ| ≤ ∥A∥. Since λ is arbitrary, this implies that ρ(A) = max |λ| ≤ ∥A∥.

Theorem 4.2 For any A and any ϵ > 0, there exists a subordinate norm such that

ρ(A) < ∥A∥ < ρ(A) + ϵ.  (4.18)

Lemma 4.2 If ρ(A) < 1, then (I − A)^{−1} exists and (I − A)^{−1} = ∑_{i=0}^{∞} A^{i} = I + A + A^{2} + · · ·.

Proof: Suppose that λ is an eigenvalue of A, then 1 − λ is an eigenvalue of I − A. Since |λ| ≤ ρ(A) < 1, this implies 1 − λ ≠ 0. Hence 0 is not an eigenvalue of I − A, which means (I − A) is nonsingular.

Next we show that (I − A)^{−1} = I + A + A^{2} + · · ·. Since

(I − A) (∑_{i=0}^{m} A^{i}) = I − A^{m+1},

and ρ(A) < 1 implies ∥A^{m}∥ → 0 as m → ∞, we have

(I − A) ( lim_{m→∞} ∑_{i=0}^{m} A^{i}) = I.

This proves (I − A)^{−1} = ∑_{k=1}^{∞} A^{k}.

Lemma 4.3 Suppose that A ∈ ℝ^{n×n} and ∥ · ∥ is a subordinate matrix norm. If ∥A∥ < 1, then I − A is nonsingular and

(I − A)^{−1} = ∑_{k=0}^{∞} A^{k},  (4.19)

with

∥(I − A)^{−1}∥ ≤ \frac{1}{1 − ∥A∥}.  (4.20)

Proof: Suppose I − A is singular. Then there exist x ∈ ℝ^{n}, x ≠ 0 (so ∥x∥ ≠ 0) such that (I − A)x = 0. Thus x = Ax and ∥x∥ = ∥Ax∥ ≤ ∥A∥∥x∥, which gives ∥A∥ > 1. However, this contradicts to the assumption that ∥A∥ < 1. Hence I − A is nonsingular.

4.2 Convergence Analysis  75

Next, one can verify that

(I − A) (∑_{k=0}^{m} A^{k}) = I − A^{m+1}.

Since ∥A∥ < 1, lim_{m→∞} A^{m} = 0, hence

(I − A) (∑_{k=0}^{∞} A^{k}) = (I − A) ( lim_{m→∞} ∑_{k=0}^{m} A^{k}) = I − lim_{m→∞} A^{m+1} = I.

This shows that (I − A)^{−1} = ∑_{k=0}^{∞} A^{k}. Finally, since ∥A∥ < 1,

∥(I − A)^{−1}∥ = ∥∑_{k=0}^{∞} A^{k}∥ ≤ ∑_{k=0}^{∞} ∥A^{k}∥ ≤ ∑_{k=0}^{∞} ∥A∥^{k} = \frac{1}{1 − ∥A∥}.

Theorem 4.3 The following statements are equivalent.

1. A is a convergent matrix, i.e., A^{k} → ∞ as k → ∞;

2. lim_{k→∞} ∥A^{k}∥ = 0 for some subordinate matrix norm;

3. lim_{k→∞} ∥A^{k}∥ = 0 for all subordinate matrix norm;

4. ρ(A) < 1;

5. lim_{k→∞} A^{k}x = 0 for any x.

Theorem 4.4 For any x^{(0)} ∈ ℝ^{n}, the sequence produced by

x^{(k)} = T x^{(k−1)} + c, k = 1, 2, …,  (4.21)

converges to the unique solution of x = T x + c if and only if

ρ(T) < 1.  (4.22)

Proof: Suppose ρ(T) < 1. The sequence of vectors x^{(k)} produced by the iterative formulation are

x^{(1)} = T x^{(0)} + c
x^{(2)} = T x^{(1)} + c = T^{2}x^{(0)} + (T + I)c
x^{(3)} = T x^{(2)} + c = T^{3}x^{(0)} + (T^{2} + T + I)c
⋮

76  Chapter 4. Iterative Methods for Solving Systems of Linear Equations

In general

x^{(k)} = T^{k}x^{(0)} + (T^{k−1} + T^{k−2} + ⋯ T + I)c.

Since ρ(T) < 1, lim_{k=0}^{∞} T^{k}x^{(0)} = 0 for any x^{(0)} ∈ ℝ^{n}. Then

x^{(k)} → (T^{k−1} + T^{k−2} + ⋯ T + I)c, as k → ∞.

By Lemma

x^{(k)} → (I − T)^{−1}c as k → ∞.

Conversely, suppose that the sequence of vectors x^{(k)} converges to x = (I − T)^{−1}c. Since x − x^{(k)} = T x + c − T x^{(k−1)} − c = T (x − x^{(k−1)}) = T^{2}(x − x^{(k−2)}) = ⋯ = T^{k}(x − x^{(0)}).

Let z = x − x^{(0)}. Then

lim_{k→∞} T^{k}z lim_{k→∞}(x − x^{(k)}) = 0.

It follows from theorem ρ(T) < 1. Corollary 4.1 If ∥T ∥ < 1 for some subordinate matrix norm, then the sequence produced by

x^{(k)} = T x^{(k−1)} + c

converges to the solution of Ax = b for any initial vector x^{(0)}. Proof: Since ρ(T) < ∥T ∥ for any subordinate matrix norm, the result follows immediately from the previous theorem. Theorem 4.5 If δ = ∥T ∥ < 1, then

∥x^{(k)} − x∥ ≤ \frac{δ}{1 − δ} ∥x^{(k)} − x^{(k−1)}∥.  (4.23)

Proof: Since x^{(k)} − x = T (x^{(k−1)} − x), ∥x^{(k)} − x∥ ≤ ∥T ∥∥x^{(k−1)} − x∥ = δ∥x^{(k−1)} − x^{(k)} + x^{(k)} − x∥ ≤ δ∥x^{(k−1)} − x^{(k)}∥ + δ∥x^{(k)} − x∥, and 1 − δ > 0, we obtain

∥x^{(k)} − x∥ ≤ \frac{δ}{1 − δ} ∥x^{(k)} − x^{(k−1)}∥.

This theorem implies that we can stop the iteration if ∥x^{(k)} − x^{(k−1)} is less than a small tolerance. Theorem 4.6 If ∥T ∥ < 1, then the sequence x^{(k)} converges to x for any initial x^{(0)} and

1. ∥x − x^{(k)}∥ ≤ ∥T ∥^{k}∥x − x^{(0)}∥

2. ∥x − x^{(k)}∥ ≤ \frac{∥T ∥^{k}}{1−∥T ∥}∥x^{(1)} − x^{(0)}∥.

Proof: Since x = T x + c and x^{(k)} = T x^{(k−1)} + c,

x − x^{(k)} = T x + c − T x^{(k−1)} − c
= T (x − x^{(k−1)})
= T^{2}(x − x^{(k−2)}) = · · · · · · = T^{k}(x − x^{(0)}).

The first statement can then be derived

∥x − x^{(k)}∥ = ∥T^{k}(x − x^{(0)})∥ ≤ ∥T ∥^{k}∥x − x^{(0)}∥.

For the second result, we first show that ∥x^{(n)} − x^{(n−1)}∥ ≤ ∥T ∥^{n−1}∥x^{(1)} − x^{(0)}∥ for any n ≥ 1. Since

x^{(n)} − x^{(n−1)} = T x^{(n−1)} + c − T x^{(n−2)} − c
= T (x^{(n−1)} − x^{(n−2)})
= T^{2}(x^{(n−2)} − x^{(n−3)}) = · · · · · · = T^{n−1}(x^{(1)} − x^{(0)}),
∥x^{(n)} − x^{(n−1)}∥ ≤ ∥T ∥^{n−1}∥x^{(1)} − x^{(0)}∥.

Let m ≥ k,

x^{(m)} − x^{(k)} = (x^{(m)} − x^{(m−1)}) + (x^{(m−1)} − x^{(m−2)}) + ⋯ + (x^{(k+1)} − x^{(k)})
= T^{m−1} (x^{(1)} − x^{(0)}) + T^{m−2} (x^{(1)} − x^{(0)}) + ⋯ + T^{k} (x^{(1)} − x^{(0)})
= (T^{m−1} + T^{m−2} + ⋯ T^{k}) (x^{(1)} − x^{(0)}) ,

hence

∥x^{(m)} − x^{(k)}∥ ≤ (∥T ∥^{m−1} + ∥T ∥^{m−2} + ⋯ + ∥T ∥^{k}) ∥x^{(1)} − x^{(0)}∥
= ∥T ∥^{k} (∥T ∥^{m−k−1} + ∥T ∥^{m−k−2} + ⋯ + 1) ∥x^{(1)} − x^{(0)}∥.

Since lim_{m→∞} x^{(m)} = x,

∥x − x^{(k)}∥ = lim_{m→∞} ∥x^{(m)} − x^{(k)}∥
≤ lim_{m→∞} ∥T ∥^{k} (∥T ∥^{m−k−1} + ∥T ∥^{m−k−2} + ⋯ + 1) ∥x^{(1)} − x^{(0)}∥
= ∥T ∥^{k}∥x^{(1)} − x^{(0)}∥ lim_{m→∞} (∥T ∥^{m−k−1} + ∥T ∥^{m−k−2} + ⋯ + 1)
= ∥T ∥^{k} \frac{1}{1 − ∥T ∥}∥x^{(1)} − x^{(0)}∥.

This proves the second result.

Theorem 4.7 If A is strictly diagonal domainant, then both the Jacobi and Gauss-Seidel methods converges for any initial vector x^{(0)}.

Proof: By assumption, A is strictly diagonal dominant, hence a_{ii} ≠ 0 (otherwise A is singular) and

|a_{ii}| > ∑_{j=1,j≠i}^{n} |a_{ij}|, i = 1, 2, …, n.

For Jacobi method, the iteration matrix T_{J} = −D^{−1}(L + U) has entries

[T_{J}]_{ij} = \begin{cases}−\frac{a_{ij}}{a_{ii}} & i ≠ j \\ 0 & i = j \end{cases}

Hence

∥T_{J}∥_{∞} = max_{1≤i≤n} | ∑_{j=1,j≠i}^{n} \frac{a_{ij}}{a_{ii}} | = max_{1≤i≤n} \frac{1}{|a_{ii}|} ∑_{j=1,j≠i}^{n} |a_{ij}| < 1,

and this implies that the Jacobi method converges.

For Gauss-Seidel method, the iteration matrix T_{GS} = −(D + L)^{−1}U. Let λ be any eigenvalue of T_{GS} and y, ∥y∥_{∞} = 1, is a corresponding eigenvector. Thus

T_{GS}y = λy ⟹ −Uy = λ(D + L)y.

Hence for i = 1, …, n,

− ∑_{j=i+1}^{n} a_{ij}y_{j} = λ ∑_{j=1}^{i} a_{ij}y_{j} = λa_{ii}y_{i} + ∑_{j=1}^{i−1} a_{ij}y_{j}.

This gives

λa_{ii}y_{i} = −λ ∑_{j=1}^{i−1} a_{ij}y_{j} + ∑_{j=i+1}^{n} a_{ij}y_{j}

and

|λ||a_{ii}||y_{i}| ≤ |λ| ∑_{j=1}^{i−1} |a_{ij}||y_{j}| + ∑_{j=i+1}^{n} |a_{ij}||y_{j}|.

Choose the index k such that |y_{k}| = 1 ≥ |y_{j}| (this index can always be found since ∥y∥_{∞} = 1). Then

|λ||a_{kk}| ≤ |λ| ∑_{j=1}^{k−1} |a_{kj}| + ∑_{j=k+1}^{n} |a_{kj}|

which gives

|λ| ≤ \frac{∑_{j=k+1}^{n} |a_{kj}|}{|a_{kk}| − ∑_{j=1}^{k−1} |a_{kj}|} < \frac{∑_{j=k+1}^{n} |a_{kj}|}{∑_{j=k+1}^{n} |a_{kj}|} = 1

Since λ is arbitrary, ρ(T_{GS}) < 1. This means the Gauss-Seidel method converges.

Theorem 4.8 If A is positive definite and the relaxation parameter ω satisfying 0 < ω < 2, then the SOR iteration converges for any initial vector x^{(0)}.

Theorem 4.9 If A is positive definite and tridiagonal, then ρ(T_{GS}) = [ρ(T_{J})]^{2} < 1 and the optimal choice of ω for the SOR iteration is

ω = \frac{2}{1 + \sqrt{1 − [ρ(T_{J})]^{2}}} .  (4.24)

With this choice of ω, ρ(T_{SOR}) = ω − 1.

80  Chapter 4. Iterative Methods for Solving Systems of Linear Equations

Solutions of Non-linear Equations

This chapter is devoted to the problem of determining roots of nonlinear equations (zeros of functions). It is a problem of frequent occurence in scientific work. The general question, posed in the simplest case of a real-valued function of one real variable, is this: Given a function f : ℝ → ℝ, find x ∈ ℝ such thatf(x) = 0.

The extension of the simple formulation is a system of nonlinear equations of the form

{ f_{1}(x_{1}, x_{2}, …, x_{n}) = 0 ; f_{2}(x_{1}, x_{2}, …, x_{n}) = 0 ; ⋮ ; f_{n}(x_{1}, x_{2}, …, x_{n}) = 0

where x_{i} ∈ ℝ are real variables and each f_{i} is a function from ℝ^{n} to ℝ. To simplify the representation, we let

x = [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] ∈ ℝ^{n}

and F : ℝ^{n} → ℝ^{n} be the function

F (x) = F (x_{1}, x_{2}, …, x_{n}) = [ f_{1}(x_{1}, x_{2}, …, x_{n}) ; f_{2}(x_{1}, x_{2}, …, x_{n}) ; ⋮ ; f_{n}(x_{1}, x_{2}, …, x_{n}) ] .

Then the problem of solving system of nonlinear equations becomes finding x ∈ ℝ^{n} such that F (x) = 0.

5.1 Preliminaries

Definition 5.1 Let {x_{n}} be a sequence of real numbers that converges to x^{∗}. We say that the rate of convergence is 1. linear if there exist a constant 0 < c < 1 and an integer N > 0 such that

|x_{n+1} − x^{∗}| ≤ c|x_{n} − x^{∗}|, ∀ n ≥ N;  (5.1)

2. superlinear if there exist a sequence {c_{n}}, c_{n} → 0 as n → ∞, and an integer N > 0

such that

|x_{n+1} − x^{∗}| ≤ c_{n}|x_{n} − x^{∗}|, ∀ n ≥ N,  (5.2)

or, equivalently,

lim_{n→∞} \frac{|x_{n+1} − x^{∗}|}{|x_{n} − x^{∗}|} = 0;  (5.3)

3. quadratic if there exist a constant c > 0 (not necessarily less than 1) and an integer

N > 0 such that

|x_{n+1} − x^{∗}| ≤ c|x_{n} − x^{∗}|^{2}, ∀ n ≥ N.  (5.4)

In general, if there are positive constants c and α and an integer N > 0 such that

|x_{n+1} − x^{∗}| ≤ c|x_{n} − x^{∗}|^{α}, ∀ n ≥ N,  (5.5)

then we say the rate of convergence is of order α.

Definition 5.2 Suppose {β_{n}} is a sequence known to converge to zero and {x_{n}} converge s to x^{∗}. If there exist a postive constant c and an integer N > 0 such that

|x_{n} − x^{∗}| ≤ c|β_{n}|, ∀ n ≥ N,  (5.6)

then we say {x_{n}} converges to x^{∗} with rate of convergence O(β_{n}), and write x_{n} = x^{∗} +O(β_{n}).

Example 5.1 Compare the convergence behavior of the sequences {x_{n}} and {y_{n}}, where

x_{n} = \frac{n + 1}{n^{2}}, and y_{n} = \frac{n + 3}{n^{3}}.

Sol: Note that both

lim_{n→∞} x_{n} = 0 and lim_{n→∞} y_{n} = 0.

Let α_{n} = \frac{1}{n} and β_{n} = \frac{1}{n^{2}} . Then

|x_{n} − 0| = \frac{n + 1}{n^{2}} ≤ \frac{n + n}{n^{2}} = \frac{2}{n} = 2α_{n},
|y_{n} − 0| = \frac{n + 3}{n^{3}} ≤ \frac{n + 3n}{n^{3}} = \frac{4}{n^{2}} = 4β_{n}.

Hence

x_{n} = 0 + O(\frac{1}{n}) and y_{n} = 0 + O(\frac{1}{n^{2}} ).

This shows that {y_{n}} converges to 0 much faster than {x_{n}}.

5.2 Bisection Method

The bisection method is based on the intermediate value theorem. The idea behind the method is that if f(x) ∈ C[a, b] and f(a)f(b) < 0, then there exists a root c ∈ (a, b) such that f(c) = 0.

Algorithm 5.1 (Biesection Method) Given a function f(x) defined on an interval (a, b), the maximal number of iterations M, and stop criteria δ and ϵ, this algorithm tries to locate one root of f(x).

compute u = f(a), v = f(b), and e = b − a. if sign(u) = sign(v) then

stop end if for k = 1, 2, …, M do

e = e/2 c = a + e w = f(c) if |e| < δ or |w| < ϵ then

stop end if if sign(w) ≠ sign(u) then

b = c v = w

else a = c u = w end if end for

Let c_{1}, c_{2}, … be the sequence of numbers produced. The algorithm should stop if one of the following conditions is satisfied.

1. the iteration number k > M,

2. |c_{k} − c_{k−1}| < δ, or

3. |f(c_{k})| < ϵ.

In fact, these criteria can be applied to any iterative techniques considered in this chapter. Also note that the algorithm computes c = a + (b − a)/2, instead of c = (a + b)/2, for numerical reason. Furthermore, the bisection method find one zero, but not all the zeros, in the given interval [a, b].

Let [a_{0}, b_{0}], [a_{1}, b_{1}], … denote the successive intervals produced by the bisection algorithm. Then

a = a_{0} ≤ a_{1} ≤ a_{2} ≤ ⋯ ≤ b_{0} = b
b = b_{0} ≥ b_{1} ≥ b_{2} ≥ ⋯ ≥ a_{0} = a.

This means that the sequences {a_{n}} and {b_{n}} are bounded. Hence lim_{n→∞} a_{n} and lim_{n→∞} b_{n} exist. Since

b_{1} − a_{1} = \frac{1}{2}(b_{0} − a_{0})
b_{2} − a_{2} = \frac{1}{2}(b_{1} − a_{1}) = \frac{1}{4}(b_{0} − a_{0})
⋮
b_{n} − a_{n} = \frac{1}{2^{n}} (b_{0} − a_{0})

hence

lim_{n→∞} b_{n} − lim_{n→∞} a_{n} = lim_{n→∞}(b_{n} − a_{n}) = lim_{n→∞} \frac{1}{2^{n}} (b_{0} − a_{0}) = 0.

Therefore

lim_{n→∞} a_{n} = lim_{n→∞} b_{n} ≡ z.

Since f is a continuous function

lim_{n→∞}f(a_{n}) = f( lim_{n→∞} a_{n}) = f(z) and lim_{n→∞}f(b_{n}) = f( lim_{n→∞} b_{n}) = f(z).

The bisection method ensures that f(a_{n})f(b_{n}) ≤ 0. This implies that lim_{n→∞} f(a_{n})f(b_{n}) =

f^{2}(z) ≤ 0., and, consequently, f(z) = 0. That is, the limit of the sequences {a_{n}} and {b_{n}} is a zero of f in [a, b].

Let c_{n} = \frac{1}{2}(a_{n} + b_{n}). Then

|z − c_{n}| = | lim_{n→∞} a_{n} − \frac{1}{2}(a_{n} + b_{n})| ≤ |b_{n} − \frac{1}{2}(a_{n} + b_{n})| = \frac{1}{2}|b_{n} − a_{n}| = \frac{1}{2^{n+1}} |b_{0} − a_{0}|.

This proves the following theorem.

Theorem 5.1 Let [a_{0}, b_{0}], [a_{1}, b_{1}], … denote the intervals produced by the bisection algorithm. Then lim_{n→∞} a_{n} and lim_{n→∞} b_{n} exist, are equal, and represent a zero of f(x). If z =

lim_{n→∞} a_{n} = lim_{n→∞} b_{n} and c_{n} = \frac{1}{2}(a_{n} + b_{n}), then

|z − c_{n}| ≤ \frac{1}{2^{n+1}} (b_{0} − a_{0}) .

Remarks 5.1 The implication of the theorem above is that {c_{n}} converges to z with the rate of O(2^{−n}).

Example 5.2 If bisection method starts with interval [50, 75], then how many steps should be taken to compute a root with relative error that is less than 10^{−12}?

Sol: We want to seek an n such that

\frac{|z − c_{n}|}{|z|} ≤ 10^{−12}.

Since the bisection method starts with the interval [50, 75], this implies that z ≥ 50, hence it is sufficient to show

\frac{|z − c_{n}|}{|z|} ≤ \frac{|z − c_{n}|}{50} ≤ 10^{−12}.

That is, we solve

2^{−(n+1)}(75 − 50) ≤ 50 × 10^{−12}

for n, which gives n ≥ 38.

5.3 Newton’s Method

5.3.1 Derivation of Newton’s Method

Newton-Raphson’s method is widely applied to problems of finding a zero of a function. It is simply called Newton’s method. Suppose that f : ℝ → ℝ and f ∈ C^{2}[a, b], i.e., f” exists and is continuous. If x_{∗} is a zero of f and x is a point close to x_{∗} such that x_{∗} = x + h, then by Taylor’s theorem

0 = f(x_{∗}) = f(x + h) = f(x) + f′(x)h + \frac{1}{2}f”(x)h^{2} + \frac{1}{3!}f′′′(x)h^{3} + ⋯
= f(x) + f′(x)h + O(h^{2}).

Since we assume that x is close to x_{∗}, i.e., h is small, O(h^{2}) is negligible. It is reasonable to drop O(h^{2}) terms. This implies

f(x) + f′(x)h ≈ 0 and h ≈ −\frac{f(x)}{f′(x)},

if f′(x) ≠ 0. Hence

x + h = x − \frac{f(x)}{f′(x)}

is a better approximation to x_{∗}. This sets the stage for the Newton-Raphson’s method, which starts with an initial approximation x_{0} and generates the sequence {x_{k}}_{k=0}^{∞} defined by

x_{k+1} = x_{k} − \frac{f(x_{k})}{f′(x_{k})}.  (5.7)

Figure gives a graphic interpretation of the Newon’s method. Since the Taylor’s expansion of f(x) at x_{k} is given by

f(x) = f(x_{k}) + f′(x_{k})(x − x_{k}) + \frac{1}{2}f”(x_{k})(x − x_{k})^{2} + · · · .

At x_{k}, one uses the tangent line

y = ℓ(x) = f(x_{k}) + f′(x_{k})(x − x_{k})

to approximate the curve of f(x) and uses the zero of the tangent line to approximate the zero of f(x), which gives the formulation, and tehn uses this point, x_{k+1}, as a new starting point to repeat the process. Hence the Newton’s method involving linearizing the function, that is, the function f(x) is replaced by a linear function ℓ(x) such that ℓ(x_{k}) = f(x_{k}) and ℓ′(x_{k}) = f′(x_{k}).

Algorithm 5.2 (Newton’s Method) Given a function f : ℝ → ℝ, an initial guess x_{0} to the zero of f, and stop criteria M, δ, and ϵ, this algorithm performs the Newton’s iteration to approximate one root of f.

u = f(x_{0}) v = f′(x_{0}) x_{1} = x_{0} − \frac{u}{v} k = 1 u = f(x_{k}) while (k < M) or (|x_{k} −x_{k−1}| < δ) or (|f(x_{k})| < ϵ do

v = f′(x_{k}) x_{k+1} = x_{k} − \frac{u}{v} k = k + 1 u = f(x_{k}) end while

Note that there are two function evaluations, f(x_{k}) and f′(x_{k}), in each Newton iteration. The stop criteria δ and ϵ are usually chosen to be a constant times of square root of the machine epsilon, i.e., \sqrt{ϵ_{M}}. Deciding when to stop is somewhat difficult, and can not be perfect for every problem, yet it needs some careful attention.

5.3.2 Convergence Analysis

Suppose that f′′ is continuous and x_{∗} is a simple zero of f, i.e., f(x_{∗}) = 0 but f′(x_{∗}) ≠ 0. Choose δ > 0 and let

D = {x||x − x_{∗}| ≤ δ}

and

γ = \frac{1}{2} · \frac{max_{x∈D} |f′′(x)|}{min_{x∈D} |f′(x)|}.

Note that it is possible to choose δ such that ρ = δγ < 1 since f′′ is continuous and when δ → 0, x → x_{∗} and γ → \frac{1}{2} · \frac{f′′(x_{∗})}{f′(x_{∗})}.

Now suppose we start Newton’s iteration with x_{0} satisfying |e_{0}| = |x_{0} − x_{∗}| ≤ δ. By Taylor’s theorem

0 = f(x_{∗}) = f(x_{0}) + f′(x_{0})(x_{∗} − x_{0}) + \frac{1}{2}f′′(ξ_{0})(x_{∗} − x_{0})^{2},

where ξ_{0} is between x_{∗} and x_{0}. Consequently |ξ_{0} − x_{∗}| ≤ δ and

−f(x_{0}) − f′(x_{0})(x_{∗} − x_{0}) = \frac{1}{2}f′′(ξ_{0})(x_{∗} − x_{0})^{2}.

One iteration of Newton’s algorithm gives

x_{1} = x_{0} − \frac{f(x_{0}}{f′(x_{0})}.

Hence

e_{1} = x_{1} − x_{∗} = x_{0} − \frac{f(x_{0})}{f′(x_{0})} − x_{∗}
= \frac{1}{f′(x_{0})} (−f(x_{0}) − x_{∗}f′(x_{0}) + x_{0}f′(x_{0}))
= \frac{1}{f′(x_{0})} (−f(x_{0}) − f′(x_{0})(x_{∗} − x_{0}))
= \frac{1}{f′(x_{0})} · \frac{1}{2} · f′′(ξ_{0})(x_{∗} − x_{0})^{2}
= \frac{f′′(ξ_{0})}{2f′(x_{0})}e_{0}^{2},

and

|e_{1}| ≤ γ|e_{0}|^{2} ≤ γδ|e_{0}| = ρ|e_{0}|.

Repeat the argument, we have, in general,

|e_{k}| ≤ ρ|e_{k−1}| ≤ ⋯ ≤ ρ^{k}|e_{0}|.

Since ρ < 1, |e_{k}| → 0 as k → 0, that is, x_{k} → x_{∗}.

In summary, Newton’s method will generate a seqence of numbers {x_{k}}_{k≥0} that converges to the zero, x_{∗}, of f if

1. f′′ is continuous;

2. x_{∗} is a simple zero of f; and

3. x_{0} is close enough to x_{∗}.

To investigate the convergence rate, we start with

x_{k+1} = x_{k+1} − x_{∗} = x_{k} − \frac{f(x_{k})}{f′(x_{k})} − x_{∗} = \frac{f′(x_{k})e_{k} − f(x_{k})}{f′(x_{k})}.

Using Taylor’s theorem

0 = f(x_{∗}) = f(x_{k} − e_{k}) = f(x_{k}) − f′(x_{k})e_{k} + \frac{1}{2}f′′(ξ_{k})e_{k}^{2},

where ξ_{k} is between x_{k} and x_{∗}, one has

f′(x_{k})e_{k} − f(x_{k}) = \frac{1}{2}f′′(ξ_{k})e_{k}^{2}.

Hence

|e_{k+1}| = \frac{|f′′(ξ_{k})|}{2|f′(x_{k})|}|e_{k}|^{2} ≈ \frac{|f′′(x_{∗})|}{2|f′(x_{∗})|}|e_{k}|^{2} ≡ C|e_{k}|^{2}.

This shows that Newton’s method is quadratic convergent.

Theorem 5.2 Suppose f′′ is continous and x_{∗} is a simple zero of f. Then there exists a neighborhood D of x_{∗} and a constant C such that when Newton’s methdo is applied with starting point x_{0} ∈ D, the seqence {x_{k}} generated converges to x_{∗} and satisfies

|x_{k+1} − x_{∗}| ≤ C|x_{k} − x_{∗}|.  (5.8)

Definition 5.3 (Lipschitz Continuous) A function f(x) is Lipschitz continuous with Lipschitz contant γ in a set X, written f ∈ Lip_{γ}(X), if

|f(x) − f(y)| ≤ γ|x − y|,

for all x, y ∈ X.

Lemma 5.1 Suppose f : Ω → ℝ for some open interval Ω ⊆ ℝ and f′ ∈ Lip_{γ}(Ω). Then for all x, y ∈ Ω,

|f(y) − f(x) − f′(x)(y − x)| ≤ \frac{γ}{2} (y − x)^{2}.  (5.9)

Proof: From Calculus

f(y) − f(x) − f′(x)(y − x) = ∫_{x}^{y}(f′(u) − f′(x)) du.

Apply changing variable u = x + t(y − x), du = (y − x) dt and the fact that f′ ∈ Lip_{γ}(Ω), we have

|f(y) − f(x) − f′(x)(y − x)| = |∫_{0}^{1} (f′(x + t(y − x)) − f′(x)) (y − x) dt|
≤ |y − x| ∫_{0}^{1} γ|t(y − x)| dt
= \frac{γ}{2} |y − x|^{2}.

Theorem 5.3 Let f : Ω → ℝ for some open interval Ω ⊆ ℝ. Assume

1. exists x_{∗} ∈ Ω such that f(x_{∗}) = 0;

2. f′ ∈ Lip_{γ}(Ω);

3. ∃ ρ > 0 such that |f′(x)| ≥ ρ for all x ∈ Ω, that is, f′(x) ≠ 0 for all x ∈ Ω.

Then there exists η > 0 such that if |x_{0} − x_{∗}| < η, then the sequence produced by Newton’s iteration

x_{k+1} = x_{k} − \frac{f(x_{k})}{f′(x_{k})}, k = 0, 1, 2, …,

exists and converges to x_{∗}. Furthermore,

|x_{k+1} − x_{∗}| ≤ \frac{γ}{2ρ}|x_{k} − x_{∗}|^{2}  (5.10)

or, equivalently,

|e_{k+1}| ≤ \frac{γ}{2ρ}|e_{k}|^{2}  (5.11)

\begin{matrix}x_{0} & 2.0 \\ x_{1} & 1.25 \\ x_{2} & 1.025 \\ x_{3} & 1.0003048780488 \\ x_{4} & 1.0000000464611 \\ x_{5} & 1.0 \end{matrix}

Proof: With the result of previous Lemma,

|x_{k+1} − x_{∗}| = |x_{k} − \frac{f(x_{k})}{f′(x_{k})} − x_{∗}|
= |x_{k} − x_{∗} − \frac{f(x_{k}) − f(x_{∗})}{f′(x_{k})}|
= \frac{1}{|f′(x_{k})|} |f(x_{∗}) − f(x_{k}) − f′(x_{k})(x_{k} − x_{∗})|
≤ \frac{γ}{2|f′(x_{k})|}|x_{k} − x_{∗}|^{2}
≤ \frac{γ}{2ρ}|x_{k} − x_{∗}|^{2}.

Note that this convergence property is local since it requires the starting point x_{0} ∈ D, that is, Newton’s method only guarantee the convergence from a good starting point x_{0} that is close enought to x_{∗}. In fact, Newton’s iteration may not converge at all if |x_{0} − x_{∗}| is large. Therefore some globally convergent algorithm needs to be incorporated with Newton’s method. Furthermore, f′(x_{∗}) must be nonzero for the method to converge quadratically. Indeed, if f′(x_{∗}) = 0, i.e., x_{∗} is a multiple root of f, then Newton’s method converges only linearly. However, when f is a linear function, Newton’s method will converge in one step.

A simple strategy to modify Newton’s method for global convergent is to incorporate a back-tracking procedure in each Newton step. The idea is that in the direction of the tangent line, we find a point where the function value decreases in magnitude. More precisely, one involves

x_{k+1} = x_{k} − \frac{f(x_{k})}{f′(x_{k})}

while |f(x_{k+1})| ≥ |f(x_{k})| do x_{k+1} = \frac{1}{2}(x_{k+1} + x_{k})

end while

5.3.3 Examples and Pitfalls

Example 5.3 The following table shows the convergence behavior of Newton’s method applied to solving f(x) = x^{2} − 1 = 0. Observe the quadratic convergence rate.

Example 5.4 Newton’s method will fail (the sequence {x_{k}} does not converge) on solving f(x) = x^{2} − 4x + 5 = 0 since f does not have any real root.

Example 5.5 When Newton’s method applied to f(x) = cos x with starting point x_{0} = 3, which is close to the root \frac{π}{2} of f, it produces x_{1} = −4.01525, x_{2} = −4.8526, ⋯, which converges to another root −\frac{3π}{2}.

Example 5.6 When Newton’s method applied to f(x) = xe^{−x} with starting point x_{x} = 2.0, it produces x_{1} = 4.0, x_{2} = 5.333, . . .. The sequence {x_{k}} diverges to ∞ slowly, however, f(x_{k}) goes to zero rapidly as x_{k} gets larger in a finite precision environment, and could be mistaken as a zero of f.

Example 5.7 When Newton’s method applied to f(x) = x^{3}−x−3 with starting point x_{0} = 0, it produces x_{1} = −3, x_{2} = −1.961538, x_{3} = −1.147176, x_{4} = −0.00679, x_{5} = −3.000389, x_{6} = −1.961818, x_{7} = −1.147430, . . .. The sequence will not converge. But if the algorithm starts with x_{0} = 2, then it produces x_{1} = 1.727272, x_{2} = 1.67369, x_{3} = 1.6717025, . . .. The sequence converges to the root 1.671699881. This example illustrates that the starting point x_{0} must be close enough to the zero of f.

5.3.4 System of Nonlinear Equations

First consider solving the following system of nonlinear equations:

\begin{cases}f_{1}(x_{1}, x_{2}) = 0, \\ f_{2}(x_{1}, x_{2}) = 0. \end{cases}  (5.12)

Suppose (x_{1}^{(k)}, x_{2}^{(k)}) is an approximation to the solution of the system above, and we try to compute h_{1}^{(k)} and h_{2}^{(k)} such that (x_{1}^{(k)} + h_{1}^{(k)}, x_{2}^{(k)} + h_{2}^{(k)}) satisfies the system. By the Taylor’s theorem for two variables,

0 = f_{1}(x_{1}^{(k)} + h_{1}^{(k)}, x_{2}^{(k)} + h_{2}^{(k)}) ≈ f_{1}(x_{1}^{(k)}, x_{2}^{(k)}) + h_{1}^{(k)} \frac{∂f_{1}}{∂x_{1}} (x_{1}^{(k)}, x_{2}^{(k)}) + h_{2}^{(k)} \frac{∂f_{1}}{∂x_{2}} (x_{1}^{(k)}, x_{2}^{(k)})
0 = f_{2}(x_{1}^{(k)} + h_{1}^{(k)}, x_{2}^{(k)} + h_{2}^{(k)}) ≈ f_{2}(x_{1}^{(k)}, x_{2}^{(k)}) + h_{1}^{(k)} \frac{∂f_{2}}{∂x_{1}} (x_{1}^{(k)}, x_{2}^{(k)}) + h_{2}^{(k)} \frac{∂f_{2}}{∂x_{2}} (x_{1}^{(k)}, x_{2}^{(k)})

Put this in matrix form

[ \begin{matrix}\frac{∂f_{1}}{∂x_{1}} (x_{1}^{(k)}, x_{2}^{(k)}) & \frac{∂f_{1}}{∂x_{2}} (x_{1}^{(k)}, x_{2}^{(k)}) \\ \frac{∂f_{2}}{∂x_{1}} (x_{1}^{(k)}, x_{2}^{(k)}) & \frac{∂f_{2}}{∂x_{2}} (x_{1}^{(k)}, x_{2}^{(k)}) \end{matrix} ] [ \begin{matrix}h_{1}^{(k)} \\ h_{2}^{(k)} \end{matrix} ] + [ \begin{matrix}f_{1}(x_{1}^{(k)}, x_{2}^{(k)}) \\ f_{2}(x_{1}^{(k)}, x_{2}^{(k)}) \end{matrix} ] ≈ [ \begin{matrix}0 \\ 0 \end{matrix} ] .

The matrix

J(x_{1}^{(k)}, x_{2}^{(k)}) ≡ [ \begin{matrix}\frac{∂f_{1}}{∂x_{1}} (x_{1}^{(k)}, x_{2}^{(k)}) & \frac{∂f_{1}}{∂x_{2}} (x_{1}^{(k)}, x_{2}^{(k)}) \\ \frac{∂f_{2}}{∂x_{1}} (x_{1}^{(k)}, x_{2}^{(k)}) & \frac{∂f_{2}}{∂x_{2}} (x_{1}^{(k)}, x_{2}^{(k)}) \end{matrix} ]  (5.13)

is called the Jacobian matrix. Set h_{1}^{(k)} and h_{2}^{(k)} be the solution of the linear system

J^{−1}(x_{1}^{(k)}, x_{2}^{(k)}) [ \begin{matrix}h_{1}^{(k)} \\ h_{2}^{(k)} \end{matrix} ] = − [ \begin{matrix}f_{1}(x_{1}^{(k)}, x_{2}^{(k)}) \\ f_{2}(x_{1}^{(k)}, x_{2}^{(k)}) \end{matrix} ]

then

[ \begin{matrix}x_{1}^{(k+1)} \\ x_{2}^{(k+1)} \end{matrix} ] = [ \begin{matrix}x_{1}^{(k)} \\ x_{2}^{(k)} \end{matrix} ] + [ \begin{matrix}h_{1}^{(k+1)} \\ h_{2}^{(k+1)} \end{matrix} ]

is expected to be a better approximation.

In general, we solve the system of n nonlinear equations f_{i}(x_{1}, ⋯, x_{n}) = 0, i = 1, …, n. Let

X = [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] and F (X) = [ f_{1}(X) ; f_{2}(X) ; ⋮ ; f_{n}(X) ] .

The problem can be formulated as solving

F (X) = 0, F : ℝ^{n} → ℝ^{n}.  (5.14)

Let F′(X), where the (i, j) entry is \frac{∂f_{i}}{∂x_{j}} (X), be the n×n Jacobian matrix. Then the Newton’s iteration is defined as

X^{(k+1)} = X^{(k)} + S^{(k)},  (5.15)

where S^{(k)} ∈ ℝ^{n} is the solution of the linear system

F′(X^{(k)})S^{(k)} = −F (X^{(k)}).  (5.16)

Newton’s method for solving systems of nonlinear equations still enjoys the property of rapid quadratic convergence if the starting point is near the exact solution point in terms of vector norm. However, a significant weakness of Newton’s method is the requirement that, at each iteration, a Jacobian matrix has to be evaluated and an n×n linear system involving this matrix must be solved. To construct the Jacobian matrix, n^{2} partial derivatives have to be available and evaluated. Then an LU-factorization has to be performed for the solution of the the linear system involved.

5.4 Quasi-Newton’s Method (Secant Method)

5.4.1 The Secant Method

The Newton’s iteration

x_{k+1} = x_{k} − \frac{f(x_{k})}{f′(x_{k})}

involves the evaluation of derivative of f. In many applications (situations), the derivative f′(x) is very expensive to compute, or the function f(x) is not given in an algebraic formula so that f′(x) is not available.

An alternative is to replace the derivative evaluation by finite difference approximation. Since

f′(x) = lim_{h→0} \frac{f(x + h) − f(x)}{h},

it is reasonable to approximate f′(x_{k}) in the Newton’s iteration by

f′(x_{k}) ≈ \frac{f(x_{k} + h_{k}) − f(x_{k})}{h_{k}}

for some small h_{k}. This leads to the so-called finite-difference Newton’s iteration

x_{k+1} = x_{k} − f(x_{k})\frac{h_{k}}{f(x_{k} + h_{k}) − f(x_{k})}  (5.17)

From geometric point of view, we use a secant line through x_{k} and a near-by point x_{k} +h_{k} instead of the tangent line to approximate the function at the point x_{k}. The slope of the secant line is

s_{k} = \frac{f(x_{k} + h_{k}) − f(x_{k})}{h_{k}}  (5.18)

and the equation is

M(x) = f(x_{k}) + s_{k}(x − x_{k}).  (5.19)

The zero of the secant line

x = x_{k} − \frac{f(x_{k})}{s_{k}} = x_{k} − f(x_{k})\frac{h_{k}}{f(x_{k} + h_{k}) − f(x_{K})}  (5.20)

is then used as a new approximate x_{k+1}. This leads to exactly the same formulation for the finite-difference Newton’s iteration.

Two questions arise so far: “Will this method work?” and “How should h_{k} be chosen?” From the limit definition of derivative, it seems reasonable to assume that the finite-difference formulation should work almost as well as Newton’s method if h_{k} is chosen small enough. However, it requires two function evaluations f(x_{k} + h_{k}) and f(x_{k}) in each iteration. If the function evaluation is expensive, we may set

h_{k} = x_{k} − x_{k−1}.

This leads to the so-called secant method or quasi-Newton method.

x_{k+1} = x_{k} − f(x_{k})\frac{x_{k} − x_{k−1}}{f(x_{k}) − f(x_{k−1})}  (5.21)

Note that the secant method requires two initial guesses x_{0} and x_{−1}. However, there is only one function evaluation required in each iteration.

It turns out that the secant method requires only a few more iterations to converge than the finite difference approach with a properly chosen h_{k}, but it’s usually more efficient in terms of total number of function evaluations.

Example 5.8 The following table shows the convergence history for the finite-difference Newton’s method with h_{k} = 10^{−7} ⋆ x_{k} and secant method for solving f(x) = x^{2} − 1 = 0.

finite-difference Newton secant method
x_{0} 2  2
x_{1} 1.2500000266453  1.2500000266453
x_{2} 1.0250000179057  1.0769230844910
x_{3} 1.0003048001120  1.0082644643823
x_{4} 1.0000000464701  1.0003048781354
x_{5} 1  1.0000012544523
x_{6}  1.0000000001912
x_{7}  1

5.4.2 Error Analysis of Secant Method

Let x_{∗} denote the exact solution of f(x) = 0, e_{k} = x_{k} − x_{∗} be the errors at the k-th step. Then

e_{k+1} = x_{k+1} − x_{∗}
= x_{k} − f(x_{k})\frac{x_{k} − x_{k−1}}{f(x_{k}) − f(x_{k−1})} − x_{∗}
= \frac{1}{f(x_{k}) − f(x_{k−1})} [x_{k}f(x_{k}) − x_{k}f(x_{k−1}) − x_{k}f(x_{k}) + x_{k−1}f(x_{k}) − x_{∗}f(x_{k}) + x_{∗}f(x_{k−1})]
= \frac{1}{f(x_{k}) − f(x_{k−1})} [(x_{k+1} − x_{∗})f(x_{k}) − (x_{k} − x_{∗})f(x_{k−1})]
= \frac{1}{f(x_{k}) − f(x_{k−1})} (e_{k−1}f(x_{k}) − e_{k}f(x_{k−1}))
= e_{k}e_{k−1} ( \frac{\frac{1}{e_{k}} f(x_{k}) − \frac{1}{e_{k−1}} f(x_{k})}{x_{k} − x_{k−1}} · \frac{x_{k} − x_{k−1}}{f(x_{k} − f(x_{k−1})})

To estimate the numerator \frac{\frac{1}{e_{k}} f(x_{k})− \frac{1}{e_{k−1}} f(x_{k})}{x_{k}−x_{k−1}}, we apply the Taylor’s theorem

f(x_{k}) = f(x_{∗} + e_{k}) = f(x_{∗}) + f′(x_{∗})e_{k} + \frac{1}{2}f′′(x_{∗})e_{k}^{2} + O(e_{k}^{3}),

to get

\frac{1}{e_{k}} f(x_{k}) = f′(x_{∗}) + \frac{1}{2}f′′(x_{∗})e_{k} + O(e_{k}^{2}).

Similarly,

\frac{1}{e_{k−1}} f(x_{k−1}) = f′(x_{∗}) + \frac{1}{2}f′′(x_{∗})e_{k−1} + O(e_{k−1}^{2}).

Hence

\frac{1}{e_{k}} f(x_{k}) − \frac{1}{e_{k−1}} f(x_{k−1}) ≈ \frac{1}{2}(e_{k} − e_{k−1})f′′(x_{∗}).

Since x_{k} − x_{k−1} = e_{k} − e_{k−1} and

\frac{x_{k} − x_{k−1}}{f(x_{k}) − f(x_{k−1})} → \frac{1}{f′(x_{∗})},

we have

e_{k+1} ≈ e_{k}e_{k−1} ( \frac{\frac{1}{2}(e_{k} − e_{k−1})f′(x_{∗})}{e_{k} − e_{k−1}} · \frac{1}{f′(x_{∗})}) = \frac{1}{2} \frac{f′′(x_{∗})}{f′(x_{∗})}e_{k}e_{k−1} ≡ Ce_{k}e_{k−1}  (5.22)

To estimate the convergence rate, we assume

|e_{k+1}| ≈ η|e_{k}|^{α},

where η > 0 and α > 0 are constants, i.e.,

\frac{|e_{k+1}|}{η|e_{k}|^{α}} → 1 as k → ∞.

Then |e_{k}| ≈ η|e_{k−1}|^{α} which implies e_{k−1} ≈ η^{−1/α}|e_{k}|^{1/α}. Hence () gives

η|e_{k}|^{α} ≈ C|e_{k}|η^{−1/α}|e_{k}|^{1/α} ⟹ C^{−1}η^{1+ \frac{1}{α}} ≈ |e_{k}|^{1−α+ \frac{1}{α}} .

Since |e_{k}| → 0 as k → ∞, and C^{−1}η^{1+ \frac{1}{α}} is a nonzero constant,

1 − α + \frac{1}{α} = 0 ⟹ α = \frac{1 + \sqrt{5}}{2} ≈ 1.62.

This result implies that C^{−1}η^{1+ \frac{1}{α}} → 1 and

η → C^{\frac{α}{1+α}} = (\frac{f′′(x_{∗})}{2f′(x_{∗})})^{0.62} .

In summary, we have shown that

|e_{k+1}| = η|e_{k}|^{α}, α ≈ 1.62,  (5.23)

that is, the convergence rate is superlinear.

The rapidity of convergence of the secant method is not as good as the Newton’s method, but is better than the bisection method. However, each iteration of the secant method requires only one function evaluation while the Newton’s method requires two, namely, f(x_{k}) and f′′(x_{k}). Therefore, we may regard two steps of secant method are comparable to one step of Newton’s method. Thus

|e_{k+2}| ≈ η|e_{k+1}|^{α} ≈ η^{1+α}|ek^{\frac{3+\sqrt{5}}{2}} .

Since \frac{3+\sqrt{5}}{2} ≈ 2.62, this shows that secant method is more efficient than Newton’s method. Of course, two steps of secant method would require a little more work than one step of Newton’s method.

Next we will extend secant method to solving system of nonlinear equations. Recall that in one dimensional case, one uses the linear model

ℓ_{k}(x) = f(x_{k}) + a_{k}(x − x_{k})  (5.24)

to approximate the function f(x) at x_{k}. That is, ℓ_{k}(x_{k}) = f(x_{k}) for any a_{k} ∈ ℝ. If we further require that ℓ′(x_{k}) = f′(x_{k}), then a_{k} = f′(x_{k}). The zero of ℓ_{k}(x) is used to give a new approximate for the zero of f(x), that is,

x_{k+1} = x_{k} − \frac{1}{f′(x_{k})}f(x_{k})

which yields Newton’s method.

If f′(x_{k}) is not available, one instead asks the linear model to satisfy

ℓ_{k}(x_{k}) = f(x_{k}) and ℓ_{k}(x_{k−1}) = f(x_{k−1}).  (5.25)

In doing this, the identity

f(x_{k−1}) = ℓ_{k}(x_{k−1}) = f(x_{k}) + a_{k}(x_{k−1} − x_{k})

gives

a_{k} = \frac{f(x_{k} − f(x_{k−1})}{x_{k} − x_{k−1}}.  (5.26)

Solving ℓ_{k}(x) = 0 yields the secant iteration

x_{k+1} = x_{k} − \frac{x_{k} − x_{k−1}}{f(x_{k} − f(x_{k−1})}f(x_{k}).  (5.27)

In multiple dimension, the analogue affine modle becomes

M_{k}(x) = F (x_{k}) + A_{k}(x − x_{k}),  (5.28)

where x, x_{k} ∈ ℝ^{n} and A_{k} ∈ ℝ^{n×n}, and satisfies

M_{k}(x_{k}) = F (x_{k}),  (5.29)

for any A_{k}. The zero of M_{k}(x) is then used to give a new approximate for the zero of F (x), that is,

x_{k+1} = x_{k} − A_{k}^{−1}F (x_{k}).  (5.30)

The Newton’s method chooses

A_{k} = F′(x_{k}) ≡ J(x_{k}) = the Jacobian matrix.  (5.31)

and yields the iteration

x_{k+1} = x_{k} − (F′(x_{k}))^{−1} F (x_{k}).  (5.32)

When the Jacobian matrix J(x_{k}) ≡ F′(x_{k}) is not available, one can require

M_{k}(x_{k−1}) = F (x_{k−1}).  (5.33)

Then

F (x_{k−1}) = M_{k}(x_{k−1}) = F (x_{k}) + A_{k}(x_{k−1} − x_{k}),

which gives

A_{k}(x_{k} − x_{k−1}) = F (x_{k}) − F (x_{k−1})  (5.34)

and this is the so-called secant equation. Let

s_{k} = x_{k} − x_{k−1} and y_{k} = F (x_{k}) − F (x_{k−1}).  (5.35)

The secant equation becomes

A_{k}s_{k} = y_{k}.  (5.36)

However, this secant equation can not uniquely determine A_{k}. One way of choosing A_{k} is to minimize M_{k} − M_{k−1} subject to the secant equation. Note

M_{k}(x) − M_{k−1}(x) = F (x_{k}) + A_{k}(x − x_{k}) − F (x_{k−1}) − A_{k−1}(x − x_{k−1})
= (F (x_{k} − F (x_{k−1})) + A_{k}(x − x_{k}) − A_{k−1}(x − x_{k−1})
= A_{k}(x_{k} − x_{k−1}) + A_{k}(x − x_{k}) − A_{k−1}(x − x_{k−1})
= A_{k}(x − x_{k−1}) − A_{k−1}(x − x_{k−1})
= (A_{k} − A_{k−1})(x − x_{k−1}).

For any x ∈ ℝ^{n}, we express

x − x_{k−1} = αs_{k} + t_{k},  (5.37)

for some α ∈ ℝ, t_{k} ∈ ℝ^{n}, and s_{k}^{T} t_{k} = 0. Then

M_{k} − M_{k−1} = (A_{k} − A_{k−1})(αs_{k} + t_{k}) = α(A_{k} − A_{k−1})s_{k} + (A_{k} − A_{k−1})t_{k}.

Since

(A_{k} − A_{k−1})s_{k} = A_{k}s_{k} − A_{k−1}s_{k} = y_{k} − A_{k−1}s_{k},

both y_{k} and A_{k−1}s_{k} are old values, we have no control over the first part (A_{k} − A_{k−1})s_{k}. In order to minimize M_{k}(x) − M_{k−1}(x), we try to choose A_{k} so that

(A_{k} − A_{k−1})t_{k} = 0  (5.38)

for all t_{k} ∈ ℝ^{n}, s_{k}^{T} t_{k} = 0. This requires that A_{k} − A_{k−1} to be a rank-one matrix of the form

A_{k} − A_{k−1} = u_{k}s_{k}^{T}  (5.39)

for some u_{k} ∈ ℝ^{n}. Then

u_{k}s_{k}^{T} s_{k} = (A_{k} − A_{k−1})s_{k} = y_{k} − A_{k−1}s_{k}

which gives

u_{k} = \frac{y_{k} − A_{k−1}s_{k}}{s_{k}^{T} s_{k}} .  (5.40)

Therefore,

A_{k} = A_{k−1} + \frac{(y_{k} − A_{k−1}s_{k})s_{k}^{T}}{s_{k}^{T} s_{k}}  (5.41)

After A_{k} is determined, the new iterate x_{k+1} is derived from solving M_{k}(x) = 0. It can be done by first noting that

s_{k+1} = x_{k+1} − x_{k} ⟹ x_{k+1} = x_{k} + s_{k+1}  (5.42)

and

M_{k}(x_{k+1}) = 0 ⟹ F (x_{k}) + A_{k}(x_{k+1} − x_{k}) = 0 ⟹ A_{k}s_{k+1} = −F (x_{k}) (5.43)

These formulations give the Broyden’s method.

Algorithm 5.3 (Broyden’s Method) Given a n-variable nonlinear function F : ℝ^{n} → ℝ^{n}, an initial iterate x_{0} and initial Jacobian matrix A_{0} ∈ ℝ^{n×n} (e.g., A_{0} = I), this algorithm finds the solution for F (x) = 0.

for k = 0, 1, ⋯, do Solve A_{k}s_{k+1} = −F (x_{k}) for s_{k+1} Update x_{k+1} = x_{k} + s_{k+1} Compute y_{k+1} = F (x_{k+1}) − F (x_{k})

Update A_{k+1} = A_{k} + \frac{(y_{k+1} − A_{k}s_{k+1})s_{k+1}^{T}}{s_{k+1}^{T}s_{k+1}} = A_{k} +

\frac{(y_{k+1} + F (x_{k})s_{k+1}^{T}}{s_{k+1}^{T}s_{k+1}}

end for

Since a new matrix A_{k} is used to approximate the Jacobian matrix in each iteration of the Broyden’s method, LU-factorization has to be performed in order to solve the linear system A_{k}s_{k+1} = −F (x_{k}) for s_{k+1}. One LU-factorization would cost frac23n^{3}+O(n^{2}) floating-point operations. But if we apply the Shermann-Morrison-Woodbury formula to (), we have

A_{k}^{−1} = A_{k−1}^{−1} + \frac{(s_{k} − A_{k}^{−1}y_{k})y_{k}^{T}}{y_{k}^{T} y_{k}} .  (5.44)

With a given A_{0}^{−1}, we can then update A_{k}^{−1} in each iteration using this formula and s_{k+1} can be easily computed by matrix-vector multiplication s_{k+1} = −A_{k}^{−1}F (x_{k}) in O(n^{2}) operations.

5.5 Fixed Point and Functional Iteration

In this section, we consider the problem of finding a fixed point with functional iteration method for a given fuction f : D → ℝ, where D ⊆ ℝ is a closed set.

Definition 5.4 x is called a fixed point of a given function f if f(x) = x.

If f : ℝ → ℝ is a function, then the fixed point of f can be viewed as the intersection of the curve y = f(x) and the straight line y = x.

The problem of finding a fixed point for a given function is equivalent to the problem of finding a zero for a nonlinear fucntion. For instance, given a function f(x) and a root x_{∗} such that f(x_{∗}) = 0, let g(x) = x − f(x). Then g(x_{∗}) = x_{∗} − f(x_{∗}) = x_{∗}. That is, x_{∗} is a fixed point for g(x). Conversely, suppose that g(x) has a fixed point at x_{∗}. Then one may define f(x) = x − g(x) so that f(x_{∗}) = x_{∗} − g(x_{∗}) = x_{∗} − x_{∗} = 0, i.e., x_{∗} is a zero of f(x).

Two questions arise: “When does a function have a fixed point?” and “How to find a fixed point?”.

5.5.1 Functional Iteration

To determine a fixed point x_{∗} for a continuous function f, we choose an initial point x_{0} and generate a sequence of points {x_{k}}_{k≥0} by

x_{k+1} = f(x_{k}), k ≥ 0.  (5.45)

This algorithm is called fixed-point iteation or functional iteration. This formulation may generate sequences that do not converge, e.g., f(x) = 3x. However, wehn the sequence converges, say,

lim_{k→∞} x_{k} = x_{∗},

then, since f is continuous,

f(x_{∗}) = f( lim_{k→∞} x_{k}) = lim_{k→∞} f(x_{k}) = lim_{k→∞} x_{k+1} = x_{∗}.

That is, x_{∗} is a fixed point of f.

Note that Newton’s method for solving g(x) = 0

x_{k+1} = x_{k} − \frac{g(x_{k})}{g′(x_{k})}

is just a special case of functional iteration in which

f(x) = x − \frac{g(x)}{g′(x)}.

Definition 5.5 A function (mapping) f is said to be contractive if there exists a constant 0 ≤ λ < 1 such that

|f(x) − f(y)| ≤ λ|x − y|  (5.46)

for all x, y in the domain of f.

Theorem 5.4 (Contractive Mapping Theorem) Suppose f : D → D, where D ⊆ ℝ is a closed set, is a contractive mapping. Then f has a unique fixed point in D. Moreover, this fixed point is the limit of every sequence obtained by

x_{k+1} = f(x_{k})

with any initial point x_{0}.

Proof: We first show that lim_{k→∞} x_{k} exists. Since

x_{k} = x_{0} + (x_{1} − x_{0}) + (x_{2} − x_{1}) + ⋯ + (x_{k} − x_{k−1}) = x_{0} + ∑_{i=1}^{k}(x_{i} − x_{i−1}),

{x_{k}}_{k≥0} converges if and only if ∑_{i=1}^{∞}(x_{i} − x_{i−1}) converges and it is sufficient to show ∑_{i=1}^{∞} |x_{i} − x_{i−1}| converges.

Since f is contractive, we have

|x_{i} − x_{i−1}| = |f(x_{i−1}) − f(x_{i−2})|
≤ λ|x_{i−1} − x_{i−2}|
≤ λ^{2}|x_{i−2} − x_{i−3}|
⋮
≤ λ^{i−1}|x_{1} − x_{0}|.

Then we have

∑_{i=1}^{∞} |x_{i} − x_{i−1}| ≤ ∑_{i=1}^{∞} λ^{i−1}|x_{1} − x_{0}|

= |x_{1} − x_{0}| ∑_{i=1}^{∞} λ^{i−1}
= \frac{1}{1 − λ}|x_{1} − x_{0}|

since 0 ≤ λ < 1. This show that ∑_{i=1}^{∞} |x_{i} − x_{i−1}| is bounded, hence it converges. This shows that the sequence {x_{k}}_{k≥0} converges for any initial point x_{0}.

Let lim_{k→∞} x_{k} = x_{∗}. Then we have showed that x_{∗} would be a fixed point of f. (note: contractivenedd implies continuity)

To prove the uniqueness, let x and y both be fixed points of f. Then

|x − y| = |f(x) − f(y)| ≤ λ|x − y|.

Since λ < 1, this forces |x − y| = 0. That is, x = y.

Theorem 5.5 If f ∈ C[a, b] such that a ≤ f(x) ≤ b for all x ∈ [a, b], then f has a fixed point in [a, b]. Suppose, in addition, that f′(x) exists in (a, b) and there exists a positive constant M < 1 such that |f′(x)| ≤ M < 1 for all x ∈ (a, b). Then the fixed point is unique.

Proof: If f(a) = a or f(b) = b, then a or b is a fixed point of f and we are done. Otherwise, it must be g(a) > a and g(b) < b. Let h(x) = f(x) − x. Then h ∈ C[a, b] since f ∈ C[a, b], and h(a) = f(a) − a > 0 and h(b) = f(b) − b < 0. By the itermediate value theorem, there exists x_{∗} ∈ [a, b] such that h(x_{∗}) = 0. This implies that f(x_{∗}) − x_{∗} = 0 and f(x_{∗}) = x_{∗}. Hence f has a fixed point x_{∗} in [a, b].

Suppose that p ≠ q are both fixed points of f in [a, b]. By the Mean-Value theorem, there exists ξ between p and q such that

f′(ξ) = \frac{f(p) − f(q)}{p − q} = \frac{p − q}{p − q} = 1.

However, this contradicts to the assumption that f′(x) ≤ M < 1 for all x in [a, b]. Therefore the fixed point of f is unique.

5.5.2 Convergence Analysis

For the underlined function f : D → D, we make the following assumptions:

1. f has a fixed point x_{∗}, and the sequence {x_{k}}_{k≥0} is generated by the iteration

x_{k+1} = f(x_{k}), k = 0, 1, …,

with an arbitrary initial point x_{0};

2. f′ exists, is continuous, and |f′(x)| < 1 for all x in the domain of f;

3. m is a positive integer such that f^{(i)}(x_{∗}) = 0 for i = 1, …, m, but f^{(m)}(x_{∗}) ≠ 0.

Let e_{k} = x_{k} − x_{∗} be the error at the k-th iteration. Then by the Mean-Value theorem

e_{k+1} = x_{k+1} − x_{∗} = f(x_{k}) − f(x_{∗}) = f′(ξ_{k})(x_{k} − x_{∗}) = f′(ξ_{k})e_{k},  (5.47)

where ξ_{k} is between x_{k} and x_{∗}. Since |f′(ξ_{k})| < 1 by assumption, it ensures the error decreases in magnitude.

When x_{k} is close to x_{∗}, i.e., e_{k} is small, ξ_{k} will be close to x_{∗}, and f′(ξ_{k}) will be close to f′(x_{∗}) since f′ is continuous. Hence () can be written as

e_{k+1} ≈ f′(x_{∗})e_{k},

and we can expect rapid convergence if f′(x_{∗}) is small.

Using Taylor’s theorem, we have

e_{k+1} = x_{k+1} − x_{∗}
= f(x_{k}) − f(x_{∗})
= f(x_{∗} + e_{k}) − f(x_{∗})
= (f(x_{∗}) + f′(x_{∗})e_{k} + \frac{1}{2}f′′(x_{∗})e_{k}^{2} + ⋯ + \frac{1}{(m − 1)!}f^{(m−1)}(x_{∗})e_{k}^{m−1} + \frac{1}{m!}f^{(m)}(η_{k})e_{k}^{m}) − f(x_{∗})
= \frac{1}{m!}f^{(m)}(η_{k})e_{k}^{m},

where η_{k} is between x_{∗} and x_{k}. Since lim_{k→∞} x_{k} = x_{∗},

η_{k} → x_{∗} ⟹ f^{(m)}(η_{k}) → f^{(m)}(x_{∗}).

Therefore we have

lim_{k→∞} \frac{|e_{k+1}|}{|e_{k}|^{m}} = \frac{1}{m!}f^{(m)}(x_{∗}).  (5.48)

This shows that the convergence rate of the functional iteration is m.

Since Newton’s method is a special case of functional iteration, we can also show the quadratic convergence of Newton’s method using this analysis. Suppose Newton’s method is applied to solving g(x) = 0. Let f(x) = x − \frac{g(x)}{g′(x)}. Then

f′(x_{∗}) = 1 − \frac{[g′(x_{∗})]^{2} − g(x_{∗})g′′(x_{∗})}{[g′(x_{∗})]^{2}} = \frac{g(x_{∗})g′′(x_{∗})}{[g′(x_{∗})]^{2}} = 0

since the fixed point of f is a zero of g. The second derivative of f at x_{∗}

f′′(x_{∗}) = \frac{1}{[g′(x_{∗})]^{4}} ([g′(x_{∗})]^{2}(g(x_{∗})g′′′(x_{∗}) + g′′(x_{∗})g′(x_{∗})) − 2g(x_{∗})g′(x_{∗})[g′′(x_{∗})]^{2})

is usually not zero. Hence it shows that Newton’s method has quadratic convergence rate.

Theorem 5.6 Suppose that f ∈ C[a, b] such that a ≤ f(x) ≤ b for all x ∈ [a, b], and that f′(x) exists on (a, b) with f′(x) ≤ M < 1 for some constant M. If x_{0} ∈ [a, b] is arbitrary, then the sequence {x_{k}}_{k≥0} generated by the functional iteration x_{k+1} = f(x_{k}) converges to the unique fixed point x_{∗} of f.

Proof: By Theorem (), the fixed point x_{∗} of f exists and is unique.

Since a ≤ f(x) ≤ b for all x ∈ [a, b],

a ≤ x_{k} = f(x_{k−1}) ≤ b, for all k = 1, 2, . . ..

By the Mean-Value theorem, there exists ξ_{k} ∈ (a, b) such that

f′(ξ_{k}) = \frac{f(x_{k}) − f(x_{∗})}{x_{k} − x_{∗}}.

Thus

|e_{k}| = |x_{k} − x_{∗}| = |f′(ξ_{k})||f(x_{k}) − f(x_{∗})|
= |f′(ξ_{k})||x_{k−1} − x_{∗}|
≤ M|e_{k−1}| ≤ M^{2}|e_{k−2}| ≤ ⋯ ≤ M^{k}|e_{0}| = M^{k}|x_{0} − x_{∗}|.

Since M < 1, M^{k} → 0 as k → ∞. Therefore

lim_{k→∞} |x_{k} − x_{∗}| ≤ lim_{k→∞} M^{k}|x_{0} − x_{∗}| = 0 ⟹ lim_{k→∞} x_{k} = x_{∗}.

Corollary 5.1 Suppose f satisfies the conditions of the previous theorem and the sequence {x_{k}}_{k≥0} is produced by the functional iteration x_{k+1} = g(x_{k}) with an arbitrary initial point x_{0}. Then

|e_{k}| = |x_{k} − x_{∗}| ≤ M^{k} max{x_{0} − a, b − x_{0}}  (5.49)

and

|e_{k}| = |x_{k} − x_{∗}| ≤ \frac{M^{k}}{1 − M} |x_{0} − x_{1}| = \frac{M^{k}}{1 − M} |e_{0}|.  (5.50)

Proof: From the proof of the previous theorem,

|e_{k}| = |x_{k} − x_{∗}| ≤ M^{k}|x_{0} − x_{∗}| ≤ M^{k} max{x_{0} − a, b − x_{0}}.

Since

|x_{k+1} − x_{k}| = |f(x_{k}) − f(x_{k−1})|
= |f′(ξ_{k})||x_{k} − x_{k−1}|
≤ M|x_{k} − x_{k−1}|
≤ M^{2}|x_{k−1} − x_{k−2}|
⋮
≤ M^{k}|x_{1} − x_{0}|,

for m > k ≤ 1,

|x_{m} − x_{k}| = |x_{m} − x_{m−1} + x_{m−1} − x_{m−2} + ⋯ + x_{k+1} − x_{k}|
≤ |x_{m} − x_{m−1}| + |x_{m−1} − x_{m−2}| + ⋯ + |x_{k+1} − x_{k}|
≤ M^{m−1}|x_{1} − x_{0}| + M^{m−2}|x_{1} − x_{0}| + ⋯ + M^{k}|x_{1} − x_{0}|
= M^{k}(M^{m−1−k} + M^{m−2−k} + ⋯ + M + 1)|x_{1} − x_{0}|.

Since lim_{m→∞} x_{m} = x_{∗},

|x_{k} − x_{∗}| = lim_{m→∞} |x_{k} − x_{m}|
≤ M^{k}|x_{1} − x_{0}| lim_{m→∞}(M^{m−1−k} + M^{m−2−k} + ⋯ + M + 1)
= M^{k}|x_{1} − x_{0}| ∑_{i=1}^{∞} M^{i}

= \frac{M^{k}}{1 − M} |x_{1} − x_{0}|.

Interpolation

In this chapter, we consider the interpolation problem: Suppose we do not know the function f, but a few information (data) about f, now we try to compute a function g that approximates f.

6.1 Polynomial Interpolation

The polynomial interpolation problem, also called Lagrange interpolation, can be described as follows: Given n + 1 data points (x_{i}, y_{i}), i = 0, 1, …, n, find a polynomial P of lowest possible degree such that

y_{i} = P (x_{i}), i = 0, 1, …, n. (6.1)

Such a polynomial is said to interpolate the data. Here y_{i} may be the value of some unknown function f at x_{i}, i.e., y_{i} = f(x_{i}).

One reason for considering the class of polynomials in approximation of functions is that they uniformly approximate continuous function.

Theorem 6.1 (Weierstrass Approximation Theorem) Suppose that f is defined and continuous on [a, b]. For any ε > 0, there exists a polynomial P (x), defined on [a, b], with the property that

|f(x) − P (x)| < ε, for all x in [a, b].

Another important reason for considering the class of polynomials in approximation of functions is that the derivatives and indefinite integral of a polynomial are easy to determine and are also polynomials. For these reasons, polynomials are often used for approximating continuous functions.

6.1.1 Existence And Uniqueness

The theorem that governs the polynomial interpolation problem stats

Theorem 6.2 If (x_{i}, y_{i}), x_{i}, y_{i} ∈ ℝ, i = 0, 1, …, n, are n + 1 distinct pairs of data point, then there is a unique polynomial P_{n} of degree at most n such that

P_{n}(x_{i}) = y_{i}, (0 ≤ i ≤ n).  (6.2)

Proof: Existence: Proof by mathematical induction. The theorem clearly holds for n = 0 (only one data point (x_{0}, y_{0})) since one may choose the constant polynomial P_{0}(x) = y_{0} for all x. Assume that the theorem holds for n ≤ k, that is, there is a polynomial P_{k}, deg(P_{k}) ≤ k, such that y_{i} = P_{k}(x_{i}), for 0 ≤ i ≤ k. Next we try to construct a polynomial of degree at most k + 1 to interpolate (x_{i}, y_{i}), 0 ≤ i ≤ k + 1. Let

P_{k+1}(x) = P_{k}(x) + c(x − x_{0})(x − x_{1}) ⋯ (x − x_{k}),

where

c = \frac{y_{k+1} − P_{k}(x_{k+1})}{(x_{k+1} − x_{0})(x_{k+1} − x_{1}) ⋯ (x_{k+1} − x_{k})}.

Since x_{i} are distinct, the polynomial P_{k+1}(x) is well-defined and deg(P_{k+1}) ≤ k + 1. It is easy to verify that

P_{k+1}(x_{i}) = y_{i}, 0 ≤ i ≤ k + 1.

Uniqueness: Suppose there are two such polynomials P_{n} and Q_{n} satisfying (6.2). Define

S_{n}(x) = P_{n}(x) − Q_{n}(x).

Since both deg(P_{n}) ≤ n and deg(Q_{n}) ≤ n, deg(S_{n}) ≤ n. Moreover

S_{n}(x_{i}) = P_{n}(x_{i}) − Q_{n}(x_{i}) = y_{i} − y_{i} = 0,

for 0 ≤ i ≤ n. This means that S_{n} has at least n + 1 zeros, it therefore must be S_{n} = 0. Hence P_{n} = Q_{n}.

6.1.2 Naive Approach for Polynomial Interpolation

Suppose we require the interpolating polynomial to be expressed in power of x as

P_{n}(x) = c_{0} + c_{1}x + c_{2}x^{2} + ⋯ + c_{n}x^{n}.  (6.3)

The straightforward approach for solving the polynomial interpolation is try to determine the coefficients c_{0}, c_{1}, … c_{n} using the conditions

P_{n}(x_{i}) = y_{i}, i = 0, 1, 2, …, n.

These lead to

c_{0} + c_{1}x_{1} + c_{2}x_{1}^{2} + ⋯ + c_{n}x_{1}^{n} = y_{0},
c_{0} + c_{1}x_{2} + c_{2}x_{2}^{2} + ⋯ + c_{n}x_{2}^{n} = y_{1},
⋮
c_{0} + c_{1}x_{n} + c_{2}x_{n}^{2} + ⋯ + c_{n}x_{n}^{n} = y_{n},

which is equivalent to the system of linear equations:

[ 1 x_{1} x_{1}^{2} ⋯ x_{1}^{n} ; 1 x_{2} x_{2}^{2} ⋯ x_{2}^{n} ; ⋮ ; 1 x_{n} x_{n}^{2} ⋯ x_{n}^{n} ] [ c_{0} ; c_{1} ; ⋮ ; c_{n} ] = [ y_{0} ; y_{1} ; ⋮ ; y_{n} ] .  (6.4)

The coefficient matrix is called Vandermonde matrix. It can be proved that

det [ 1 x_{1} x_{1}^{2} ⋯ x_{1}^{n} ; 1 x_{2} x_{2}^{2} ⋯ x_{2}^{n} ; ⋮ ; 1 x_{n} x_{n}^{2} ⋯ x_{n}^{n} ] = ∏_{0≤j<k≤n}(x_{k} − x_{j}).  (6.5)

Since x_{i} are distinct, the Vandermonde system is nonsingular, and solving this system gives the coefficients c_{0}, c_{1}, …, c_{n} which solves the polynomial interpolation problem (6.1). However, a Vandermonde matrix is often very ill-conditioned, thus the computed solutions c_{i} will be inaccurate. Moreover, the amount of computational cost, including forming the matrix, factorization, and triangular substitutions, is excessive. Therefore, this approach is not recommended.

6.1.3 Lagrange Form and Neville’s Method

Next we present an alternative form for the interpolating polynomial P (x) associated with the n + 1 distinct data points (x_{i}, y_{i}) for 0 ≤ i ≤ n. It is important to remember that there is one and only one polynomial of degree less than or equal to n that solves the problem (6.1), various approaches produce the same polynomial in different forms.

Polynomial interpolation in Lagrange form expresses the polynomial as

P_{n}(x) = y_{0}L_{0}(x) + y_{1}L_{1}(x) + ⋯ + y_{n}L_{n}(x) = ∑_{k=0}^{n} y_{k}L_{k}(x),  (6.6)

where

L_{k}(x) = \frac{(x − x_{0}) ⋯ (x − x_{k−1})(x − x_{k+1}) ⋯ (x − x_{n})}{(x_{k} − x_{0}) ⋯ (x_{k} − x_{k−1})(x_{k} − x_{k+1}) ⋯ (x_{k} − x_{n})} = ∏_{{i=0 \atop i≠k}}^{n} \frac{x − x_{i}}{x_{k} − x_{i}} ,  (6.7)

are known as cardinal polynomials.

One advantage of using the Lagrange form of the interpolating polynomial is that it can be written down at once, since the coefficients in the Lagrange formula are the given y_{i}. This fact will be useful in numerical integration when quadrature formulas are constructed.

Example 6.1 Given the following 4 data points,

\begin{matrix}x_{i} & 0 & 1 & 3 & 5 \\ y_{i} & 1 & 2 & 6 & 7 \end{matrix}

find a polynomial in Lagrange form to interpolate these data.

Sol: The cardinal functions are

L_{0}(x) = \frac{(x − 1)(x − 3)(x − 5)}{(0 − 1)(0 − 3)(0 − 5)} = −\frac{1}{15}(x − 1)(x − 3)(x − 5),
L_{1}(x) = \frac{(x − 0)(x − 3)(x − 5)}{(1 − 0)(1 − 3)(1 − 5)} = \frac{1}{8}x(x − 3)(x − 5),
L_{2}(x) = \frac{(x − 0)(x − 1)(x − 5)}{(3 − 0)(3 − 1)(3 − 5)} = −\frac{1}{12}x(x − 1)(x − 5),
L_{3}(x) = \frac{(x − 0)(x − 1)(x − 3)}{(5 − 0)(5 − 1)(5 − 3)} = \frac{1}{40}x(x − 1)(x − 3).

The interpolating polynomial in the Lagrange form is

P_{3}(x) = L_{0}(x) + 2L_{1}(x) + 6L_{2}(x) + 7L_{3}(x).

One difficulty for the interpolating polynomial in the Lagrange form is that if more data points are added to the interpolation problem, all the cardinal functions have to be recalculated. We will now derive the interpolating polynomials in a manner that uses the previous calculations to greater advantage.

Definition 6.1 Let (x_{i}, y_{i}), 0 ≤ i ≤ n, where y_{i} = f(x_{i}) for some unknown function f, be (n + 1) given distinct data points. Suppose that m_{1}, m_{2}, …, m_{k} are k distinct integers with 0 ≤ m_{i} ≤ n for each i. The Lagrange polynomial that interpolates f at the k points x_{m_{1}}, x_{m_{2}}, …, x_{m_{k}} is denoted P_{m_{1},m_{2},...,m_{k}}(x).

Theorem 6.3 Let f be defined at distinct points x_{0}, x_{1}, …, x_{k}, and 0 ≤, i, j ≤ k, i ≠ j. Then

P_{0,1,...,k}(x) = \frac{(x − x_{j})}{(x_{i} − x_{j})}P_{0,1,...,j−1,j+1,...,k}(x) − \frac{(x − x_{i})}{(x_{i} − x_{j})}P_{0,1,...,i−1,i+1,...,k}(x)  (6.8)

describes the k-th Lagrange polynomial that interpolates f at the k + 1 points x_{0}, x_{1}, …, x_{k}.

Proof: Since deg(P_{0,1,...,j−1,j+1,...,k}) ≤ k − 1 and deg(P_{0,1,...,i−1,i+1,...,k}) ≤ k − 1, by definition deg(P_{0,1,...,k}) ≤ k. If r ≠ i, j, then

P_{0,1,...,k}(x_{r}) = \frac{(x_{r} − x_{j})}{(x_{i} − x_{j})}P_{0,1,...,j−1,j+1,...,k}(x_{r}) − \frac{(x_{r} − x_{i})}{(x_{i} − x_{j})}P_{0,1,...,i−1,i+1,...,k}(x_{r})
= \frac{(x_{r} − x_{j})}{(x_{i} − x_{j})}f(x_{r}) − \frac{(x_{r} − x_{i})}{(x_{i} − x_{j})}f(x_{r}) = f(x_{r}).

Moreover

P_{0,1,...,k}(x_{i}) = \frac{(x_{i} − x_{j})}{(x_{i} − x_{j})}P_{0,1,...,j−1,j+1,...,k}(x_{i}) − \frac{(x_{i} − x_{i})}{(x_{i} − x_{j})}P_{0,1,...,i−1,i+1,...,k}(x_{i}) = f(x_{i})

and

P_{0,1,...,k}(x_{j}) = \frac{(x_{j} − x_{j})}{(x_{i} − x_{j})}P_{0,1,...,j−1,j+1,...,k}(x_{j}) − \frac{(x_{j} − x_{i})}{(x_{i} − x_{j})}P_{0,1,...,i−1,i+1,...,k}(x_{j}) = f(x_{j}).

Therefore P_{0,1,...,k}(x) agrees with f at all points x_{0}, x_{1}, …, x_{k}. By the uniqueness theorem, P_{0,1,...,k}(x) is the k-th Lagrange polynomial that interpolates f at the k + 1 points x_{0}, x_{1}, …, x_{k}.

The theorem implies that the Lagrange interpolating polynomial can be generated recursively. The procedure is called the Neville’s method. For example, the polynomials can be computed in a manner as shown in the following table.

x_{0} P_{0} = Q_{0,0}
x_{1} P_{1} = Q_{1,0} P_{0,1} = Q_{1,1}
x_{2} P_{2} = Q_{2,0} P_{1,2} = Q_{2,1} P_{0,1,2} = Q_{2,2}
x_{3} P_{3} = Q_{3,0} P_{2,3} = Q_{3,1} P_{1,2,3} = Q_{3,2} P_{0,1,2,3} = Q_{3,3}
x_{4} P_{4} = Q_{4,0} P_{3,4} = Q_{4,1} P_{2,3,4} = Q_{4,2} P_{1,2,3,4} = Q_{4,3} P_{0,1,2,3,4} = Q_{4,4}

For ease of notation, we denote, in the table

Q_{i,j} = P_{i−j,i−j+1,...,i−1,i}.

Hence Q_{i,j}, 0 ≤ j ≤ i, denotes the interpolating polynomial of degree j on the j + 1 points x_{i−j}, x_{i−j+1}, …, x_{i−1}, x_{i}.

Note that the Neville’s method is usually used to generated successively higher degree polynomial approximations at a specified point, but not to generate the polynomials themselves.

6.1.4 Newton’s Form of Polynomial Interpolation

Since there is one and only one polynomial of degree ≤ n that takes prescribed values at given n + 1 distinct points, various algorithms will produce the same polynomial but in different forms. For example, the polynomial can be constructed as a linear combination of the basic polynomials 1, x, x^{2}, …, x^{n}. For numerical work, it is preferable to use the following basis

q_{0}(x) = 1,
q_{1}(x) = (x − x_{0}),
q_{2}(x) = (x − x_{0})(x − x_{1}),
⋮
q_{n}(x) = (x − x_{0})(x − x_{1}) ⋯ (x − x_{n−1}),

and express P (x) as

P_{n}(x) = c_{0} + c_{1}(x − x_{0}) + c_{2}(x − x_{0})(x − x_{1}) + ⋯ + c_{n}(x − x_{0})(x − x_{1}) ⋯ (x − x_{n−1}).
= ∑_{k=0}^{n} c_{k}q_{k}(x)  (6.9)

This polynomial is called the interpolating polynomial in Newton’s form.

The interpolation conditions

P_{n}(x_{i}) = ∑_{k=0}^{n} c_{k}q_{k}(x_{i}) = y_{i}, i = 0, 1, …, n,

give rise to n + 1 equations with n + 1 parameters c_{0}, c_{1}, …, c_{n} to be determined

c_{0} = y_{0},
c_{0} + c_{1}(x_{1} − x_{0}) = y_{1},
c_{0} + c_{1}(x_{2} − x_{0}) + c_{2}(x_{2} − x_{0})(x_{2} − x_{1}) = y_{2},
⋮
c_{0} + c_{1}(x_{n} − x_{0}) + c_{2}(x_{n} − x_{0})(x_{n} − x_{1}) + ⋯ + c_{n}(x_{n} − x_{0}) ⋯ (x_{n} − x_{n−1}) = y_{n}.

In matrix equation, we have a triangular linear system

[ \begin{matrix}1 & 0 & 0 & ⋯ & 0 \\ 1 & x_{1} − x_{0} & 0 & ⋯ & 0 \\ 1 & x_{2} − x_{0} & (x_{2} − x_{0})(x_{2} − x_{1}) & ⋯ & 0 \\ ⋮ & ⋮ & ⋮ & ... & ⋮ \\ 1 & x_{n} − x_{0} & (x_{n} − x_{0})(x_{n} − x_{1}) & ⋯ & (x_{n} − x_{0}) ⋯ (x_{n} − x_{n−1}) \end{matrix} ] [ c_{0} ; c_{1} ; c_{2} ; ⋮ ; c_{n} ] = [ y_{0} ; y_{1} ; y_{2} ; ⋮ ; y_{n} ] .

(6.10)

Use forward substitution, we can solve for c_{0}, c_{1}, …, c_{n}. Computational work involved in this approach includes forming and solving the system.

Example 6.2 Given the following 4 points (n = 3)

\begin{matrix}x_{i} & 0 & 1 & 3 & 5 \\ y_{i} & 1 & 2 & 6 & 7 \end{matrix}

find a polynomial of degree 3 in Newton’s form to interpolate these data.

Sol: Write

P (x) = c_{0} + c_{1}(x − 0) + c_{2}(x − 0)(x − 1) + c_{3}(x − 0)(x − 1)(x − 3)
= c_{0} + c_{1}x + c_{2}x(x − 1) + c_{3}x(x − 1)(x − 3).

Plug in the interpolation conditions,

P (0) = c_{0} = 1.
P (1) = c_{0} + c_{1} = 2.
P (3) = c_{0} + 3c_{1} + 6c_{2} = 6.
P (5) = c_{0} + 5c_{1} + 20c_{2} + 40c_{3} = 7,

which lead to

[ \begin{matrix}1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 1 & 3 & 6 & 0 \\ 1 & 5 & 20 & 40 \end{matrix} ] [ \begin{matrix}c_{0} \\ c_{1} \\ c_{2} \\ c_{3} \end{matrix} ] = [ \begin{matrix}1 \\ 2 \\ 6 \\ 7 \end{matrix} ]  ⟹  [ \begin{matrix}c_{0} \\ c_{1} \\ c_{2} \\ c_{3} \end{matrix} ] = [ 1 ; 1 ; \frac{1}{3} ; −\frac{17}{120} ] .

Thus

P (x) = 1 + x + \frac{1}{3}x(x − 1) − \frac{17}{120}x(x − 1)(x − 3).

6.1.5 Divided Differences Scheme

With proper arrangement, the triangular linear system in the previous subsection can be solved with the formulations

c_{0} = f(x_{0})
c_{1} = \frac{f(x_{1}) − f(x_{0})}{x_{1} − x_{0}}
c_{2} = \frac{1}{x_{2} − x_{0}} [\frac{f(x_{2}) − f(x_{1})}{x_{2} − x_{1}} − \frac{f(x_{1}) − f(x_{0})}{x_{1} − x_{0}}]
⋮

and so on. An efficient procedure, called the divided-difference method, can be derived for computing the n+1 coefficients c_{0}, c_{1}, ⋯, c_{n} in the Newton’s form interpolating polynomial.

First we introduce the divided-difference notation. The zero divided difference of the function f with respect to x_{i},

f[x_{i}] = f(x_{i}),  (6.11)

is simply the value of f at x_{i}. The first divided difference of f with respect to x_{i} and x_{i+1} is denoted and defined as

f[x_{i}, x_{i+1}] = \frac{f[x_{i+1}] − f[x_{i}]}{x_{i+1} − x_{i}}.  (6.12)

The second divided difference of f is defined as

f[x_{i}, x_{i+1}, x_{i+2}] = \frac{f[x_{i+1}, x_{i+2}] − f[x_{i}, x_{i+1}]}{x_{i+2} − x_{i}}.  (6.13)

In general, the k-th divided difference relative to x_{i}, x_{i+1}, …, x_{i+k} is given by

f[x_{i}, x_{i+1}, …, x_{i+k}] = \frac{f[x_{i+1}, x_{i+2}, …, x_{i+k}] − f[x_{i}, x_{i+1}, …, x_{i+k−1}]}{x_{i+k} − x_{i}}.  (6.14)

As might be expected from the evaluation of c_{0}, c_{1}, …, c_{n} in the Newton form interpolating polynomial, the required coefficients are given by

c_{k} = f[x_{0}, x_{1}, …, x_{k}],  k = 0, 1, …, n.  (6.15)

Therefore the interpolating polynomial in Newton’s form can be expressed as

P_{n}(x) = f[x_{0}] + ∑_{k=1}^{n} f[x_{0}, x_{1}, …, x_{k}](x − x_{0})(x − x_{1}) ⋯ (x − x_{k−1}).  (6.16)

This formula is known as the Newton’s interpolatory divided-difference formula. The determination of the divided differences is best described by using tabular form as shown in the following table as an example.

110  Chapter 6. Interpolation
x_{i} k = 0  k = 1  k = 2  k = 3
x_{0} f[x_{0}]
> f[x_{0}, x_{1}]
x_{1} f[x_{1}]  > f[x_{0}, x_{1}, x_{2}]
> f[x_{1}, x_{2}]  > f[x_{0}, x_{1}, x_{2}, x_{3}]
x_{2} f[x_{2}]  > f[x_{1}, x_{2}, x_{3}]
> f[x_{2}, x_{3}]
x_{3} f[x_{3}]

Example 6.3 Given the following 4 points (n = 3)

\begin{matrix}x_{i} & 0 & 1 & 3 & 5 \\ y_{i} & 1 & 2 & 6 & 7 \end{matrix}

find a polynomial of degree 3 in Newton’s form to interpolate these data. Sol:

x_{i} k = 0 k = 1 k = 2 k = 3
0  1
> 1
1 2 > \frac{1}{3}
> 2  > −\frac{17}{120}
3 6 > −\frac{3}{8}
> \frac{1}{2}
5  7

So,

P (x) = 1 + x + \frac{1}{3}x(x − 1) − \frac{17}{120}x(x − 1)(x − 3).

Note that x_{i} can be reordered, but must be distinct. When the order of x_{i} are changed, one obtains the same polynomial but in different form. Example 6.4 Given the following 4 points (n = 3)

\begin{matrix}x_{i} & 3 & 1 & 5 & 0 \\ y_{i} & 6 & 2 & 7 & 1 \end{matrix}

find a polynomial of degree 3 in Newton’s form to interpolate these data. Sol:

x_{i} k = 0 k = 1 k = 2 k = 3
3  6
> 2
1 2 > −\frac{3}{8}
> \frac{5}{4}  > −\frac{17}{120}
5 7 > \frac{1}{20}
> \frac{6}{5}
0  1

So,

P (x) = 6 + 2(x − 3) − \frac{3}{8}(x − 3)(x − 1) − \frac{17}{120}(x − 3)(x − 1)(x − 5).

Note that c_{0}, c_{1}, …, c_{n−1} are changed, but c_{n} is unchanged, and the polynomial is expressed in different form.

Theorem 6.4 Suppose f ∈ C^{n}[a, b] and x_{0}, x_{1}, …, x_{n} are distinct numbers in [a, b]. Then there exists ξ ∈ (a, b) such that

f[x_{0}, x_{1}, …, x_{n}] = \frac{f^{(n)}(ξ)}{n!}.

Sol: Let

P_{n}(x) = f[x_{0}] + ∑_{k=1}^{n} f[x_{0}, x_{1}, …, x_{k}](x − x_{0})(x − x_{1}) ⋯ (x − x_{k−1})

be the interpolating polynomial of f in Newton’s form. Define

g(x) = f(x) − P_{n}(x).

Since P_{n}(x_{i}) = f(x_{i}) for i = 0, 1, …, n, the function g has n + 1 distinct zeros in [a, b]. By the generalized Rolle’s Theorem, there exists ξ ∈ (a, b) such that

g^{(n)}(ξ) = f^{(n)}(ξ) − P_{n}^{(n)}(ξ) = 0.

Note that

P_{n}^{(n)}(x) = n!f[x_{0}, x_{1}, …, x_{n}].

As a consequence

f[x_{0}, x_{1}, …, x_{n}] = \frac{f^{(n)}(ξ)}{n!}.

An algorithm, to be introduced, for computing the divided-difference table is very efficient and is recommended as the best scheme for producing an interpolating polynomial in Newton’s form. First we change the notation to

a_{i,k} = f[x_{i}, x_{i+1}, …, x_{i+k}]
= \frac{f[x_{i+1}, x_{i+2}, …, x_{i+k}] − f[x_{i}, x_{i+1}, …, x_{i+k−1}]}{x_{i+k} − x_{i}} = \frac{a_{i+1,k−1} − a_{i,k−1}}{x_{i+k} − x_{i}}

so that the divided-difference table has the entries as shown in the following.

x_{i} k = 0  k = 1  k = 2  k = n
x_{0} a_{00}
⟩ a_{01}
x_{1} a_{10}  ⟩ a_{02}
⟩ a_{12}  ...
x_{2} a_{20}  ⟩ a_{12}  ...
⟩ a_{21}  ⋯ a_{0,n}
x_{3} a_{30}  ⋮  ...
⋮  ⋮  ...
⋮  ⋮  ...
⟩ a_{n−1,1}
x_{n} a_{n,0}

The interpolating polynomial, of course, is

P_{n}(x) = a_{00} + ∑_{k=1}^{n} a_{0,k} ∏_{j=0}^{k−1}(x − x_{j}).  (6.17)

Algorithm 6.1 (Divided-Difference Table) Given n + 1 distinct data points (x_{i}, y_{i}), i = 0, 1, …, n, this algorithm computes the coefficients of the interpolating polynomial in Newton’s form by constructing the divided-difference table.

for i = 0, 1, … n do

a(i, 0) = y(i) = f(x(i)) end for for k = 1, …, n do

for i = 0, 1, …, n − k do

a(i, k) = \frac{a(i + 1, k − 1) − a(i, k − 1)}{x(i + k) − x(i)}

end for end for

This algorithm uses one 2-d array of (n + 1) × (n + 1) for the storage of all a_{ij}. The inner i-loop can be performed in parallel. If only the coefficients in the Newton’s interpolating polynomial are required for one set of data, an improved algorithm which uses only one 1-d array of size n + 1 can be derived. The modified algorithm, however, can not be executed in parallel.

Algorithm 6.2 (Divided-Difference Algorithm) Given n + 1 distinct points (x_{i}, y_{i}), i = 0, 1, …, n, this algorithm computes the coefficients of the interpolating polynomial in Newton’s form using an 1-d array of size n + 1.. for i = 0, 1, …, n do

c(i) = y(i) = f(x(i)) end for for k = 1, …, n do

for i = n, n − 1, …, k do

c(i) = \frac{c(i) − c(i − 1)}{x(i) − x(i − k)}

end for end for

Newton’s interpolatory divided-difference formula can be expressed in a simplified form when x_{0}, x_{1}, …, x_{n} are arranged consecutively with equal spacing. Let

h = \frac{x_{n} − x_{0}}{n} = x_{i+1} − x_{i}, i = 0, 1, …, n − 1.

Then each x_{i} = x_{0} + i ∗ h, i = 0, 1, …, n. For any x ∈ [a, b], write

x = x_{0} + s ∗ h, s ∈ ℝ.

Hence x − x_{i} = (s − i) ∗ h and

P_{n}(x) = f[x_{0}] + ∑_{k=1}^{n} f[x_{0}, x_{1}, …, x_{k}](x − x_{0})(x − x_{1}) ⋯ (x − x_{k−1})

= f(x_{0}) + ∑_{k=1}^{n} f[x_{0}, x_{1}, …, x_{k}](s − 0) ∗ h ∗ (s − 1) ∗ h ∗ ⋯ (s − k + 1) ∗ h

= f(x_{0}) + ∑_{k=1}^{n} f[x_{0}, x_{1}, …, x_{k}]s(s − 1) ⋯ (s − k + 1)h^{k}

= f(x_{0}) + ∑_{k=1}^{n} f[x_{0}, x_{1}, …, x_{k}]k!\binom{s}{k}h^{k},  (6.18)

where the binomial formula

\binom{s}{k} = \frac{s(s − 1) ⋯ (s − k + 1)}{k!}

is used. This formula is called the Newton forward divided-difference formula. Next we introduce the forward difference notation △

△f(x_{i}) = f(x_{i+1}) − f(x_{i})  (6.19)

and

△^{k}f(x_{i}) = △^{k−1}f(x_{i+1}) − △^{k−1}f(x_{i}) = △ (△^{k−1}f(x_{i})) ,  (6.20)

for i = 0, 1, …, n − 1. With this notation,

f[x_{0}, x_{1}] = \frac{f(x_{1}) − f(x_{0})}{x_{1} − x_{0}} = \frac{1}{h}△f(x_{0})
f[x_{0}, x_{1}, x_{2}] = \frac{f[x_{1}, x_{2}] − f[x_{0}, x_{1}]}{x_{2} − x_{0}} = \frac{\frac{1}{h}△f(x_{1}) − \frac{1}{h}△f(x_{0})}{2h} = \frac{1}{2h^{2}} △^{2}f(x_{0}),

and, in general

f[x_{0}, x_{1}, …, x_{k}] = \frac{1}{k! h^{k}} △^{k}f(x_{0}).  (6.21)

Using the forward difference △ notation, the Newton’s forward divided-difference formula can be expressed as

P_{n}(x) = f(x_{0}) + ∑_{k=1}^{n} \binom{s}{k}△^{k}f(x_{0}).  (6.22)

This formula is called the Newton forward-difference formula.

If the interpolation nodes are arranged as x_{n}, x_{n−1}, …, x_{0}, a formula for the interpolating polynomial similar to results

P_{n}(x) = f[x_{n}] + ∑_{k=1}^{n} f[x_{n}, x_{n−1}, …, x_{n−k}](x − x_{n})(x − x_{n−1}) ⋯ (x − x_{n−k+1}).  (6.23)

If the nodes are equally spaced with

h = \frac{x_{n} − x_{0}}{n} ,  x_{i} = x_{n} − (n − i) ∗ h,  x = x_{n} + s ∗ h,

then

P_{n}(x) = f[x_{n}] + ∑_{k=1}^{n} f[x_{n}, x_{n−1}, …, x_{n−k}]s ∗ h ∗ (s + 1) ∗ h ∗ ⋯ ∗ (s + k − 1) ∗ h

= f(x_{n}) + ∑_{k=1}^{n} f[x_{n}, x_{n−1}, …, x_{n−k}](−1)^{k}k!\binom{−s}{k}h^{k}  (6.24)

where the binomial formula is extended to include all real value s

\binom{−s}{k} = \frac{−s(−s − 1) ⋯ (−s − k + 1)}{k!} = (−1)^{k} \frac{s(s + 1) ⋯ (s + k − 1)}{k!}.

This form is called the Newton backward divided-difference formula. Similarly, we can introduce the backward-difference notation

∇^{k}f(x_{i}) = ∇^{k−1}f(x_{i}) − ∇^{k−1}f(x_{i−1}) = ∇ (∇^{k−1}f(x_{i})) ,  (6.25)

then

f[x_{n}, x_{n−1}] = \frac{1}{h}∇f(x_{n}), f[x_{n}, x_{n−1}, x_{n−2}] = \frac{1}{2h^{2}} ∇^{2}f(x_{n}),

and, in general,

f[x_{n}, x_{n−1}, …, x_{n−k}] = \frac{1}{k! h^{k}} ∇^{k}f(x_{n}).  (6.26)

Using this backward-difference ∇ notation, the Newton backward divided-difference formula can be written as

P_{n}(x) = f(x_{0}) + ∑_{k=1}^{n}(−1)^{k}\binom{−s}{k}∇^{k}f(x_{k}).  (6.27)

This is called the Newton backward-difference formula.

6.1.6 Error Analysis for Polynomial Interpolation

Theorem 6.5 Suppose f(x) ∈ C^{n+1}[a, b], and P (x) is a polynomial of degree ≤ n that interpolates f at n + 1 given distinct points x_{0}, x_{1} …, x_{n} in [a, b]. Then for each x ∈ [a, b], there exists a point ξ_{x} ∈ (a, b) such that

f(x) − P (x) = \frac{1}{(n + 1)!}f^{(n+1)}(ξ_{x}) ∏_{k=0}^{n}(x − x_{k}).  (6.28)

Proof: Note first that if x = x_{i} for some 0 ≤ i ≤ n, then P (x) = f(x), the theorem holds for any arbitrary ξ_{x} ∈ (a, b). Suppose that x ∈ [a, b] but x ≠ x_{i} for all i = 0, 1, …, n. Define a new function

g(t) = f(t) − P (t) − [f(x) − P (x)] ∏_{k=0}^{n} \frac{(t − x_{k})}{(x − x_{k})}, t ∈ [a, b].

Since f ∈ C^{n+1}[a, b], P ∈ C^{∞}[a, b], and x ≠ x_{i}, ∀ i, it follows that g(t) is well-defined for all t ∈ [a, b], and g ∈ C^{n+1}[a, b]. Then

g(x_{i}) = f(x_{i}) − P (x_{i}) − [f(x) − P (x)] ∏_{k=0}^{n} \frac{(x_{i} − x_{k})}{(x − x_{k})} = 0, i = 0, 1, …, n.

Moreover

g(x) = f(x) − P (x) − [f(x) − P (x)] ∏_{k=0}^{n} \frac{(x − x_{k})}{(x − x_{k})} = 0.

Thus g ∈ C^{n+1}[a, b] and g has n + 2 zeros in [a, b]. By the generalized Rolle’s theorem, there exists ξ_{x} ∈ (a, b) for which g^{(n+1)}(ξ_{x}) = 0. Hence

0 = g^{(n+1)}(ξ_{x}) = f^{(n+1)}(ξ_{x}) − P^{(n+1)}(ξ_{x}) − [f(x) − P (x)] [\frac{d^{n+1}}{dt^{n+1}} ∏_{k=0}^{n} \frac{(t − x_{k})}{(x − x_{k})}] |_{t=ξ_{x}}

= f^{(n+1)}(ξ_{x}) − 0 − [f(x) − P (x)] · \frac{(n + 1)!}{∏(x − x_{k})}.

Therefore

f(x) − P (x) = \frac{1}{(n + 1)!}f^{(n+1)}(ξ_{x}) ∏_{k=0}^{n}(x − x_{k}).

Example 6.5 Suppose f(x) = sin x is approximated by an interpolating polynomial p(x) of degree 9 in [0, 1]. Estimate |f(x) − p(x)|, for all x ∈ [0, 1].

Sol: With n = 9, and since f(x) = sin x, |f^{(10)}(ξ)| ≤ 1, and x ∈ [0, 1],

|x − x_{i}| ≤ 1, ⟹ |∏_{i=0}^{10}(x − x_{i})| ≤ 1,

we have

|f(x) − p(x)| = \frac{1}{10!} |f^{(10)}(ξ)| |∏_{i=0}^{10}(x − x_{i})| ≤ \frac{1}{10!}.

6.2 Hermite Interpolation

If values of a function f and some of its derivatives are to be interpolated by a polynomial, the topic is known as Birkhoff interpolation. The general problem of this type may have some intriguing difficulties associated with it because the linear system of equations from which we expect to compute the coefficients in the polynomial may be singular.

Example 6.6 Find a polynomial p that assumes theses values:

p(0) = 0, p(1) = 1, p′ (\frac{1}{2}) = 2.

Sol: Since there are three conditions, we try a quadratic polynomial

p(x) = a + bx + cx^{2}.

Then p′(x) = b + 2cx. Plug in the given conditions, we have

a = 0
a + b + c = 1
b + c = 2

However the coefficient matrix of this linear system of equations

[ \begin{matrix}1 & 0 & 0 \\ 1 & 1 & 1 \\ 0 & 1 & 1 \end{matrix} ]

is singular. There is no solution for coefficients a, b, and c. Thus there is no polynomial of degree at most 2 that solves this problem.

The problems in a restricted class are the ones generally known as Hermite interpolation. Consider the given n + 1 data points x_{0} < x_{1} < ⋯ < x_{n}, and real numbers y_{i}^{(k)}, k = 0, 1, …, m_{i} − 1, i = 0, 1, …, n, where

y_{0}^{(0)} = f(x_{0})  y_{1}^{(0)} = f(x_{1})  ⋯  y_{n}^{(0)} = f(x_{n})
y_{0}^{(1)} = f′(x_{0})  y_{1}^{(1)} = f′(x_{1})  ⋯  y_{n}^{(1)} = f′(x_{n})
⋮  ⋮  ⋮
y_{0}^{(m_{0}−1)} = f^{(m_{0}−1)}(x_{0}) y_{1}^{(m_{1}−1)} = f^{(m_{1}−1)}(x_{1}) ⋯ y_{n}^{(m_{n}−1)} = f^{(m_{n}−1)}(x_{n})
↓  ↓  ↓
m_{0} conditions  m_{1} conditions  ⋯  m_{n} conditions

for some function f. The Hermite interpolation problem for these data consists of determining a polynomial P of degree at most N, where

N = (∑_{i=0}^{n} m_{i}) − 1.  (6.29)

satisfies the following interpolation conditions

P^{(k)}(x_{i}) = y_{i}^{(k)},  j = 0, 1, … (m_{i} − 1), i = 0, 1, …, n.  (6.30)

In a Hermite interpolation problem, it is assumed that whenever a derivative p^{(j)}(x_{i}) is to be prescribed at a node x_{i}, then p^{(j−1)}(x_{i}), p^{(j−2)}(x_{i}), … p′(x_{i}), and p(x_{i}) will also prescribed. The polynomial interpolation of the previous section is the special case where n_{i} = 1 for all i = 0, 1, …, n.

6.2.1 Existence and Uniqueness

Theorem 6.6 For arbitrary numbers x_{0} < x_{1} < ⋯ < x_{n}, and y_{i}^{(k)}, k = 0, 1, …, (m_{i} − 1), i = 0, 1, …, n, there exists a unique polynomial P of degree at most N, where N = (∑_{i=0}^{n} m_{i}) − 1, which solves the Hermite interpolation problem (6.30).

Proof: Since the polynomial P to be sought has degree at most N, write

P (x) = a_{0} + a_{1}x + ⋯ + a_{N}x^{N},

There are N + 1 coefficents, a_{0}, a_{1}, …, a_{N}, to be determined and there are exactly N + 1 conditions given, thus we have a square system of N + 1 linear equations in N + 1 unknowns to solve, and we wish to show that the coefficient matrix A of this system is nonsingular.

To prove A is nonsingular, it suffices to prove that the homogeneous system Au = 0 has only the trivial solution u = 0. In the Hermite interpolation problem under discussion, the homogeneous problem is to find a polynomial P of degree at most N such that

P^{(k)}(x_{i}) = 0, k = 0, 1, …, m_{i} − 1, i = 0, 1, …, n.

Such a polynomial has a zero of multiplicity m_{i} at x_{i} and must therefore be a multiple of the polynomial given by

q(x) = ∏_{i=0}^{n} (x − x_{i})^{m_{i}} .

However, deg(q) = ∑_{i=0}^{n} m_{i} = N +1 whereas P has degree at most N. We therefore conclude that P (x) = q(x) = 0. That is, A is nonsingular, and the Hermite interpolation problem has a unique solution.

6.2.2 Lagrange Form for Hermite Interpolation

Hermite interpolating polynomial can be given explicitly in a form analogous to the polynomial interpolation formula in Lagrange form. The Hermite interpolating polynomial in Lagrange form is given by

P (x) = ∑_{i=0}^{n} ∑_{k=0}^{m_{i}−1} y_{i}^{(k)}L_{ik}(x),  (6.31)

where, for i = 0, 1, …, n,

L_{ik}(x) = \begin{cases}ℓ_{i,m_{i}−1}(x), & for k = m_{i} − 1, \\ ℓ_{ik}(x) − ∑_{j=k+1}^{m_{i}−1} ℓ_{ik}^{(j)}(x_{i})L_{ij}(x), & for k = m_{i} − 2, m_{i} − 3, …, 0, \end{cases} (6.32)

with the auxiliary polynomial

ℓ_{ik}(x) = \frac{(x − x_{i})^{k}}{k!} ∏_{{j=0 \atop j≠i}}^{n} (\frac{x − x_{j}}{x_{i} − x_{j}} )^{m_{j}} ,  0 ≤ k ≤ m_{i} − 1, 0 ≤ i ≤ n.  (6.33)

By induction

L_{ik}^{(s)}(x_{t}) = \begin{cases}1, & if i = t and k = s; \\ 0, & otherwise. \end{cases}

Thus P (x) is indeed the desired Hermite interpolating polynomial.

6.2.3 Divided Difference Method for Hermite Interpolation

Next we illustrate how the Newton divided difference method can be extended to solve Hermite interpolation problems. First we denote the N + 1 conditions pairs

(x_{0}, y_{0}^{(0)}), (x_{0}, y_{0}^{(1)}), … (x_{0}, y_{0}^{(m_{0}−1)}), (x_{1}, y_{1}^{(0)}), (x_{1}, y_{1}^{(1)}), … (x_{1}, y_{1}^{(m_{1}−1)}), … (x_{n}, y_{1}^{(m_{n}−1)})

by consecutive coordinate pairs

(\underbrace{z_{0}}_{x_{0}}, y_{0}^{(0)}), (\underbrace{z_{1}}_{x_{0}}, y_{0}^{(1)}), ⋯, (\underbrace{z_{m_{0}−1}}_{x_{0}}, y_{0}^{(m_{0}−1)}), (\underbrace{z_{m_{0}}}_{x_{1}}, y_{1}^{(0)}), (\underbrace{z_{m_{0}+1}}_{x_{1}}, y_{1}^{(1)}), ⋯ (\underbrace{z_{N}}_{x_{n}}, y_{n}^{(m_{n}−1)}).

Note that z_{0} ≤ z_{1} ≤ ⋯ ≤ z_{N}. The unique Hermite interpolating polynomial in Newton’s form can then be written as

P (x) = f[z_{0}] + ∑_{k=1}^{N} f[z_{0}, z_{1}, …, z_{k}](x − z_{0})(x − z_{1}) ⋯ (x − z_{k−1}).  (6.34)

The coefficient f[z_{0}, z_{1}, …, z_{k}] is again computed by the divided-difference method. If z_{i} ≠ z_{i+k}, then

f[z_{i}, z_{i+1}, …, z_{i+k}] = \frac{f[z_{i+1}, z_{i+2}, …, z_{i+k}] − f[z_{i}, z_{i+1}, …, z_{i+k−1}]}{z_{i+k} − z_{i}}.  (6.35)

However the divided-difference formula has to be modified because there may be repetitions among z_{i}. The extension of the definition of divided-differences to the case of repeated arguments involves transition to a limit. When z_{i} = z_{i+1} = ⋯ = z_{i+k}, let

ξ_{i} < ξ_{i+1} < ⋯ < ξ_{i+k}

be distinct coordinates and ξ_{j} → z_{j}, for i ≤ j ≤ i + k. Then, by applying Theorem 6.4,

f[z_{i}, z_{i+1}, …, z_{i+k}] = lim_{{ξ_{j}→z_{j} \atop i≤j≤i+k}} f[ξ_{i}, ξ_{i+1}, …, ξ_{i+k}] = \frac{f^{(k)}(z_{i})}{k!}  (6.36)

In summary, the following algorithm computes the coefficients of the polynomial which solves the Hermite interpolation problem.

Algorithm 6.3 (Divided-Difference Method for Hermite Interpolation) Given arbitrary x_{0} < x_{1} < ⋯ < x_{n}, and y_{i}^{(k)}, k = 0, 1, …, (m_{i} − 1), i = 0, 1, …, n, this algorithm computes the coefficients of the polynomial P (x) such that P^{(k)}(x_{i}) = y_{i}^{(k)}, j = 0, 1, … (m_{i} − 1), i = 0, 1, …, n.

if z_{i} = z_{i+k} then

f[z_{i}, z_{i+1}, …, z_{i+k}] = \frac{f^{(k)}(z_{i})}{k!}

else

f[z_{i}, z_{i+1}, …, z_{i+k}] = \frac{f[z_{i+1}, …, z_{i+k}] − f[z_{i+1}, …, z_{i+k−1}]}{z_{i+k} − z_{i}}

end if

Example 6.7 Given

x_{0} = 0  x_{1} = 1
f(0) = 1, f(1) = 2,
f′(0) = 2, f′(1) = −2,
f′′(1) = −3.

Find a polynomial of degree 4 which interpolates these data.

Sol: Constructing the divided-difference table with the previous algorithm.

z_{i} f[z_{i}]
0 1
> 2
0 1  > -1
> 1  > -2
1 2 > -3 > \frac{7}{2}
> -2  > \frac{3}{2}
1 2  > −\frac{3}{2}
> -2
1 2

The polynomial sought is

p(x) = 1 + 2x + (−1)x^{2} − 2x^{2}(x − 1) + \frac{7}{2}x^{2}(x − 1)^{2}.

6.2.4 Error Analysis for Hermite Interpolation

The interpolation error which is incurred by Hermite interpolation can be estimated in the same fashion as for the usual polynomial interpolation.

Theorem 6.7 Suppose x_{0} < x_{1} < ⋯ < x_{n}, x_{i} ∈ [a, b]. If the polynomial P is of degree at most N, where

N = (∑_{i=0}^{n} m_{i}) − 1,

and satisfies the interpolation conditions

P^{(k)}(x_{i}) = f^{(k)}(x_{i}), k = 0, 1, …, m_{i} − 1, i = 0, 1, …, n,

and f ∈ C^{N+1}[a, b], then for any x ∈ [a, b] there exists ξ_{x} ∈ [a, b] such that

f(x) − P (x) = \frac{1}{(N + 1)!}f^{(N+1)}(ξ_{x}) ∏_{i=0}^{n}(x − x_{i})^{m_{i}}.  (6.37)

6.3 Spline Interpolation

The previous sections concern the approximation of an arbitrary function on a closed interval by a polynomial. However, the oscillatory nature of high-degree polynomials restricts their use. An alternative approach is to divide the interval into a collection of subintervals and construct different approximation on each subinterval. Approximation of this type is called piecewise polynomial interpolation.

The simplest piecewise polynomial approximation is piecewise linear interpolation which consists of joining a set of data points by a series of straight line segments. A disadvantage of linear function approximation is that there is no assurance of differentiability at each of the endpoints of the subintervals, which, in a geometrical context, means that the interpolating function is not smooth at these points. It is often required that the approximating function is continuously differentiable.

An alternative procedure is to use a piecewise polynomial of Hermite type. However, to use Hermite piecewise polynomials for general interpolation, we need to know the derivatives of the function being approximated, which is frequently not available.

In this section, we consider approximation using spline function that require no derivative information, except perhaps at the endpoints of the intervals on which the function is being approximated. A spline function consists of polynomial pieces on subintervals joined together with certain continuity conditions.

Definition 6.2 Suppose the interval [a, b] is partitioned by a = x_{0} < x_{1} < ⋯ < x_{n} = b. A spline function S_{Δ} of degree k defined on [a, b] is a function such that • S_{Δ} is a polynomial of degree at most k on each subinterval [x_{i}, x_{i+1}), namely,

S_{Δ} = \begin{cases}S_{0}(x), & x ∈ [x_{0}, x_{1}), \\ S_{1}(x), & x ∈ [x_{1}, x_{2}), \\ ⋮ & ⋮ \\ S_{n−1}(x), & x ∈ [x_{n−1}, x_{n}], \end{cases}

and deg(S_{i}) ≤ k, i = 0, 1, …, n − 1;

• S_{Δ} is k − 1 continuous differentiable on [x_{0}, x_{n}], that is, S_{Δ} ∈ C^{k−1}[x_{0}, x_{n}].

The spline function S_{Δ}(x) can be viewed as a collection of n piecewise continuous polynomials of degree at most k having continuous derivatives of all order up to k − 1. Spline functions yield smooth interpolating curves which are less likely to exhibit the large oscillations characteristics of high-degree polynomials.

Given n + 1 distinct points x_{0} < x_{1} < ⋯ < x_{n} and the values, y_{i} = f(x_{i}), i = 0, 1, …, n, of function f, we try to find a spline function S_{Δ} to interpolate f(x).

6.3.1 Cubic Spline

A spline of degree 3 is called a cubic spline.

• S_{Δ} consists of cubic polynomials (k = 3) on each of the subinterval [x_{i}, x_{i+1}) glued together;

• S_{Δ} is twice continuous differentiable on [x_{0}, x_{n}]. i.e., S_{Δ} ∈ C^{2}[x_{0}, x_{n}].

S_{Δ} = \begin{cases}S_{0}(x), & x ∈ [x_{0}, x_{1}], \\ S_{1}(x), & x ∈ [x_{1}, x_{2}], \\ S_{2}(x), & x ∈ [x_{2}, x_{3}], \\ ⋮ & ⋮ \\ S_{n−1}(x), & x ∈ [x_{n−1}, x_{n}]. \end{cases}

with

S_{i−1}(x_{i}) = y_{i} = S_{i}(x_{i}), i = 1, 2, …, n − 1.

Each S_{i}(x) is a polynomial of degree 3, therefore, there are 4 parameters to be determined for each S_{i}(x). Hence, 4n parameters in total need to be computed. On each subinterval,

S_{i}(x_{i}) = y_{i},  S_{i}(x_{i+1}) = y_{i+1},  i = 0, 1, 2, …, n − 1.  (6.38)

These give 2n conditions. The twice conditions differentiable conditions lead to

S′_{i−1}(x_{i}) = S′_{i}(x_{i}),  S′′_{i−1}(x_{i}) = S′′_{i}(x_{i}),  i = 1, 2, …, n − 1,  (6.39)

and give 2n − 2 conditions. Together, we have 4n − 2 conditions and 4n parameters to be determined. This leaves 2 degree of freedom.

Since S_{i}(x) is a polynomial of degree 3, hence S′′_{i}(x) is a polynomial of degree 1. We may write

S′′_{i}(x) = A_{i}x + B_{i}.

Assume

\begin{cases}S′′_{i}(x_{i}) & = z_{i}, \\ S′′_{i}(x_{i+1}) & = z_{i+1}, \end{cases}  i = 0, 1, 2, …, n − 1.  (6.40)

Then

\begin{cases}A_{i}x_{i} + B_{i} & = z_{i}, \\ A_{i}x_{i+1} + B_{i} & = z_{i+1}. \end{cases}

Let h_{i} = x_{i+1} − x_{i}. Then

A_{i} = \frac{z_{i+1} − z_{i}}{x_{i+1} − x_{i}} = \frac{z_{i+1} − z_{i}}{h_{i}} .

and

B_{i} = z_{i} − A_{i}x_{i} = z_{i} − \frac{z_{i+1} − z_{i}}{h_{i}} x_{i}.

Hence,

S′′_{i}(x) = \frac{z_{i+1} − z_{i}}{h_{i}} x + z_{i} − \frac{z_{i+1} − z_{i}}{h_{i}} x_{i}
= \frac{z_{i+1}}{h_{i}} x − \frac{z_{i}}{h_{i}} x + z_{i} − \frac{z_{i}}{h_{i}} x_{i} + \frac{z_{i+1}}{h_{i}} x_{i}
= \frac{z_{i+1}}{h_{i}} (x − x_{i}) − \frac{z_{i}}{h_{i}} x + z_{i} + \frac{z_{i}}{h_{i}} (x_{i+1} − h_{i})
= \frac{z_{i+1}}{h_{i}} (x − x_{i}) − \frac{z_{i}}{h_{i}} x + z_{i} + \frac{z_{i}}{h_{i}} x_{i+1} − z_{i}
= \frac{z_{i+1}}{h_{i}} (x − x_{i}) + \frac{z_{i}}{h_{i}} (x_{i+1} − x)
= \frac{z_{i+1}}{h_{i}} (x − x_{i}) − \frac{z_{i}}{h_{i}} (x − x_{i+1}),  i = 1, 2, …, n − 1,

and

S′_{i}(x) = \frac{z_{i+1}}{h_{i}} ∫ (x − x_{i}) dx + \frac{z_{i}}{h_{i}} ∫ (x_{i+1} − x) dx
= \frac{1}{2} \frac{z_{i+1}}{h_{i}} (x − x_{i})^{2} − \frac{1}{2} \frac{z_{i}}{h_{i}} (x_{i+1} − x)^{2} + F_{i},

Therefore

S_{i}(x) = \frac{1}{6} \frac{z_{i+1}}{h_{i}} (x − x_{i})^{3} − \frac{1}{6} \frac{z_{i}}{h_{i}} (x − x_{i+1})^{3} + F_{i}(x − x_{i}) + G_{i},

Using the conditions S_{i}(x_{i}) = y_{i} and S_{i}(x_{i+1}) = y_{i+1}, we can determine F_{i} and G_{i}.

y_{i} = −\frac{1}{6} \frac{z_{i}}{h_{i}} (x_{i} − x_{i+1})^{3} + G_{i}, ⟹ G_{i} = y_{i} − \frac{1}{6}z_{i}h_{i}^{2}.

and

y_{i+1} = \frac{1}{6} \frac{z_{i+1}}{h_{i}} (x_{i+1} − x_{i})^{3} + F_{i}(x_{i+1} − x_{i}) + G_{i} ⟹ F_{i} = \frac{y_{i+1} − y_{i}}{h_{i}} + \frac{1}{6}h_{i}(z_{i} − z_{i+1}).

Hence,

S_{i}(x) = \frac{1}{6} \frac{z_{i+1}}{h_{i}} (x − x_{i})^{3} − \frac{1}{6} \frac{z_{i}}{h_{i}} (x − x_{i+1})^{3} + [\frac{1}{h_{i}} (y_{i+1} − y_{i}) + \frac{1}{6}h_{i}(z_{i} − z_{i+1})] (x − x_{i})

+(y_{i} − \frac{1}{6}z_{i}h_{i}^{2}), i = 0, 1, 2, … n − 1,

and

S′_{i}(x) = \frac{1}{2} \frac{z_{i+1}}{h_{i}} (x − x_{i})^{2} − \frac{1}{2} \frac{z_{i}}{h_{i}} (x − x_{i+1})^{2} + \frac{1}{h_{i}} (y_{i+1} − y_{i}) + \frac{1}{6}h_{i}(z_{i} − z_{i+1}).

What left is to determine z_{i} = S′′_{i}(x_{i}), for i = 0, 1, …, n − 1. Since S′_{i−1}(x_{i}) = S′_{i}(x_{i}), for i = 1, 2, …, n − 1,

\frac{1}{6}h_{i−1}z_{i−1} + \frac{1}{3}h_{i−1}z_{i} + \frac{1}{h_{i−1}} (y_{i} − y_{i−1}) = −\frac{1}{3}h_{i}z_{i} − \frac{1}{6}h_{i}z_{i+1} + \frac{1}{h_{i}} (y_{i+1} − y_{i}),

which leads to

h_{i−1}z_{i−1} + 2(h_{i−1} + h_{i})z_{i} + h_{i}z_{i+1} = \frac{6}{h_{i}} (y_{i+1} − y_{i}) − \frac{6}{h_{i−1}} (y_{i} − y_{i−1}).

If we define

c_{i} = \frac{6}{h_{i}} (y_{i+1} − y_{i}),

then

\frac{6}{h_{i}} (y_{i+1} − y_{i}) − \frac{6}{h_{i−1}} (y_{i} − y_{i−1}) = c_{i} − c_{i−1}.

Let b_{i} = c_{i} − c_{i−1}. Then

h_{i−1}z_{i−1} + 2(h_{i−1} + h_{i})z_{i} + h_{i}z_{i+1} = b_{i},  (6.41)

for i = 1, 2, …, n − 1. There are n − 1 equations with n + 1 unknowns z_{0}, z_{1}, …, z_{n}. Recall, we have 2 degrees of freedom. The widely used choice is to let z_{0} = z_{n} = 0, i.e.,

S′′_{Δ}(x_{0}) = S′′_{Δ}(x_{n}) = 0

This is called the natural spline. With this convention, we have n − 1 equations and want to solve n − 1 unknowns z_{1}, z_{2}, …, z_{n−1}. equation () leads to a linear system of following form.

[ \begin{matrix}2(h_{0} + h_{1}) & h_{1} \\ h_{1} & 2(h_{1} + h_{2}) & h_{2} \\ & h_{2} & 2(h_{2} + h_{3}) & h_{3} \\ & & ... & ... & ... \\ & & & ... & ... & h_{n−2} \\ & & & & h_{n−2} & 2(h_{n−2} + h_{n−1}) \end{matrix} ] [ z_{1} ; z_{2} ; z_{3} ; ⋮ ; ⋮ ; z_{n−1} ] = [ b_{1} ; b_{2} ; b_{3} ; ⋮ ; ⋮ ; b_{n−1} ]

The coefficient matrix is tridiagonal, symmetric, and diagonally dominant. Hence, Gaussian elimination without pivoting can be used to solve the system for z_{1}, …, z_{n}. Once z_{i} are computed, S_{i}(x) can be determined.

Numerical Differentiation and Integration

7.1 Numerical Differentiation

7.1.1 Finite Difference Method

Suppose a given function f has continuous first derivative and f′′ exists. From Taylor’s theorem

f(x + h) = f(x) + f′(x)h + \frac{1}{2}f′′(ξ)h^{2}, , h > 0,

where ξ is between x and x + h, one has

f′(x) = \frac{f(x + h) − f(x)}{h} − \frac{h}{2}f′′(ξ) = \frac{f(x + h) − f(x)}{h} + O(h).

Hence it is reasonable to use the approximation

f′(x) ≈ \frac{f(x + h) − f(x)}{h}  (7.1)

difference, and the error involved is

|e| = \frac{h}{2}|f′′(ξ)| ≤ max_{t∈(x,x+h)} |f′′(t)|.  (7.2)

This kind of error is called the truncation error and is unavoidable when using finite difference approach because the Taylor series is truncated. A only special case is when f is a linear function in which e = 0.

Similarly one can derive the backward finite difference approximation

f′(x) ≈ \frac{f(x) − f(x − h)}{h}  (7.3)

which has the same order of truncation error as the forward finite difference scheme.

In theory, the truncation error e → 0 as h → 0. However, the roundoff error will kick in when h is small. This is because when h is small

x ≈ x + h ⟹ f(x) ≈ f(x + h)

since f is continuous. This results cancellation of significance in evaluating f(x + h) − f(x). Higher precision operation should be used to prevent such subtractive cancellation.

The forward difference is an O(h) scheme. An O(h^{2}) scheme can also be derived from the Taylor’s theorem

f(x + h) = f(x) + f′(x)h + \frac{1}{2}f′′(x)h^{2} + \frac{1}{6}f′′′(ξ_{1})h^{3}
f(x − h) = f(x) − f′(x)h + \frac{1}{2}f′′(x)h^{2} − \frac{1}{6}f′′′(ξ_{2})h^{3},

where ξ_{1} is between x and x + h and ξ_{2} is between x and x − h. Hence

f(x + h) − f(x − h) = 2f′(x)h + \frac{1}{6}[f′′′(ξ_{1}) + f′′′(ξ_{2})]h^{3}

and

f′(x) = \frac{f(x + h) − f(x − h)}{2h} − \frac{1}{12}[f′′′(ξ_{1}) + f′′′(ξ_{2})]h^{2}

Let

M = max_{z∈[x−h,x+h]} f′′′(z) and m = min_{z∈[x−h,x+h]} f′′′(z).

If f′′′ is continuous on [x − h, x + h], then by the intermediate value theorem, there exists ξ ∈ [x − h, x + h] such that f(ξ) = \frac{1}{2}[f′′′(ξ_{1}) + f′′′(ξ_{2})]. Hence

f′(x) = \frac{f(x + h) − f(x − h)}{2h} − \frac{1}{6}f′′′(ξ)h^{2} = \frac{f(x + h) − f(x − h)}{2h} + O(h^{2}).  (7.4)

This is called center difference approximation and the truncation error is

|e| = \frac{h^{2}}{6} f′′′(ξ)

Similarly, we can derive an O(h^{2}) scheme from Taylor’s theorem for f′′(x)

f′′(x) = \frac{f(x + h) − 2f(x) + f(x − h)}{h^{2}} − \frac{1}{12}f^{(4)}(ξ)h^{2},  (7.5)

where ξ is between x − h and x + h.

7.1.2 Polynomial Interpolation Method

A general approach for numerical differentiation is to use polynomial interpolation. For a given function f (or a set of function values), we find a polynomial p such that p ≈ f and expect p′ ≈ f′.

Suppose that n + 1 points, x_{0}, x_{1}, …, x_{n}, and values of f, f(x_{0}), f(x_{1}), …, f(x_{n}), have been given or evaluated, we apply the Lagrange polynomial interpolation scheme to derive

p(x) = ∑_{i=0}^{n} f(x_{i})ℓ_{i}(x),  (7.6)

where

ℓ_{i}(x) = ∏_{j=0,j≠i}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} .  (7.7)

Since f(x) can be written as

f(x) = ∑_{i=0}^{n} f(x_{i})ℓ_{i}(x) + \frac{1}{(n + 1)!}f^{(n+1)}(ξ_{x})w(x),  (7.8)

where

w(x) = ∏_{j=0}^{n}(x − x_{j}),  (7.9)

we have, by taking derivative,

f′(x) = ∑_{i=0} nf(x_{i})ℓ′_{i}(x) + \frac{1}{(n + 1)!}f^{(n+1)}(ξ_{x})w′(x) + \frac{1}{(n + 1)!}w(x)\frac{d}{dx}f^{(n+1)}(ξ_{x}). (7.10)

Note that

w′(x) = ∑_{j=0}^{n} ∏_{i=0,i≠j}^{n}(x − x_{i}).  (7.11)

Hence a reasonable approximation for the first derivative of f is

f′(x) ≈ ∑_{i=0}^{n} f(x_{i})ℓ′_{i}(x).  (7.12)

When x = x_{k} for some 0 ≤ k ≤ n,

w(x_{k}) = 0 and w′(x_{k}) = ∏_{i=0,i≠k}^{n}(x_{k} − x_{i}).

Hence

f′(x_{k}) = ∑_{i=0}^{n} f(x_{i})ℓ′_{i}(x_{k}) + \frac{1}{(n + 1)!}f^{(n+1)}(ξ_{x}) ∏_{i=0,i≠k}^{n}(x_{k} − x_{i}).  (7.13)

7.1.3 Richardson Extrapolation Method

Again from the Taylor’s theorem

f(x + h) = f(x) + hf′(x) + \frac{h^{2}}{2!} f′′(x) + \frac{h^{3}}{3!} f′′′(x) + \frac{h^{4}}{4!} f^{(4)}(x) + \frac{h^{5}}{5!} f^{(5)}(x) + \frac{h^{6}}{6!} f^{(6)}(x) + ⋯
f(x − h) = f(x) − hf′(x) + \frac{h^{2}}{2!} f′′(x) − \frac{h^{3}}{3!} f′′′(x) + \frac{h^{4}}{4!} f^{(4)}(x) − \frac{h^{5}}{5!} f^{(5)}(x) + \frac{h^{6}}{6!} f^{(6)}(x) + ⋯

we have

f(x + h) − f(x − h) = 2hf′(x) + \frac{2h^{3}}{3!} f′′′(x) + \frac{2h^{5}}{5!} f^{(5)}(x) + ⋯,

and, consequently,

f′(x) = \frac{f(x + h) − f(x − h)}{2h} − [\frac{h^{2}}{3!} f′′′(x) + \frac{h^{4}}{5!} f^{(5)}(x) + ⋯] .  (7.14)

Denote

L ≡ f′(x) and ϕ(h) = \frac{f(x + h) − f(x − h)}{2h}.  (7.15)

Then equation () can be expressed as

L = ϕ(h) + a_{2}h^{2} + a_{4}h^{4} + a_{6}h^{6} + · · · .  (7.16)

This yields the O(h^{2}) central difference scheme.

Now replace h with \frac{h}{2} in the Taylor’s expansion, one has instead

f′(x) = \frac{f(x + \frac{h}{2}) − f(x − \frac{h}{2})}{h} − [\frac{1}{3!} (\frac{h}{2})^{2} f′′′(x) + \frac{1}{5!} (\frac{h}{2})^{4} f^{(5)}(x) + ⋯] .  (7.17)

This gives

L = ϕ(\frac{h}{2}) + \frac{1}{4}a_{2}h^{2} + \frac{1}{16}a_{4}h^{4} + \frac{1}{64}a_{6}h^{6} + ⋯,  (7.18)

or, equivalently,

4L = 4ϕ(\frac{h}{2}) + a_{2}h^{2} + \frac{1}{4}a_{4}h^{4} + \frac{1}{16}a_{6}h^{6} + · · · .  (7.19)

Subtract () from ()

3L = 4ϕ(\frac{h}{2}) − ϕ(h) − \frac{3}{4}a_{4}h^{4} − \frac{15}{16}a_{6}h^{6} + · · · .

This yields an O(h^{4}) approximation scheme

f′(x) = L = \frac{4}{3}ϕ(\frac{h}{2}) − \frac{1}{3}ϕ(h) − \frac{1}{4}a_{4}h^{4} − \frac{5}{16}a_{6}h^{6} + · · · .  (7.20)

We may repeat this idea by letting

ψ(h) = \frac{4}{3}ϕ(\frac{h}{2}) − \frac{1}{3}ϕ(h).  (7.21)

Then

L = ψ(h) + b_{4}h^{4} + b^{6}h^{6} + ⋯  (7.22)

and also

16L = 16ψ(\frac{h}{2}) + b_{4}h^{4} + \frac{1}{4}b_{6}h^{6} + · · · .  (7.23)

Subtract () from () to give

15L = 16ψ(\frac{h}{2}) − ψ(h) − \frac{3}{4}b_{6}h^{6} + · · · .

It leads to an O(h^{6}) approximation

f′(x) = L = \frac{16}{15}ψ(\frac{h}{2}) − \frac{1}{15}ψ(h) − \frac{1}{20}b_{6}h^{6} + · · · .  (7.24)

The process can be repeated again and again, and is called the Richardson extrapolation. For practical implementation, we denote

7.2 Numerical Integration

Calculating the definite integral of a given function f(x) over an interval [a, b],

∫_{a}^{b} f(x) dx,  (7.25)

is a classic problem. By the fundamental theorem of Calculus, this problem is solved by finding an anti-derivative F of f, that is, F′(x) = f(x), and then

∫_{a}^{b} f(x) dx = F (b) − F (a).

But finding an anti-derivative is not an easy task in general. Many elementary functions do not have simple anti-derivatives. Hence it is certainly not a good approach for numerical computation.

Numerical integration is the process of producing a numerical value for the definite integral problem (7.25). As a rule, definite integrals are computed using discretization methods which approximate the integral by finite sums corresponding to some partition of the interval of integration [a, b]. One powerful stratagem for computing the integral (7.25) numerically is to replace f by another function g that approximates f well and is easy to integrate. Hopefully, when f ≈ g, we can expect

∫_{a}^{b} f(x) dx ≈ ∫_{a}^{b} g(x) dx.

It should be no surprise that polynomials are good candidates for the function g, and indeed g can be a polynomial that interpolates f at a certain set of nodes. With this observation, we would expect that the interpolation techniques play an important role in numerical integration.

7.2.1 Elements of Numerical Integration

The basic method involved in approximating the integration (7.25) is called numerical quadrature and uses a sum of the type

∫_{a}^{b} f(x) dx ≈ ∑_{i=0}^{n} c_{i}f(x_{i}).  (7.26)

The method of quadrature in this section is based on the polynomial interpolation. We first select a set of distinct nodes {x_{0}, x_{1}, …, x_{n}} from the interval [a, b]. Then the Lagrange polynomial

P_{n}(x) = ∑_{i=0}^{n} f(x_{i})L_{i}(x) = ∑_{i=0}^{n} f(x_{i}) ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}}

is used to approximate f(x). With the error term we have

f(x) = P_{n}(x) + E_{n}(x) = ∑_{i=0}^{n} f(x_{i})L_{i}(x) + \frac{f^{(n+1)}(ζ_{x})}{(n + 1)!} ∏_{i=0}^{n}(x − x_{i}),

where ζ_{x} ∈ [a, b] and depends on x, and

∫_{a}^{b} f(x) dx = ∫_{a}^{b} P_{n}(x) dx + ∫_{a}^{b} E_{n}(x) dx
= ∑_{i=0}^{n} f(x_{i}) ∫_{a}^{b} L_{i}(x) dx + \frac{1}{(n + 1)!} ∫_{a}^{b} f^{(n+1)}(ζ_{x}) ∏_{i=0}^{n}(x − x_{i}) dx. (7.27)

The quadrature formula is, therefore,

∫_{a}^{b} f(x) dx ≈ ∫_{a}^{b} P_{n}(x) dx = ∑_{i=0}^{n} f(x_{i}) ∫_{a}^{b} L_{i}(x) dx ≡ ∑_{i=0}^{n} c_{i}f(x_{i}),  (7.28)

where

c_{i} = ∫_{a}^{b} L_{i}(x) dx = ∫_{a}^{b} ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} dx.  (7.29)

Moreover, the error in the quadrature formula is given by

E = \frac{1}{(n + 1)!} ∫_{a}^{b} f^{(n+1)}(ζ_{x}) ∏_{i=0}^{n}(x − x_{i}) dx,  (7.30)

for some ζ_{x} ∈ [a, b]. If |f^{(n+1)}(x)| ≤ M on [a, b], then

|∫_{a}^{b} f(x) dx − ∑_{i=0}^{n} c_{i}f(x_{i})| ≤ \frac{M}{(n + 1)!} ∫_{a}^{b} ∏_{i=0}^{n}(x − x_{i}) dx.  (7.31)

The choice of nodes that makes the right-hand side of this error bound as small as possible is know to be

x_{i} = \frac{a + b}{2} + \frac{b − a}{2} cos [\frac{(i + 1)π}{n + 2} ] ,  i = 0, 1, …, n.  (7.32)

Of course, a polynomial interpolation to f can be obtained in other ways, for example, polynomial in Newton’s form using divided-difference method,

P_{n}(x) = f(x_{0}) + ∑_{i=1}^{n} f[x_{0}, x_{1}, …, x_{i}] ∏_{j=0}^{i−1}(x − x_{j})

where f[x_{0}, x_{1}, …, x_{i}] are evaluated with the divided difference algorithm. Then

∫_{a}^{b} f(x) dx ≈ f(x_{0})(b − a) + ∑_{i=1}^{n} f[x_{0}, x_{1}, …, x_{i}] ∫_{a}^{b} ∏_{j=0}^{i−1}(x − x_{j}) dx.  (7.33)

The standard derivation of quadrature error formulas is based on determining the class of polynomials for which theses formulas produce exact results. The next definition is used to facilitate the discussion of this derivation.

Definition 7.1 The degree of accuracy, or precision, of a quadrature formula is the largest positive integer n such that the formula is exact for x^{k}, when k = 0, 1, …, n.

The definition implies that the degree of accuracy of a quadrature formula is n if and only if the error E = 0 for all polynomials P (x) of degree less than or equal to n, but E ≠ 0 for some polynomials of degree greater than n.

7.2.2 Newton-Cotes Formulas

A quadrature formula of the form (7.26) is called a Newton-Cotes formula if the nodes {x_{0}, x_{1}, …, x_{n}} are equally spaced. Consider a uniform partition of the closed interval [a, b] by

x_{i} = a + ih, i = 0, 1, …, n, h = \frac{b − a}{n},

where n is a positive integer and h is called the step length. By introduction a new variable t such that x = a + ht, the fundamental Lagrange polynomial becomes

L_{i}(x) = ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} = ∏_{j≠i^{j=0}}^{n} \frac{a + ht − a − jh}{a + ih − a − jh} = ∏_{j≠i^{j=0}}^{n} \frac{t − j}{i − j} ≡ φ_{i}(t).

Therefore, the integration (7.29) gives

c_{i} = ∫_{a}^{b} L_{i}(x) dx = ∫_{0}^{n} φ_{i}(t)h dt = h ∫_{0}^{n} ∏_{j≠i^{j=0}}^{n} \frac{t − j}{i − j} dt,  (7.34)

and the general Newton-Cotes formula has the form

∫_{a}^{b} f(x) dx = h ∑_{i=0}^{n} f(x_{i}) ∫_{0}^{n} ∏_{j≠i^{j=0}}^{n} \frac{t − j}{i − j} dt + \frac{1}{(n + 1)!} ∫_{a}^{b} f^{(n+1)}(ζ_{x}) ∏_{i=0}^{n}(x − x_{i}) dx. (7.35)

The simplest case is to choose n = 1, x_{0} = a, x_{1} = b, h = b − a, and use the linear Lagrange polynomial

P_{1}(x) = f(x_{0})\frac{x − x_{1}}{x_{0} − x_{1}} + f(x_{1})\frac{x − x_{0}}{x_{1} − x_{0}} = f(a)\frac{x − b}{a − b} + f(b)\frac{x − a}{b − a}.

to interpolate f(x). Then

c_{0} = h ∫_{0}^{1} \frac{t − 1}{0 − 1} dt = \frac{h}{2}, c_{1} = h ∫_{0}^{1} \frac{t − 0}{1 − 0} dt = \frac{h}{2},

and

∫_{a}^{b} P_{1}(x) dx = c_{0}f(x_{0}) + c_{1}f(x_{1}) = \frac{h}{2} [f(a) + f(b)] .

Since (x − x_{0})(x − x_{1}) = (x − a)(x − b) does not change sign on [a, b], by the Weighted Mean-Value Theorem for integrals, there exists some ξ ∈ (a, b) such that

∫_{a}^{b} f′′(ζ_{x})(x − x_{0})(x − x_{1}) dx = f′′(ξ) ∫_{a}^{b}(x − x_{0})(x − x_{1}) dx
= f′′(ξ) ∫_{a}^{b}(x − a)(x − b) dx

= f′′(ξ) [\frac{1}{3}x^{3} − \frac{1}{2}(a + b)x^{2} + abx] |_{a}^{b}
= −\frac{1}{6}f′′(ξ)(b − a)^{3} = −\frac{1}{6}f′′(ξ)h^{3}.

Consequently,

∫_{a}^{b} f(x) dx = \frac{h}{2} [f(a) + f(b)] − \frac{h^{3}}{12}f′′(ξ).

This gives the so-called Trapezoidal rule.

Trapezoidal Rule:

∫_{a}^{b} f(x) dx = \frac{1}{2}(b − a) [f(a) + f(b)] − \frac{h^{3}}{12}f′′(ξ),  (7.36)

where h = b − a and ξ ∈ (a, b).

It is evident that the error term of the Trapezoidal rule is O(h^{3}). Since the rule involves f′′, it gives the exact result when applied to any function whose second derivative is identically zero, e.g., any polynomial of degree 1 or less. Hence the degree of accuracy of Trapezoidal rule is one.

If we choose n = 2, x_{0} = a, x_{1} = \frac{1}{2}(a + b), x_{2} = b, h = (b − a)/2, and the second order Lagrange polynomial

P_{2}(x) = f(x_{0})\frac{(x − x_{1})(x − x_{2})}{(x_{0} − x_{1})(x_{0} − x_{2})} + f(x_{1})\frac{(x − x_{0})(x − x_{2})}{(x_{1} − x_{0})(x_{1} − x_{2})} + f(x_{2})\frac{(x − x_{0})(x − x_{1})}{(x_{2} − x_{0})(x_{2} − x_{1})}

to interpolate f(x), then

c_{0} = h ∫_{0}^{2} \frac{t − 1}{0 − 1} · \frac{t − 2}{0 − 2} dt = \frac{h}{2} ∫_{0}^{2}(t^{2} − 3t + 2) dt = \frac{h}{3},
c_{1} = h ∫_{0}^{2} \frac{t − 0}{1 − 0} · \frac{t − 2}{1 − 2} dt = −h ∫_{0}^{2}(t^{2} − 2t) dt = \frac{4h}{3},
c_{2} = h ∫_{0}^{2} \frac{t − 0}{2 − 0} · \frac{t − 1}{2 − 1} dt = \frac{h}{2} ∫_{0}^{2}(t^{2} − t) dt = \frac{h}{3},

and

∫_{a}^{b} P_{2}(x) dx = c_{0}f(x_{0}) + c_{1}f(x_{1}) + c_{2}f(x_{2}) = h [\frac{1}{3}f(a) + \frac{4}{3}f(\frac{a + b}{2}) + \frac{1}{3}f(b)]

gives the so-called Simpson’s rule. Deriving the formulation this way, however, the error term

\frac{1}{6} ∫_{a}^{b} f^{(3)}(ζ_{x})(x − x_{0})(x − x_{1})(x − x_{2}) dx

provides only an O(h^{4}) formulation involving f^{(3)}. A higher order error analysis can be derived by expanding f in the third Taylor’s formula about x_{1}. Then for each x ∈ [a, b], there exists ζ_{x} ∈ (a, b) such that

f(x) = f(x_{1}) + f′(x_{1})(x − x_{1}) + \frac{f′′(x_{1})}{2}(x − x_{1})^{2} + \frac{f′′′(x_{1})}{6}(x − x_{1})^{3} + \frac{f^{(4)}(ζ_{x})}{24}(x − x_{1})^{4}.

Then

∫_{a}^{b} f(x) dx = [f(x_{1})(x − x_{1}) + \frac{f′(x_{1})}{2}(x − x_{1})^{2} + \frac{f′′(x_{1})}{6}(x − x_{1})^{3} + \frac{f′′′(x_{1})}{24}(x − x_{1})^{4}] |_{a}^{b}

+ \frac{1}{24} ∫_{a}^{b} f^{(4)}(ζ_{x})(x − x_{1})^{4} dx.

Note that (b − x_{1}) = h, (a − x_{1}) = −h, and since (x − x_{1})^{4} does not change sign in [a, b], by the Weighted Mean-Value Theorem for Integral, there exists ξ_{1} ∈ (a, b) such that

∫_{a}^{b} f^{(4)}(ζ_{x})(x − x_{1})^{4} dx = f^{(4)}(ξ_{1}) ∫_{a}^{b}(x − x_{1})^{4} dx = \frac{2f^{(4)}(ξ_{1})}{5}h^{5}.

Consequently,

∫_{a}^{b} f(x) dx = 2f(x_{1})h + \frac{f′′(x_{1})}{3}h^{3} + \frac{f^{(4)}(ξ_{1})}{60}h^{5}.

Finally we replace f′′(x_{1}) by the central finite difference formulation

f′′(x_{1}) = \frac{f(x_{0}) − 2f(x_{1}) + f(x_{2})}{h^{2}} − \frac{f^{(4)}(ξ_{2})}{12}h^{2},

for some ξ_{2} ∈ (a, b), to obtain

∫_{a}^{b} f(x) dx = 2hf(x_{1}) + \frac{h}{3} (f(x_{0}) − 2f(x_{1}) + f(x_{2})) − \frac{f^{(4)}(ξ_{2})}{36}h^{5} + \frac{f^{(4)}(ξ_{1})}{60}h^{5}
= h [\frac{1}{3}f(x_{0}) + \frac{4}{3}f(x_{1}) + \frac{1}{3}f(x_{2})] + \frac{1}{90} [\frac{3}{2}f^{(4)}(ξ_{1}) − \frac{5}{2}f^{(4)}(ξ_{2})] h^{5}.

By letting f(x) = x^{4}, one can show that there exists ξ ∈ (a, b) such that

∫_{a}^{b} f(x) dx = h [\frac{1}{3}f(x_{0}) + \frac{4}{3}f(x_{1}) + \frac{1}{3}f(x_{2})] + \frac{f^{(4)}(ξ)}{90}h^{5}.

This gives the Simpson’s rule formulation.

Simpson’s Rule:

∫_{a}^{b} f(x) dx = (\frac{b − a}{2}) [\frac{1}{3}f(a) + \frac{4}{3}f(\frac{a + b}{2}) + \frac{1}{3}f(b)] + \frac{f^{(4)}(ξ)}{90}h^{5},  (7.37)

for some ξ ∈ (a, b). The Simpson’s rule is an O(h^{5}) scheme and the degree of accuracy is three.

The Trapezoidal and Simpson’s rules are examples of a class of methods known as closed Newton-Cotes formula. The (n+1)-point closed Newton-Cotes method uses nodes x_{i} = a+ih, for i = 0, 1, …, n, where h = (b − a)/n. Note that both endpoints, a = x_{0} and b = x_{n}, of the closed interval [a, b] are included as nodes. The following theorem details the Newton-Cotes formulas and the associated error analysis.

Theorem 7.1 (Closed Newton-Cotes Formulas) For a given function f(x) and closed interval [a, b], the (n + 1)-point closed Newton-Cotes method uses nodes

x_{i} = a + ih, i = 0, 1, …, n, h = \frac{b − a}{n}.

If n is even and f ∈ C^{n+2}[a, b], then

∫_{a}^{b} f(x) dx = h ∑_{i=0}^{n} α_{i}f(x_{i}) + \frac{h^{n+3}f^{(n+2)}(ξ)}{(n + 2)!} ∫_{0}^{n} t^{2}(t − 1) ⋯ (t − n) dt,  (7.38)

and if n is odd and f ∈ C^{n+1}[a, b], then

∫_{a}^{b} f(x) dx = h ∑_{i=0}^{n} α_{i}f(x_{i}) + \frac{h^{n+2}f^{(n+1)}(ξ)}{(n + 1)!} ∫_{0}^{n} t(t − 1) ⋯ (t − n) dt,  (7.39)

where ξ ∈ (a, b) and

α_{i} = ∫_{0}^{n} ∏_{j≠i^{j=0}}^{n} \frac{t − j}{i − j} dt,  i = 0, 1, …, n.  (7.40)

Consequently, the degree of accuracy is n + 1 when n is an even integer, and n when n is an odd integer.

The weights α_{i} in the Newton-Cotes formula has the property

∑_{i=0}^{n} α_{i} = n.  (7.41)

This can be shown by applying the formula to f(x) = 1 with interpolating polynomial P_{n}(x) = 1. Let s be the common denominator of α_{i}, that is,

α_{i} = \frac{σ_{i}}{s} (⇒ σ_{i} = sα_{i})

such that σ_{i} are integers, then the formulation for approximating the definite integral can be expressed as

∫_{a}^{b} f(x) dx ≈ h ∑_{i=0}^{n} α_{i}f(x_{i}) = \frac{h}{s} ∑_{i=0}^{n} σ_{i}f(x_{i}).  (7.42)

Some of the most common closed Newton-Cotes formulas with their error terms are listed in the following table.

Name  n  s σ_{i}  Error
Trapezoidal rule 1  2 1, 1  −\frac{1}{12}f^{(2)}(ξ)h^{3}
Simpson’s rule 2  3 1, 4, 1  −\frac{1}{90}f^{(4)}(ξ)h^{5}
3/8-rule  3  \frac{8}{3} 1, 3, 3, 1  −\frac{3}{80}f^{(4)}(ξ)h^{5}
Milne’s rule  4 \frac{45}{2} 7, 32, 12, 32, 7  −\frac{8}{945}f^{(6)}(ξ)h^{7}
5 \frac{288}{5} 19, 75, 50, 50, 75, 19  −\frac{275}{12096}f^{(6)}(ξ)h^{7}
Weddle’s rule  6 140 41, 216, 27, 272, 27, 216, 41 −\frac{9}{1400}f^{(8)}(ξ)h^{9}

Another class of Newton-Cotes formulas is the open Newton-Cotes formulas in which the nodes

x_{i} = x_{0} + ih, i = 0, 1, …, n, where x_{0} = a + h and h = \frac{b − a}{n + 2},

are used. This implies that x_{n} = b − h, and the endpoints, a and b, are not used. Hence we label a = x_{−1} and b = x_{n+1}. The following theorem summarizes the open Newton-Cotes formulas.

Theorem 7.2 (Open Newton-Cotes Formulas) For a given function f(x) and closed interval [a, b], the (n + 1)-point open Newton-Cotes method uses nodes

x_{i} = x_{0} + ih, i = 0, 1, …, n, where x_{0} = a + h and h = \frac{b − a}{n + 2}.

If n is even and f ∈ C^{n+2}[a, b], then

∫_{a}^{b} f(x) dx = h ∑_{i=0}^{n} α_{i}f(x_{i}) + \frac{h^{n+3}f^{(n+2)}(ξ)}{(n + 2)!} ∫_{−1}^{n+1} t^{2}(t − 1) ⋯ (t − n) dt,  (7.43)

and if n is odd and f ∈ C^{n+1}[a, b], then

∫_{a}^{b} f(x) dx = h ∑_{i=0}^{n} α_{i}f(x_{i}) + \frac{h^{n+2}f^{(n+1)}(ξ)}{(n + 1)!} ∫_{−1}^{n+1} t(t − 1) ⋯ (t − n) dt,  (7.44)

where ξ ∈ (a, b) and

α_{i} = ∫_{−1}^{n+1} ∏_{j≠i^{j=0}}^{n} \frac{t − j}{i − j} dt,  i = 0, 1, …, n.  (7.45)

Consequently, the degree of accuracy is n + 1 when n is an even integer, and n when n is an odd integer.

The simplest open Newton-Cotes formula is choosing n = 0 and only using the midpoint x_{0} = \frac{a+b}{2}. Then the coefficient and the error term can be computed easily as

α_{0} = ∫_{1}^{−1} dt = 2, and \frac{h^{3}f′′(ξ)}{2!} ∫_{−1}^{1} t^{2} dt = \frac{1}{3}f′′(ξ)h^{3}.

These gives the so-called Midpoint rule or Rectangular rule.

Midpoint Rule:

∫_{a}^{b} f(x) dx = 2hf(x_{0}) + \frac{1}{3}f′′(ξ)h^{3} = (b − a)f(\frac{a + b}{2}) + \frac{1}{3}f′′(ξ)h^{3},  (7.46)

for some ξ ∈ (a, b).

Analogous to the closed Newton-Cotes formulas, we list some of the commonly used open Newton-Cotes formulas in the following table.

Name  n s σ_{i}  Error
Midpoint rule 0 1 2  \frac{1}{3}f^{(2)}(ξ)h^{3}
1 2 3, 3  \frac{3}{4}f^{(2)}(ξ)h^{3}
2 3 8, −4, 8  \frac{14}{45}f^{(4)}(ξ)h^{5}
3 24 55, 5, 5, 55 \frac{95}{144}f^{(4)}(ξ)h^{5}

7.2.3 Composite Newton-Cotes Forumlas

It is obvious that the Newton-Cotes formulas are generally not suitable for numerical integration over large interval. Higher degree formulas would be required, and the coefficients in these formulas are difficult to obtain. Also the Newton-Cotes formulas which are based on polynomial interpolation would be inaccurate over a large interval because of the oscillatory nature of high-degree polynomials. Now we discuss a piecewise approach, called composite rule, to numerical integration over large interval that uses the low-order Newton-Cotes formulas.

A composite rule is one obtained by applying an integration formula for a single interval to each subinterval of a partitioned interval. To illustrate the procedure, we choose an even integer n and partition the interval [a, b] into n subintervals by nodes x_{0} < x_{1} < ⋯ < x_{n} = b, and apply Simpson’s rule on each consecutive pair of subintervals. With

h = \frac{b − a}{n} and x_{j} = a + jh, j = 0, 1, …, n,

we have on each interval [x_{2j−2}, x_{2j}],

∫_{x_{2j−2}}^{x_{2j}} f(x) dx = \frac{h}{3} [f(x_{2j−2}) + 4f(x_{2j−1}) + f(x_{2j})] − \frac{h^{5}}{90}f^{(4)}(ξ_{j}),

for some ξ_{j} ∈ (x_{2j−2}, x_{2j}), provided that f ∈ C^{4}[a, b]. The composite rule is obtained by summing up over the entire interval, that is,

∫_{a}^{b} f(x) dx = ∑_{j=1}^{n/2} ∫_{x_{2j−2}}^{x_{2j}} f(x) dx

= ∑_{j=1}^{n/2} [\frac{h}{3} (f(x_{2j−2}) + 4f(x_{2j−1}) + f(x_{2j})) − \frac{h^{5}}{90}f^{(4)}(ξ_{j})]

= \frac{h}{3} [f(x_{0}) + 4f(x_{1}) + f(x_{2}) + f(x_{2}) + 4f(x_{3}) + f(x_{4}) + f(x_{4}) + 4f(x_{5})

+ ⋯ + f(x_{n−2}) + 4f(x_{n−1}) + f(x_{n})] − \frac{h^{5}}{90} ∑_{j=1}^{n/2} f^{(4)}(ξ_{j})

= \frac{h}{3} [f(x_{0}) + 4f(x_{1}) + 2f(x_{2}) + 4f(x_{3}) + 2f(x_{4}) + 4f(x_{5})

+ ⋯ + 2f(x_{n−2}) + 4f(x_{n−1}) + f(x_{n})] − \frac{h^{5}}{90} ∑_{j=1}^{n/2} f^{(4)}(ξ_{j})

= \frac{h}{3} [f(x_{0}) + 4 ∑_{j=1}^{n/2} f(x_{2j−1}) + 2 ∑_{j=1}^{(n/2)−1} f(x_{2j}) + f(x_{n})] − \frac{h^{5}}{90} ∑_{j=1}^{n/2} f^{(4)}(ξ_{j}).

To estimate the error associated with approximation, since f ∈ C^{4}[a, b], we have, by the Extreme Value Theorem,

min_{x∈[a,b]} f^{(4)}(x) ≤ f^{(4)}(ξ_{j}) ≤ max_{x∈[a,b]} f^{(4)}(x),

for each ξ_{j} ∈ (x_{2j−2}, x_{2j}). Hence

\frac{n}{2} min_{x∈[a,b]} f^{(4)}(x) ≤ ∑_{j=1}^{n/2} f^{(4)}(ξ_{j}) ≤ \frac{n}{2} max_{x∈[a,b]} f^{(4)}(x),

and

min_{x∈[a,b]} f^{(4)}(x) ≤ \frac{2}{n} ∑_{j=1}^{n/2} f^{(4)}(ξ_{j}) ≤ max_{x∈[a,b]} f^{(4)}(x).

By the Intermediate Value Theorem, there exists μ ∈ (a, b) such that

f^{(4)}(μ) = \frac{2}{n} ∑_{j=1}^{n/2} f^{(4)}(ξ_{j}).

Thus, by replacing n = (b − a)/h,

∑_{j=1}^{n/2} f^{(4)}(ξ_{j}) = \frac{n}{2}f^{(4)}(μ) = \frac{b − a}{2h}f^{(4)}(μ).

Consequently, the composite Simpson’s rule is derived.

Composite Simpson’s Rule:

∫_{a}^{b} f(x) dx = \frac{h}{3} [f(a) + 4 ∑_{j=1}^{n/2} f(x_{2j−1}) + 2 ∑_{j=1}^{(n/2)−1} f(x_{2j}) + f(b)] − \frac{b − a}{180}f^{(4)}(μ)h^{4}, (7.47)

where n is an even integer, h = (b−a)/n, x_{j} = a+jh, for j = 0, 1, …, n, and some μ ∈ (a, b).

The composite Midpoint rule can be derived in a similar way, except the midpoint rule is applied on each subinterval [x_{2j−1}, x_{2j}] instead. That is,

∫_{x_{2j−2}}^{x_{2j}} f(x) dx = 2hf(x_{2j−1}) + \frac{h^{3}}{3} f′′(ξ_{j}), j = 1, 2, …, \frac{n}{2}.

Note that n must again be even. Consequently,

∫_{a}^{b} f(x) dx = 2h ∑_{j=1}^{n/2} f(x_{2j−1}) + \frac{h^{3}}{3} ∑_{j=1}^{n/2} f′′(ξ_{j}).

The error term can be written as

∑_{j=1}^{n/2} f′′(ξ_{j}) = \frac{n}{2}f′′(μ) = \frac{b − a}{2h}f′′(μ),

for some μ ∈ (a, b). Therefore, the composite Midpoint rule has the following formulation.

Composite Midpoint Rule:

∫_{a}^{b} f(x) dx = 2h ∑_{j=1}^{n/2} f(x_{2j−1}) − \frac{b − a}{6}f′′(μ)h^{2},  (7.48)

where n is an even integer, h = (b−a)/n, x_{j} = a+jh, for j = 0, 1, …, n, and some μ ∈ (a, b).

To derive the composite Trapezoidal rule, we partition the interval [a, b] by n equally spaced nodes a = x_{0} < x_{1} < ⋯ < x_{n} = b, where n can be either odd or even. We then apply the trapezoidal rule on each subinterval [x_{j−1}, x_{j}] and sum them up to obtain

∫_{a}^{b} f(x) dx = ∑_{j=1}^{n} ∫_{x_{j−1}}^{x_{j}} f(x) dx

= ∑_{j=1}^{n} [\frac{h}{2} (f(x_{j−1}) + f(x_{j})) − \frac{h^{3}}{12}f′′(ξ_{j})]

= \frac{h}{2} [f(x_{0}) + f(x_{1}) + f(x_{1}) + f(x_{2}) + ⋯ + f(x_{n−1}) + f(x_{n})] − \frac{h^{3}}{12} ∑_{j=1}^{n} f′′(ξ_{j})

= \frac{h}{2} [f(x_{0}) + 2f(x_{1}) + 2f(x_{2}) + ⋯ + 2f(x_{n−1}) + f(x_{n})] − \frac{h^{3}}{12} ∑_{j=1}^{n} f′′(ξ_{j})

= \frac{h}{2} [f(a) + ∑_{j=1}^{n−1} f(x_{j}) + f(b)] − \frac{h^{3}}{12} ∑_{j=1}^{n} f′′(ξ_{j})

= \frac{h}{2} [f(a) + ∑_{j=1}^{n−1} f(x_{j}) + f(b)] − \frac{b − a}{12}f′′(μ)h^{2},

where each ξ_{j} ∈ (x_{j−1}, x_{j}) and μ ∈ (a, b).

Composite Trapezoidal Rule

∫_{a}^{b} f(x) dx = \frac{h}{2} [f(a) + ∑_{j=1}^{n−1} f(x_{j}) + f(b)] − \frac{b − a}{12}f′′(μ)h^{2},  (7.49)

where n is an integer, h = (b − a)/n, x_{j} = a + jh, for j = 0, 1, …, n, and some μ ∈ (a, b).

7.3 Gaussian Quadrature

In the preceding section, the quadrature formulas of the type

∫_{a}^{b} f(x) dx ≈ ∑_{i=0}^{n} c_{i}f(x_{i})  (7.50)

are based on polynomial interpolation, where the choice of nodes x_{0}, x_{1}, …, x_{n} was made a priori. All Newton-Cotes formulas use values of the function at equally spaced points. Once the nodes were fixed, the coefficients were determined, e.g., by integrating the fundamental Lagrange polynomials of degree n. Furthermore, these formulas are exact for polynomials of degree ≤ n.

This approach is convenient when the formulas are combined to form the composite rules, but the restriction may decrease the accuracy of the approximation. It is nature to ask whether some choices of nodes are better than others in formula (7.50). In this section, we consider the Gaussian quadrature which chooses the points for evaluation in an optimal, rather than pre-fixed or equally-spaced, way. The theory can be formulated for quadrature rule of a slightly more general form, namely,

∫_{a}^{b} f(x)w(x) dx ≈ ∑_{i=0}^{n} c_{i}f(x_{i}),  (7.51)

where w(x) is a fixed positive weight function. The case when w(x) ≡ 1 is, naturally, of special importance. The nodes x_{0}, x_{1}, …, x_{n} in the interval [a, b] and the coefficients c_{0}, c_{1}, …, c_{n} are chosen to minimize the expected error obtained in performing the approximation (7.51), and to produce the exact result for the largest class of polynomials, that is, the choice which gives the greatest degree of precision.

The coefficients c_{0}, c_{1}, …, c_{n} in the approximation formula (7.51) are arbitrary, and the nodes x_{0}, x_{1}, …, x_{n} are restricted only by the specification that they lie in [a, b], the interval of integration. These give 2n+2 degrees of freedom. Thus we can expect that the quadrature formula of the form (7.51) can be discovered that will be exact for polynomials of degree ≤ 2n + 1.

7.3.1 Orthogonal Polynomials and Quadrature Rule

Recall that a real inner-product space E is a linear vector space in which an inner product < ·, · > and a norm ∥ · ∥ have been defined with the following properties:

1. ⟨f, g⟩ = ⟨g, f⟩

2. ⟨f, αg + βh⟩ = α⟨f, g⟩ + β⟨f, h⟩

3. ⟨f, f⟩ > 0 if f ≠ 0, and ⟨f, f⟩ = 0 if and only if f = 0

4. ∥f∥ = \sqrt{⟨f, f⟩}

where f, g, h ∈ E and α, β ∈ ℝ.

In a inner-product space, we say f is orthogonal to g, and write f ⊥ g if ⟨f, g⟩ = 0. We write f ⊥ G if f ⊥ g for all g ∈ G. We say that a finite or infinite sequence of vectors f_{1}, f_{2}, … in an inner-product space is orthogonal if ⟨f_{i}, f_{j}⟩ = 0 for all i ≠ j, and orthonormal if ⟨f_{i}, f_{j}⟩ = δ_{ij}.

Lemma 7.1 In an inner-product space, we have 1. ⟨∑_{i=1}^{n} a_{i}f_{i}, g⟩ = ∑_{i=1}^{n} a_{i}⟨f_{i}, g⟩

2. ∥f + g∥^{2} = ∥f∥^{2} + 2⟨f, g⟩ + ∥g∥^{2}

3. If f ⊥ g, then ∥f + g∥^{2} = ∥f∥^{2} + ∥g∥^{2}

4. |⟨f, g⟩| ≤ ∥f∥∥g∥

5. ∥f + g∥^{2} + ∥f − g∥^{2} = 2∥f∥^{2} + 2∥g∥^{2}

The space of continuous functions on [a, b], C[a, b], with inner-product defined as

⟨f, g⟩ = ∫_{a}^{b} f(x)g(x)w(x) dx,  (7.52)

where w(x) is a fixed continuous positive function called weight function, is an inner-product space.

Definition 7.2 {ϕ_{0}, ϕ_{1}, …, ϕ_{n}}, where ϕ_{i} ∈ C[a, b] for all i = 0, 1, …, n, is said to be an orthogonal set of functions for the interval [a, b] with respect to the weight function w if

⟨ϕ_{i}, ϕ_{j}⟩ = ∫_{a}^{b} ϕ_{i}(x)ϕ_{j}(x)w(x) dx = \begin{cases}0, & when i ≠ j, \\ α_{i} > 0, & when i = j. \end{cases}

If, in addition, α_{i} = 1 for all i = 0, 1, …, n, then the set is said to be orthonormal.

Theorem 7.3 The set of polynomials {p_{0}, p_{1}, …, p_{n}} defined inductively as follows is orthogonal:

p_{n}(x) = (x − a_{n})p_{n−1}(x) − b_{n}p_{n−2}(x) (n ≥ 2)

with p_{0}(x) = 1, p_{1}(x) = x − a_{1}, and

a_{n} = \frac{⟨xp_{n−1}, p_{n−1}⟩}{⟨p_{n−1}, p_{n−1}⟩}, b_{n} = \frac{⟨xp_{n−1}, p_{n−2}⟩}{⟨p_{n−2}, p_{n−2}⟩}.

Proof: It is clear from the formulation that each p_{n} is a monic polynomial of degree n and is therefore not zero. Hence, the denominators in a_{n} and b_{n} are not zero. Now we prove the theorem by induction on n that ⟨p_{n}, p_{i}⟩ = 0 for i = 0, 1, …, n − 1.

For n = 1, it can be shown directly from the definition that

⟨p_{1}, p_{0}⟩ = ⟨(x − a_{1})p_{0}, p_{0}⟩ = ⟨xp_{0}, p_{0}⟩ − a_{1}⟨p_{0}, p_{0}⟩ = ⟨xp_{0}, p_{0}⟩ − \frac{⟨xp_{0}, p_{0}⟩}{⟨p_{0}, p_{0}⟩}⟨p_{0}, p_{0}⟩ = 0.

Now suppose that the assertion is true for an index n − 1, where n ≥ 2. That is

⟨p_{n−1}, p_{i}⟩ = 0, i = 0, 1, …, n − 2, n ≥ 2.

Now we can verify that

⟨p_{n}, p_{n−1}⟩ = ⟨(x − a_{n})p_{n−1} − b_{n}p_{n−2}, p_{n−1}⟩
= ⟨xp_{n−1}, p_{n−1}⟩ − a_{n}⟨p_{n−1}, p_{n−1}⟩ − b_{n}⟨p_{n−2}, p_{n−1}⟩
= ⟨xp_{n−1}, p_{n−1}⟩ − \frac{⟨xp_{n−1}, p_{n−1}⟩}{⟨p_{n−1}, p_{n−1}⟩}⟨p_{n−1}, p_{n−1}⟩
= 0,
⟨p_{n}, p_{n−2}⟩ = ⟨(x − a_{n})p_{n−1} − b_{n}p_{n−2}, p_{n−2}⟩
= ⟨xp_{n−1}, p_{n−2}⟩ − a_{n}⟨p_{n−1}, p_{n−2}⟩ − b_{n}⟨p_{n−2}, p_{n−2}⟩
= ⟨xp_{n−1}, p_{n−2}⟩ − \frac{⟨xp_{n−1}, p_{n−2}⟩}{⟨p_{n−2}, p_{n−2}⟩}⟨p_{n−2}, p_{n−2}⟩
= 0,

and, for i = 1, …, n − 3,

⟨p_{n}, p_{i}⟩ = ⟨(x − a_{n})p_{n−1} − b_{n}p_{n−2}, p_{i}⟩
= ⟨xp_{n−1}, p_{i}⟩ − a_{n}⟨p_{n−1}, p_{i}⟩ − b_{n}⟨p_{n−2}, p_{i}⟩
= ⟨xp_{n−1}, p_{i}⟩
= ⟨p_{n−1}, xp_{i}⟩
= ⟨p_{n−1}, xp_{i} − a_{i+1}p_{i} − b_{i+1}p_{i−1} + a_{i+1}p_{i} + b_{i+1}p_{i−1}⟩
= ⟨p_{n−1}, p_{i+1} + a_{i+1}p_{i} + b_{i+1}p_{i−1}⟩
= ⟨p_{n−1}, p_{i+1}⟩ + a_{i+1}⟨p_{n−1}, p_{i}⟩ + b_{i+1}⟨p_{n−1}, p_{i−1}⟩
= 0.

Finally, when i = 0,

⟨p_{n}, p_{0}⟩ = ⟨(x − a_{n})p_{n−1} − b_{n}p_{n−2}, p_{0}⟩ = ⟨xp_{n−1}, p_{0}⟩ = ⟨p_{n−1}, xp_{0}⟩ = ⟨p_{n−1}, p_{1} + a_{1}p_{0}⟩ = 0.

Hence the theorem is proved.

Corollary 7.1 For any n > 0, the set of polynomials {p_{0}, p_{1}, …, p_{n}} given in the previous theorem is linearly independent and

⟨q, p_{n}⟩ = ∫_{a}^{b} q(x)p_{n}(x)w(x) dx = 0

for any polynomial q(x) with deg(q(x)) < n.

Proof: Suppose α_{0}, α_{1}, …, α_{n} are real numbers such that

P (x) = α_{0}p_{0}(x) + α_{1}p_{1}(x) + ⋯ + α_{n}p_{n}(x) = 0.

This means that the coefficients of the terms 1, x, x^{2}, … x^{n} are all zero. Since deg(p_{j}) = j and α_{n}p_{n} is the only term in P (x) which contains the x^{n} term, this implies α_{n} = 0. Hence P (x) can be reduced to

P (x) = ∑_{j=0}^{n−1} α_{j}p_{j}(x) = 0.

Similarly, we have α_{n−1} = 0, α_{n−2} = 0, …, α_{0} = 0. Therefore, {p_{0}, p_{1}, …, p_{n}} is linearly independent.

Suppose deg(q) = k < n. Write

q(x) = a_{0}p_{0}(x) + a_{1}p_{1}(x) + ⋯ + a_{k}p_{k}(x) = ∑_{j=0}^{k} a_{j}p_{j}(x),

for some real numbers a_{0}, a_{1}, …, a_{k}. Then

⟨q, p_{n}⟩ = ∫_{a}^{b} q(x)p_{n}(x)w(x) dx

= ∫_{a}^{b} (∑_{j=0}^{k} a_{j}p_{j}(x)) p_{n}(x)w(x) dx = ∑_{j=0}^{k} ∫_{a}^{b} p_{j}(x)p_{n}(x)w(x) dx = 0.

Let Π_{n} denote the set of polynomials of degree at most n, that is,

Π_{n} = {p(x) | p(x) is a polynomial and deg(p) ≤ n}.

Definition 7.3 Let w be a positive weight function and f(x) be any nonzero continuous function. We say f is w-orthogonal to Π_{n}, and denote f ⊥_{w} Π_{n}, if

⟨f, p⟩ = ∫_{a}^{b} f(x)p(x)w(x) dx = 0,

for any p(x) ∈ Π_{n}.

Theorem 7.4 Let q(x) be any nonzero polynomial of degree n + 1, and q(x) ⊥_{w} Π_{n}. If x_{0}, x_{1}, …, x_{n} are the roots of q(x) in [a, b], and

c_{i} = ∫_{a}^{b} w(x) ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} dx,

then

∫_{a}^{b} f(x)w(x) dx = ∑_{i=0}^{n} c_{i}f(x_{i}),

for any f ∈ Π_{2n+1}. That is, the quadrature rule is exact for any polynomial of degree ≤ 2n+1.

Proof: For any polynomial f ∈ Π_{2n+1}, we can write

f(x) = q(x)p(x) + r(x),

where p(x), r(x) ∈ Π_{n}. Since x_{0}, x_{1}, …, x_{n} are roots of q(x), we have

f(x_{i}) = q(x_{i})p(x_{i}) + r(x_{i}) = r(x_{i}), i = 0, 1, …, n.

By assumption, q ⊥_{w} Π_{n}, we have

⟨q, p⟩ = ∫_{a}^{b} q(x)p(x)w(x) dx = 0.

Since r(x) ∈ Π_{n}, it can be expressed exactly in the Lagrange form

r(x) = ∑_{i=0}^{n} r(x_{i}) ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} .

Hence

∫_{a}^{b} f(x)w(x) dx = ∫_{a}^{b} q(x)p(x)w(x) dx + ∫_{a}^{b} r(x)w(x) dx
= ∫_{a}^{b} r(x)w(x) dx = ∫_{a}^{b} w(x) ∑_{i=0}^{n} r(x_{i}) ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} dx

= ∑_{i=0}^{n} r(x_{i}) ∫_{a}^{b} w(x) ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} = ∑_{i=0}^{n} f(x_{i}) ∫_{a}^{b} w(x) ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}}

= ∑_{i=0}^{n} c_{i}f(x_{i}).

Next theorem shows that the roots of q(x) are simple roots and lie in the interior of the interval [a, b].

Theorem 7.5 Let w ∈ C[a, b] be a positive weight function, and f ∈ C[a, b] be a nonzero function such that f ⊥_{w} Π_{n}. Then f changes sign at least n + 1 times on (a, b).

Proof: Since 1 ∈ Π_{n} and f ⊥_{w} Π_{n}, we have

∫_{a}^{b} f(x)w(x) dx = 0,

and this shows that f changes sign at least once in [a, b].

Suppose that f only changes sign m times and m ≤ n. Let t_{0} = a < t_{1} < ⋯ < t_{m} < t_{m+1} = b such that f remains the same sign in each interval (t_{0}, t_{1}), (t_{1}, t_{2}), …, (t_{m}, t_{m+1}). Define an m-th degree polynomial as follows

p(x) = (x − t_{1})(x − t_{2}) ⋯ (x − t_{m}) = ∏_{i=1}^{m}(x − t_{i}).

Then p(x) has the same sign property as f. This implies that

∫_{a}^{b} f(x)p(x)w(x) dx ≠ 0.

However, deg(p) = m ≤ n, hence p ∈ Π_{n}. The above conclusion contradicts to the assumption that

f ⊥_{w} Π_{n} ⟹ ∫_{a}^{b} f(x)p(x)w(x) dx = 0.

Therefore f changes sign at least n + 1 times in (a, b).

These theorems show that the quadrature rule (7.51) is exact for polynomials of degree ≤ 2n + 1 if the nodes x_{0}, x_{1}, …, x_{n} are chosen to be the distinct roots of a polynomial of degree n + 1 which is w-orthogonal to Π_{n}.

7.3.2 Gaussian Quadrature Rule

If the positive weight function w(x) ≡ 1 and the interval is [−1, 1], then we can use Theorem with inner product

⟨f, g⟩ = ∫_{−1}^{1} f(x)g(x) dx

to obtain a set of orthogonal polynomials called the Lengendre polynomials. The first few Lengendre polynomials are

p_{0}(x) = 1
p_{1}(x) = x
p_{2}(x) = x^{2} − \frac{1}{3}
p_{3}(x) = x^{3} − \frac{3}{5}x
p_{4}(x) = x^{4} − \frac{6}{7}x^{2} + \frac{3}{35}
p_{5}(x) = x^{5} − \frac{10}{9}x^{3} + \frac{5}{21}x

For a given integer n, the Lengendre polynomial p_{n+1} is a monic polynomial of degree n + 1 and is orthogonal to the set of first n Lengendre polynomials {p_{0}, p_{1}, …, p_{n}}. By theorem, {p_{0}, p_{1}, …, p_{n}} is linearly independent, and, hence, is a basis for the vector space Π_{n}. This implies that p_{n+1} ⊥ Π_{n}. Thus, by theorem, the quadrature rule

∫_{−1}^{1} f(x) dx ≈ ∑_{i=0}^{n} c_{i}f(x_{i})

will be exact for f ∈ Π_{2n+1} if the nodes x_{0}, x_{1}, …, x_{n} are chosen to be the roots of p_{n+1}. That is, we proved the following theorem.

Theorem 7.6 Suppose that x_{0}, x_{1}, …, x_{n} are the roots of the (n+1)-st Lengendre polynomial p_{n+1}, and that for each i = 0, 1, …, n,

c_{i} = ∫_{−1}^{1} ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} dx.

146  Chapter 7. Numerical Differentiation and Integration

If f(x) is any polynomial of degree ≤ 2n + 1, then

∫_{−1}^{1} f(x) dx = ∑_{i=0}^{n} c_{i}f(x_{i}).

The discussion above gives the Gaussian quadrature rule. Gaussian Quadrature Rule: For a given function f(x) ∈ C[−1, 1] and integer n,

∫_{−1}^{1} f(x) dx ≈ ∑_{i=0}^{n} c_{i}f(x_{I}),  (7.53)

where x_{0}, x_{1}, …, x_{n} are the roots of the (n + 1)-st Lengendre polynomial p_{n+1}, and

c_{i} = ∫_{−1}^{1} ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} dx. i = 0, 1, …, n.

The following table give some frequently used values for x_{i} and c_{i}.

n x_{i}  c_{i}
0 x_{0} = 0  c_{0} = 2
1 x_{0} = −0.5773502692 c_{0} = c_{1} = 1

x_{1} = 0.5773502692

2 x_{0} = −0.7745966692 c_{0} = \frac{5}{9}
x_{1} = 0  c_{1} = \frac{8}{9}
x_{2} = 0.7745966692 c_{2} = \frac{5}{9}
3 x_{0} = −0.8611363116 c_{0} = 0.3478548451
x_{1} = −0.3399810436 c_{1} = 0.6521451549
x_{2} = 0.3399810436 c_{2} = 0.6521451549
x_{3} = 0.8611363116 c_{3} = 0.3478548451
4 x_{0} = −0.9061798459 c_{0} = 0.2369268851
x_{1} = −0.5384693101 c_{1} = 0.4786286705
x_{2} = 0  c_{2} = \frac{128}{225} = 0.568888889
x_{3} = 0.5384693101 c_{3} = 0.4786286705
x_{4} = 0.9061798459 c_{4} = 0.2369268851

A definite integral over an arbitrary interval [a, b] can be transformed into an integral over [−1, 1] by using a simple change of variable technique:

t = \frac{2x − a − b}{b − a}  ⟺  x = \frac{1}{2} [(b − a)t + a + b] .  (7.54)

The idea of Gaussian quadrature rule is not restricted to the use of the Lengendre polynomials. For a given weight function w(x) and interval [a, b], the nodes x_{0}, x_{1}, …, x_{n} can be chosen to be the roots of any monic polynomial q_{n+1} of degree n + 1 which is w-orthogonal to Π_{n} in the sense that

∫_{a}^{b} q_{n+1}(x)p(x)w(x) dx = 0, for all p ∈ Π_{n}.

The computation of the coefficients c_{i} in a Gaussian quadrature formula proceeds in the manner already indicated for non-Gaussian formulas once the nodes x_{i} have been determined. That is,

c_{i} = ∫_{a}^{b} w(x) ∏_{j≠i^{j=0}}^{n} \frac{x − x_{j}}{x_{i} − x_{j}} dx,  (7.55)

and

∫_{a}^{b} f(x)w(x) dx ≈ ∑_{i=0}^{n} c_{i}f(x_{i}).  (7.56)

An interesting characteristic of the Gaussian quadrature formulas can be verified immediately.

Lemma 7.2 In a Gaussian quadrature formula, the coefficients c_{i} are positive and their sum is ∫_{a}^{b} w(x) dx.

Proof: For any 0 ≤ k ≤ n, let

p(x) = \frac{q_{n+1}(x)}{x − x_{k}}.

Since the polynomial q_{n+1} is of degree n + 1, the polynomial p^{2}(x) is of degree at most 2n. The Gaussian quadrature rule will be exact, that is,

0 < ∫_{a}^{b} p^{2}(x)w(x) dx = ∑_{i=0}^{n} c_{i}p^{2}(x_{i}) = c_{k}p^{2}(x_{k}).

This shows that c_{k} > 0 for any 0 ≤ k ≤ n. Since the Gaussian quadrature is of course exact for f(x) = 1,

∫_{a}^{b} f(x)w(x) dx = ∫_{a}^{b} w(x) dx = ∑_{i=0}^{n} c_{i}f(x_{i}) = ∑_{i=0}^{n} c_{i}.

Note that in the Gaussian quadrature formulas the interval [a, b] can be infinite, e.g., [0, ∞) or (−∞, ∞). Some other important cases which lead to Gaussian integration rules are listed in the following table.

[a, b]  w(x)  Orthogonal polynomials
[−1, 1]  (1 − x^{2})^{−1/2} T_{n}(x), Chebyshev polynomials
[0, ∞) e^{−x} L_{n}(x), Laguerre polynomials
(−∞, ∞) e^{−x^{2}}  H_{n}(x), Hermite polynomials

7.3.3 Error Analysis

Theorem 7.7 Consider a Gaussian quadrature formula with error term

∫_{a}^{b} f(x)w(x) dx = ∑_{i=0}^{n} c_{i}f(x_{i}) + E.  (7.57)

For any f ∈ C^{2n+2}[a, b], we have

E = \frac{f^{(2n+2)}(ξ)}{(2n + 2)!} ∫_{a}^{b} q^{2}(x)w(x) dx,  where ξ ∈ (a, b) and q(x) = ∏_{i=0}^{n}(x − x_{i}). (7.58)

Proof: By Hermite interpolation, there is a unique polynomial p(x) of degree at most 2n + 1 such that

p(x_{i}) = f(x_{i}),  p′(x_{i}) = f′(x_{i}),  i = 0, 1, …, n.

The error for this interpolation is

f(x) − p(x) = \frac{1}{(2n + 2)!}f^{(2n+2)}(ξ_{x}) ∏_{i=0}^{n}(x − x_{i})^{2} = \frac{1}{(2n + 2)!}f^{(2n+2)}(ξ_{x})q^{2}(x),

where ξ_{x} ∈ (a, b) depends on x. It follows that

∫_{a}^{b} f(x)w(x) dx − ∫_{a}^{b} p(x)w(x) dx = \frac{1}{(2n + 2)!} ∫_{a}^{b} f^{(2n+2)}(ξ_{x})q^{2}(x)w(x) dx.

Since deg(p) ≤ 2n + 1, the Guassian quadrature formula is exact for p(x),

∫_{a}^{b} p(x)w(x) dx = ∑_{i=0}^{n} c_{i}p(x_{i}) = ∑_{i=0}^{n} c_{i}f(x_{i}).

Furthermore, f ∈ C^{2n+2}[a, b], and w(x) is a positive weight function, q^{2}(x)w(x) > 0 does not change sign in [a, b], by the weighted Mean-Value Theorem for integral, there exist ξ ∈ (a, b) such that

∫_{a}^{b} f^{(2n+2)}(ξ_{x})q^{2}(x)w(x) dx = f^{(2n+2)}(ξ) ∫_{a}^{b} q^{2}(x)w(x) dx.

Therefore,

∫_{a}^{b} f(x)w(x) dx = ∑_{i=0}^{n} c_{i}f(x_{i}) + \frac{f^{(2n+2)}(ξ)}{(2n + 2)!} ∫_{a}^{b} q^{2}(x)w(x) dx.

7.4 Adaptive Quadrature

7.5 Romberg Integration

Recall the trapezoidal rule integral formulation

∫_{a}^{b} f(x) dx ≈ T (n)
= \frac{h}{2}[f(a) + 2f(a + h) + 2f(a + 2h) + ⋯ + 2f(a + (n − 1)h) + f(a + nh)],

where h = \frac{b−a}{n}. Observe in the following example that the interval [a, b] is partitioned equally-spaced by x_{0} = a, x_{1}, x_{2}, x_{3}, and x_{4} = b. Let n = 2 and h = \frac{b−a}{2n}. If we only consider the partitions by x_{0}, x_{2}, and x_{4}, and apply the Trapezoidal rule, we have the approximation

T (n) = \frac{2h}{2}[f(a) + 2f(a + 2h) + f(a + 4h)] = h[f(a) + 2f(a + 2h) + f(a + 4h)].

On the other hand, if we apply Trapezoidal rule to the partitions by all points x_{0}, x_{1}, x_{2}, x_{3}, and x_{4}, then we have

T (2n) = \frac{h}{2}[f(a) + 2f(a + h) + 2f(a + 2h) + 2f(a + 3h) + f(a + 4h)]
= \frac{1}{2}T (n) + h[f(a + h) + f(a + 3h)].

This observation shows that if we have computed T (n) and the step size is halfed, we don’t have to compute all the function values all over again, but just at those newly added points. In general, suppose T (n) has been computed and the step size becomes h = \frac{b−a}{2n}, then

T (2n) = \frac{1}{2}T (n) + h ∑_{i=1}^{n} f(a + (2i − 1)h).  (7.59)

With this idea in mind, we can apply the trapezoidal rule recursively, i.e., we partition the interval [a, b] into 2^{n} subintervals with n = 1, 2, 3, …, and the integral formulation becomes

T (2^{n}) = \frac{1}{2}T (2^{n−1}) + \frac{b − a}{2^{n}} ∑_{i=1}^{2^{n−1}} f(a + (2i − 1)\frac{b − a}{2^{n}})  (7.60)

This is called the Romberg algorithm.

Algorithm 7.1 (Romberg Integration Algorithm) Use the Romberg algorithm to evaluate ∫_{a}^{b} f(x) dx.

R(0, 0) = \frac{1}{2}(b − a)[f(a) + f(b)] for n = 1, 2, …, M do

R(n, 0) = \frac{1}{2}R(n − 1, 0) + \frac{b−a}{2^{n}} ∑_{i=1}^{2^{n−1}} f(a + (2i − 1)\frac{b−a}{2^{n}}) end for for k = 1, 2, …, M do

for n = k, k + 1, …, M do

R(n, k) = R(n, k −1)+ \frac{1}{4^{k}−1}[R(n, k −1)−R(n−1, k −1)] end for end for

Remarks 7.1 1. The Romberg algorithm produces a table looks like the following.

2. M is modest, not too large.

3. No duplicate function evaluations in R(0, 0), R(1, 0), …, R(M, 0).

4. No function evaluation is needed im computing R(m, k) when k ≥ 1.

Theorem 7.8 If f ∈ C[a, b], then for each column k,

lim_{n→∞} R(n, k) = ∫_{a}^{b} f(x) dx.  (7.61)

Moreover, if f ∈ C^{2m}[a, b], then R(n, m) converges to ∫_{a}^{b} f(x) dx with a rate of O(h^{2m}), where h = \frac{b−a}{2^{n}}.

Numerical Solutions of Ordinary Differential Equations

In this chapter, we discuss numerical methods for solving ordinary differential equations of initial-value problems (IVP) of the form

\begin{cases}y′ = f(x, y),  x ∈ ℝ \\ y(x_{0}) = y_{0}, \end{cases}  (8.1)

where y is a function of x, f is a function of y and x, x_{0} is called the initial point, and y_{0} the initial value. The numerical values of y(x) on an interval containing x_{0} are to be determined.

8.1 Existence and Uniqueness of Solutions

Theorem 8.1 If f(x, y) is continuous in a region Ω, where

Ω = {(x, y)| |x − x_{0}| ≤ α, |y − y_{0}| ≤ β}  (8.2)

then the IVP (8.1) has a solution y(x) for |x−x_{0}| ≤ min\{α, \frac{β}{M} \}, where M = max_{(x,y)∈Ω} |f(x, y)|.

Theorem 8.2 If f and \frac{∂f}{∂x} are continuous in Ω, then the IVP (8.1) has a unique solution in the interval |x − x_{0}| ≤ min\{α, \frac{β}{M} \}.

Theorem 8.3 If f is continuous in a ≤ x ≤ b, −∞ < y < ∞ and

|f(x, y_{1}) − f(x, y_{2})| ≤ L|y_{1} − y_{2}|

for some positive constant L, (that is, f is Lipschitz continuous in y), then IVP (8.1) has a unique solution in the interval [a, b].

8.2 Euler’s Method

On of the simplest methods for solving the IVP (8.1) is the Euler’s method. Note that the slope of the tangent line at x_{0} is

y′(x_{0}) = y′_{0} = \frac{y_{1} − y_{0}}{h} ,

hence

y_{1} = y_{0} + hy′_{0} = y_{0} + hf(x_{0}, y_{0})).

Repeat this idea at (x_{1}, y_{1}) and we get

y_{2} = y_{1} + hy′_{1} = y_{1} + hf(x_{1}, y_{1}),

and so on. If x_{i} are equally spaced, and h is the increment then we have the formulation of Euler’s mehtod

x_{k+1} = x_{k} + h = x_{0} + (k + 1)h
y_{k+1} = y_{k} + hf(x_{k}, y_{k})

Euler’s method is a first-order (O(h)) method.

The Improved Euler’s method

x_{k+1} = x_{k} + h = x_{0} + (k + 1)h
y_{k+1}^{∗} = y_{k} + hf(x_{k}, y_{k})  (8.3)
y_{k+1} = y_{k} + \frac{h}{2} [f(x_{k}, y_{k}) + f(x_{k+1}, y_{k+1}^{∗})]

8.3 Runge-Kutta Methods

One of the most important methods for solving the IVP (8.1) is the Runge-Kutta method. Recall the Taylor’s Theorem

y(x + h) = y(x) + hy′(x) + \frac{h^{2}}{2} y′′ + \frac{h^{3}}{2} y′′′ + …
= y(x) + hy′(x) + \frac{h^{2}}{2} y′′ + O(h^{3})

By differentiating y(x), we have

y′ = f(x, y) ≡ f
y′′ = \frac{d}{dx}f = f_{x} + f_{y}y′ = f_{x} + f_{y}f
y′′′ = f_{xx} + f \frac{d}{dx}f_{y} + f_{y} \frac{d}{dx}f
= f_{xx} + f(\frac{f_{y}}{dy} \frac{dy}{dx} + f_{yx}) + f_{y}(f_{x} + f_{y}f)
= f_{xx} + f(f_{yx} + f_{yy}f) + f_{y}(f_{x} + f_{y}f)

plug in, we get

y(x + h) = y + hy′ + \frac{h^{2}}{2} y′′ + O(h^{3})
= y + hf + \frac{h^{2}}{2} (f_{x} + f_{y}f) + O(h^{3})
= y + \frac{h}{2}f + \frac{h}{2}f + \frac{h^{2}}{2} (f_{x} + f_{y}f) + O(h^{3})
= y + \frac{h}{2}f + \frac{h}{2}(f + hf_{x} + hf_{y}f) + O(h^{3})

Apply Taylor’s Theorem on f

f(x + h, y + hf) = f(x, y) + hf_{x} + hff_{y} + O(h^{2})
⇒ f + hf_{x} + hff_{y} ≈ f(x + h, y + hf)
⇒ y(x + h) = y + \frac{1}{2}hf + \frac{h}{2}f(x + h, y + hf) + O(h^{3})

Let

F_{1} = hf(x, y) F_{2} = hf(x + h, y + F_{1})

Then

y(x + h) = y + \frac{1}{2}(F_{1} + F_{2})

This is called the second-order Runge-Kutta method. Two function evaluations are required at each step.

Algorithm for second-order Runge-Kutta mehtod :

for k = 0, 1, 2, … do x_{k+1} = x_{k} + h = x_{0} + (k + 1)h F_{1} = hf(x_{k}, y_{k}) F_{2} = hf(x_{k+1}, y_{k} + F_{1}) y_{k+1} = y_{k} + \frac{1}{2}(F_{1} + F_{2})

end for

General form of second-order Runge-Kutta method :

y(x + h) = y + ω_{1}hf + ω_{2}hf(x + αh, y + βhf) + O(h^{3})

where ω_{1}, ω_{2}, α, β are constants to be defined, and

\begin{cases}ω_{1} + ω_{2} = 1 \\ ω_{2}α = \frac{1}{2} \\ ω_{2}β = \frac{1}{2} \end{cases} .

By letting ω_{1} = 0, ω_{2} = 1, α = β = \frac{1}{2} leads to the modified Euler’s mehtod.

Fourth-Order Runge-kutta method

y(x + h) = y + \frac{1}{6}(F_{1} + 2F_{2} + 2F_{3} + F_{4}) + O(h^{5})

where

\begin{cases}F_{1} = hf(x, y) \\ F_{2} = hf(x + \frac{1}{2}h, y + \frac{1}{2}F_{1}) \\ F_{3} = hf(x + \frac{1}{2}h, y + \frac{1}{2}F_{2}) \\ F_{4} = hf(x + h, y + F_{3}) \end{cases}

Algorithm for fourth-order Runge-Kutta mehtod :

for k = 0, 1, 2, … do x_{k+ \frac{1}{2}} = x_{k} + \frac{1}{2}h x_{k+1} = x_{k} + h = x_{0} + (k + 1)h F_{1} = hf(x_{k}, y_{k}) F_{2} = hf(x_{k+ \frac{1}{2}} , y_{k} + \frac{1}{2}F_{1}) F_{3} = hf(x_{k+ \frac{1}{2}} , y_{k} + \frac{1}{2}F_{2}) F_{4} = hf(x_{k+1}, y_{k} + F_{3}) y_{k+1} = y_{k} + \frac{1}{6}(F_{1} + 2F_{2} + 2F_{3} + F_{4}) + O(h^{5})

end for

Local truncation error in the Runge-Kutta mehtod is the error that arises in each step simply because of the truncated Taylor series. This error is inevitable. The fourth Runge-Kutta method involves a local truncation error of O(h^{5}).

The Number of function evaluations in the second order method is two and the fourth order is four.

8.4 Systems and Higher-Order Ordinary Differential Equations

Consider a system of first-order ODE’s.

{y1^{′} = f_{1}(x, y_{1}, y_{2}, …, y_{n}) ; y2^{′} = f_{2}(x, y_{1}, y_{2}, …, y_{n}) ; ⋮ ; yn^{′} = f_{n}(x, y_{1}, y_{2}, …, y_{n})

with initial conditions

{y_{1}(x_{0}) = y_{1}^{0} ; y_{2}(x_{0}) = y_{2}^{0} ; ⋮ ; y_{n}(x_{0}) = y_{n}^{0}

Define

Y = [ y_{1} ; y_{2} ; ⋮ ; y_{n} ] ,  F = [ f_{1} ; f_{2} ; ⋮ ; f_{n} ] ,  Y_{0} = [ y_{1}^{0} ; y_{2}^{0} ; ⋮ ; y_{n}^{0} ]

8.4 Systems and Higher-Order Ordinary Differential Equations  155

and transform the system into vector form

\begin{cases}Y′ & = F (x, Y) \\ Y (x_{0}) & = Y_{0} \end{cases}

The Runge-Kutta methods can be easily extended to vector form. Example 8.1

\begin{cases}y1^{′} & = y_{1} + 4y_{2} − e^{x} \\ y2^{′} & = y_{1} + y_{2} + 2e^{x} \end{cases}  with initial conditions  \begin{cases}y_{1}(0) & = 4 \\ y_{2}(0) & = \frac{5}{4} \end{cases}

Sol: Transform to

Y′ = [ \begin{matrix}y1^{′} \\ y2^{′} \end{matrix} ] = [ \begin{matrix}y_{1} + 4y_{2} − e^{x} \\ y_{1} + y_{2} + 2e^{x} \end{matrix} ] = F (x, Y)
Y_{0} = [ 4_{\frac{5}{4}} ] ,  x_{0} = 0

Higher-Order ODE’s. An higher order ODE can be converted into a system of first-order equations, hence higher-order ODEs can be solved in vector form. Example 8.2

(sin x)y′′′ + cos(xy) + sin(x^{2} + y′′) + (y′)^{3} = log x
y(2) = 7
y′(2) = 3
y′′(2) = −4

Sol: Let u_{1} = y, u_{2} = y′, and u_{3} = y′′

(sin x)y′′′ + cos(xy) + sin(x^{2} + y′′) + (y′)^{3} = log x

⇒ (sin x)u3^{′} + cos(xu_{1}) + sin(x^{2} + u_{3}) + (u_{2})^{3} = log x
⇒ u′_{3} = [log x − cos(xu_{1}) − sin(x^{2} + u_{3}) − (u_{2})^{3}] / sin x

system:

\begin{cases}u′_{1} & = u_{2} \\ u′_{2} & = u_{3} \\ u3^{′} & = [log x − cos(xu_{1}) − sin(x^{2} + u_{3}) − u_{2}^{3}]/ sin x \end{cases}  \begin{cases}u_{1}(2) & = 7 \\ u_{2}(2) & = 3 \\ u_{3}(2) & = −4 \end{cases}

let

U = [ \begin{matrix}u_{1} \\ u_{2} \\ u_{3} \end{matrix} ] , F = [ \begin{matrix}u_{2} \\ u_{3} \\ [log x − cos(xu_{1}) − sin(x^{2} + u_{3}) − u_{2}^{3}] / sin x \end{matrix} ] ,

and

U_{0} = [ \begin{matrix}7 \\ 3 \\ −4 \end{matrix} ] , x_{0} = 2.

The higher-order ODE becomes

\begin{cases}U′ & = F (x, U) \\ U(x_{0}) & = U_{0} \end{cases}

and can be solved by Runge-Kutta methods.

Boundary-Value Problems for Ordinary Differential Equations

Physical problems that are position-dependent rather than time dependent are often modeled in terms of differential equations with boundary conditions imposed at more than one point. The two-point boundary-value problems (BVP) considered in this chapter involve a secondorder differential equation together with boundary condition in the following form:

\begin{cases}y′′ = f(x, y, y′) \\ y(a) = α,  y(b) = β \end{cases}  (9.1)

The numerical procedures for finding approximate solutions to the initial-value problems can not be adapted to the solution of this problem since the specification of conditions involve two different points, x = a and x = b. New techniques are introduced in this chapter for handling problems (9.1) in which the conditions imposed are of a boundary-value rather than an initial-value type.

9.1 Mathematical Theories

Before considering numerical methods, a few mathematical theories about the two-point boundary-value problem (9.1), such as the existence and uniqueness of solution, shall be discussed in this section. Existence theorems for solutions of (9.1) tend to be rather complicated. The following results give some insight into this aspect.

Theorem 9.1 Suppose that the function f in the two-point boundary-value problem (9.1) is continuous on the set

D = {(x, y, y′)|a ≤ x ≤ b, −∞ < y < ∞, −∞ < y′ < ∞} ,

and that \frac{∂f}{∂y} and \frac{∂f}{∂y′} are also continuous on D. If

1. \frac{∂f}{∂y} (x, y, y′) > 0 for all (x, y, y′) ∈ D, and

158 Chapter 9. Boundary-Value Problems for Ordinary Differential Equations

2. a constant M exists, with |\frac{∂f}{∂y′} (x, y, y′)| ≤ M, for all (x, y, y′) ∈ D,

then the two-point boundary-value problem (9.1) has a unique solution.

When the function f(x, y, y′) has the special form

f(x, y, y′) = p(x)y′ + q(x)y + r(x),  (9.2)

the differential equation become a so-called linear problem. The previous theorem can be simplified for this case.

Corollary 9.1 If the linear two-point boundary-value problem

\begin{cases}y′′ = p(x)y′ + q(x)y + r(x) \\ y(a) = α,  y(b) = β \end{cases}  (9.3)

satisfies

1. p(x), q(x), and r(x) are continuous on [a, b], and

2. q(x) > 0 on [a, b],

then the problem has a unique solution.

Many theories and application models consider the boundary-value problem in a special form as follows.

\begin{cases}y′′ = f(x, y) \\ y(0) = 0,  y(1) = 0 \end{cases}  (9.4)

We will show that this simple form can be derived from the original problem by simple techniques such as changes of variables and linear transformation. To do this, we begin by changing the variable. Suppose that the original problem is

\begin{cases}y′′ = f(x, y) \\ y(a) = α,  y(b) = β \end{cases}  (9.5)

where y = y(x). Now let λ = b − a and define a new variable

t = \frac{x − a}{b − a} = \frac{1}{λ}(x − a).

That is, x = a + λt. Notice that t = 0 corresponds to x = a, and t = 1 corresponds to x = b. Then we define

z(t) = y(a + λt) = y(x)

with λ = b − a. This gives

z′(t) = \frac{d}{dt}z(t) = \frac{d}{dt}y(a + λt) = [\frac{d}{dx}y(x)] [\frac{d}{dt}(a + λt)] = λy′(x)

and, analogously,

z′′(t) = \frac{d}{dt}z′(t) = λ^{2}y′′(x) = λ^{2}f(x, y(x)) = λ^{2}f(a + λt, z(t)).

Likewise the boundary conditions are changed to

z(0) = y(a) = α and z(1) = y(b) = β.

With all these together, the problem (9.5) is transformed into

\begin{cases}z′′(t) = λ^{2}f(a + λt, z(t)) \\ z(0) = α,  z(1) = β \end{cases}  (9.6)

Thus, if y(x) is a solution for (9.5), then z(t) = y(a+λt) is a solution for the boundary-value problem (9.6). Conversely, if z(t) is a solution for (9.6), then y(x) = z(\frac{1}{λ}(x−a)) is a solution for (9.5).

Theorem 9.2 Consider the two-point boundary-value problems

\begin{cases}y′′(x) = f(x, y) \\ y(a) = α,  y(b) = β \end{cases}  (9.7)

and

\begin{cases}z′′(t) = g(t, z) \\ z(0) = α,  z(1) = β \end{cases}  (9.8)

in which

g(p, q) = (b − a)^{2}f(a + (b − a)p, q).

Then

1. if z(t) is a solution of (9.8), then the function defined by y(x) = z((x − a)/(b − a)) is a solution of (9.7), and

2. if y(x) is a solution of (9.7), then z(t) = y(a + (b − a)t) is a solution of (9.8).

Example 9.1 Simplify the boundary conditions of the following equation by use of changing variables.

\begin{cases}y′′ = sin(xy) + y^{2} \\ y(1) = 3,  y(4) = 7 \end{cases}

Sol: In this problem a = 1, b = 4, hence λ = 3. Now define the new variable t = \frac{1}{3}(x − 1), hence x = 1 + 3t, and let z(t) = y(x) = y(1 + 3t). Then

λ^{2}f(a + λt, z) = 9 [sin(1 + 3t)z + z^{2}] ,

160 Chapter 9. Boundary-Value Problems for Ordinary Differential Equations

and the original equation is reduced to

\begin{cases}z′′(t) = 9 sin((1 + 3t)z) + 9z^{2} \\ z(0) = 3,  z(1) = 7 \end{cases}

To reduce a two-point boundary-value problem

\begin{cases}z′′(t) = g(t, z) \\ z(0) = α,  z(1) = β \end{cases}

to a homogeneous system, we simply subtract from z a linear function that takes the values α and β at 0 and 1. That is, let

u(t) = z(t) − [α + (β − α)t]

then u′′(t) = z′′(t), and

u(0) = z(0) − α = 0 and u(1) = z(1) − β = 0

Moreover,

g(t, z) = g(t, u + α + (β − α)t) ≡ h(t, u).

The system is now transformed into

\begin{cases}u′′(t) = h(t, u) \\ u(0) = 0,  u(1) = 0 \end{cases}

Here is the theorem. Theorem 9.3 Consider the two-point boundary-value problems

\begin{cases}z′′(t) = g(t, z) \\ z(0) = α,  z(1) = β \end{cases}  (9.9)

and

\begin{cases}u′′(t) = h(t, u) \\ u(0) = 0, u(1) = 0 \end{cases} (9.10)

in which

h(p, q) = g(p, q + α + (β − α)p).

Then 1. if u(t) solves (9.10), then the function z(t) = u(t) + α + (β − α)t solves (9.9), and 2. if z(t) solves (9.9), then u(t) = z(t) − [α + (β − α)t] solves (9.10).

9.1 Mathematical Theories  161

Example 9.2 Reduce the system

\begin{cases}z′′ = [5z − 10t + 35 + sin(3z − 6t + 21)]e^{t} \\ z(0) = −7,  z(1) = −5 \end{cases}

to a homogeneous problem by linear transformation technique. Sol: Let

u(t) = z(t) − [−7 + (−5 + 7)t] = z(t) − 2t + 7.

Then z(t) = u(t) + 2t − 7, and

u′′ = z′′ = [5z − 10t + 35 + sin(3z − 6t + 21)]e^{t}
= [5(u + 2t − 7) − 10t + 35 + sin(3(u + 2t − 7) − 6t + 21)]e^{t}
= [5u + sin(3u)]e^{t}

The system is transformed to

\begin{cases}u′′(t) = [5u + sin(3u)]e^{t} \\ u(0) = u(1) = 0 \end{cases}

Example 9.3 Reduce the following two-point boundary-value problem

\begin{cases}y′′ = y^{2} + 3 − x^{2} + xy \\ y(3) = 7,  y(5) = 9 \end{cases}

to a homogeneous system. Sol: In the original system, a = 3, b = 5, α = 7, β = 9. Let λ = b − a = 2 and define a new variable

t = \frac{1}{2}(x − 3) ⟹ x = 2t + 3.

Let the function z(t) = y(x) = y(2t + 3). Then

z′′(t) = λ^{2}y′′(2t + 3) = λ^{2}f(2t + 3, u)
= 4[z^{2} + 3 − (2t + 3)^{2} + (2t + 3)z]
= 4[z^{2} + 3z + 2tz − 4t^{2} − 12t − 6]

The original problem is first transformed into

\begin{cases}z′′(t) = 4[z^{2} + 3z + 2tz − 4t^{2} − 12t − 6] \\ z(0) = 7,  z(1) = 9 \end{cases}

Next let

u(t) = z(t) − [7 + 2t], or equivalently, z(t) = u(t) + 2t + 7.

Then

u′′(t) = 4[(z + 2t + 7)^{2} + 3(u + 2t + 7) + 2t(u + 2t + 7) − 4t^{2} − 12t − 6]
= 4[u^{2} + 6tu + 17u + 4t^{2} + 36t + 64].

The original problem is transformed into the following homogeneous system

\begin{cases}u′′(t) = 4[u^{2} + 6tu + 17u + 4t^{2} + 36t + 64] \\ u(0) = u(1) = 0 \end{cases}

When a two-point boundary-value problem is expressed in the homogeneous system from, we have the following elegant results.

Theorem 9.4 The boundary-value problem

\begin{cases}y′′ = f(x, y) \\ y(0) = 0,  y(1) = 0 \end{cases}  (9.11)

has a unique solution if \frac{∂f}{∂y} is continuous, non-negative, and bounded in the strip 0 ≤ x ≤ 1

and −∞ < y < ∞.

Theorem 9.5 If f is a continuous function of (s, t) in the domain 0 ≤ s ≤ 1 and −∞ < t < ∞ such that

|f(s, t_{1}) − f(s, t_{2})| ≤ K|t_{1} − t_{2}|, (K < 8).

Then the boundary-value problem

\begin{cases}y′′ = f(x, y) \\ y(0) = 0,  y(1) = 0 \end{cases}

has a unique solution in C[0, 1].

9.2 Finite Difference Method For Linear Problems

We consider finite difference method for solving the linear two-point boundary-value problem of the form

\begin{cases}y′′ = p(x)y′ + q(x)y + r(x) \\ y(a) = α,  y(b) = β. \end{cases}  (9.12)

Methods involving finite differences for solving boundary-value problems replace each of the derivatives in the differential equation by an appropriate difference-quotient approximation. The difference-quotient is chosen to maintain a specified order of truncation error. The linear second-order problem (9.12) requires that difference-quotient approximations be used to approximate both y′ and y′′.

9.2.1 The Finite Difference Formulation

First, we select an integer n > 0 and partition the interval [a, b] into n equally-spaced subintervals by points a = x_{0} < x_{1} < … < x_{n} < x_{n} = b. Each mesh point x_{i} can be computed by

x_{i} = a + i ∗ h, i = 0, 1, …, n,

where

h = \frac{b − a}{n}

is called the mesh size. These mesh points need not be equally spaces. Indeed, if the points are not uniformly distributed, then more complicated formulations will be involved in the derivation that follows. For simplicity, they are usually equally spaced in practice.

At the interior mesh points, x_{i}, for i = 1, 2, …, n − 1, the differential equation to be approximated satisfies

y′′(x_{i}) = p(x_{i})y′(x_{i}) + q(x_{i})y(x_{i}) + r(x_{i}).  (9.13)

The central finite difference formulae

y′(x_{i}) = \frac{y(x_{i+1}) − y(x_{i−1})}{2h} − \frac{h^{2}}{6} y^{(3)}(η_{i}),  (9.14)

for some η_{i} in the interval (x_{i−1}, x_{i+1}), and

y′′(x_{i}) = \frac{y(x_{i+1}) − 2y(x_{i}) + y(x_{i−1})}{h^{2}} − \frac{h^{2}}{12}y^{(4)}(ξ_{i}),  (9.15)

for some ξ_{i} in the interval (x_{i−1}, x_{i+1}), can be derived from Taylor’s theorem by expanding y about x_{i}.

Let u_{i} denote the approximate value of y_{i} = y(x_{i}). If y ∈ C^{4}[a, b], then a finite difference method with truncation error of order O(h^{2}) can be obtained by using the approximations

y′(x_{i}) ≈ \frac{u_{i+1} − u_{i−1}}{2h} and y′′(x_{i}) ≈ \frac{u_{i+1} − 2u_{i} + u_{i−1}}{h^{2}}  (9.16)

for y′(x_{i}) and y′′(x_{i}), respectively. Furthermore, let

p_{i} = p(x_{i}), q_{i} = q(x_{i}), r_{i} = r(x_{i}).

The discrete version of equation (9.12) is then

\frac{u_{i+1} − 2u_{i} + u_{i−1}}{h^{2}} = p_{i} \frac{u_{i+1} − u_{i−1}}{2h} + q_{i}u_{i} + r_{i},  i = 1, 2, …, n − 1,  (9.17)

together with boundary conditions u_{0} = α and u_{n} = β. Equation (9.17) can be written in the form

(1 + \frac{h}{2}p_{i}) u_{i−1} − (2 + h^{2}q_{i}) u_{i} + (1 − \frac{h}{2}p_{i}) u_{i+1} = h^{2}r_{i},  i = 1, 2, …, n − 1. (9.18)

In equation (9.17), u_{1}, u_{2}, …, u_{n−1} are the unknown, and there are n − 1 linear equations to be solved. The resulting system of linear equations can be expressed in the matrix form

Au = f,  (9.19)

where

A = [ \begin{matrix}−2 − h^{2}q_{1} & 1 − \frac{h}{2}p_{1} \\ 1 + \frac{h}{2}p_{2} & −2 − h^{2}q_{2} & 1 − \frac{h}{2}p_{2} \\ & ... & ... & ... \\ & & ... & ... & ... \\ & & & 1 + \frac{h}{2}p_{n−2} & −2 − h^{2}q_{n−2} & 1 − \frac{h}{2}p_{n−2} \\ & & & & 1 + \frac{h}{2}p_{n−1} & −2 − h^{2}q_{n−1} \end{matrix} ] , (9.20)

u = [ u_{1} ; u_{2} ; ⋮ ; u_{n−2} ; u_{n−1} ] ,  and f = [ h^{2}r_{1} − (1 + \frac{h}{2}p_{1}) α ; h^{2}r_{2} ; ⋮ ; h^{2}r_{n−2} ; h^{2}r_{n−1} − (1 − \frac{h}{2}p_{n−1}) β ]  (9.21)

Since the matrix A is tridiagonal, this system can be solved by a special Gaussian elimination in O(n^{2}) flops. Note that if q_{i} > 0 for all i (Corollary 9.1), and h is small so that |\frac{h}{2}p_{i}| < 1 for all i, then A is diagonally dominant, since

|−2 − h^{2}q_{i}| > 2 = |1 + \frac{h}{2}p_{i}| + |1 − \frac{h}{2}p_{i}| ,  (9.22)

and the linear system (9.19) has a unique solution.

Theorem 9.6 Suppose that p(x), q(x), and r(x) in the linear two-point boundary-value problem (9.12) are continuous on [a, b], and q(x) > 0 on [a, b]. Then the tridiagonal linear system (9.19) has a unique solution provided that h < 2/L, where L = max_{a≤x≤b} |p(x)|.

9.2.2 Convergence Analysis

We shall analyze that when h converges to zero, the solution u_{i} of the discrete problem (9.17) converges to the solution y_{i} of the original continuous problem (9.13). Note that u_{i} depends on the mesh size h. We shall estimate |y_{i} − u_{i}| and show that it converges to zero as h → 0.

With the aid of formulas (9.14) and (9.15), we see that the exact solution y_{i} satisfies the following system of equations

\frac{y_{i+1} − 2y_{i} + y_{i−1}}{h^{2}} − \frac{h^{2}}{12}y^{(4)}(ξ_{i}) = p_{i} (\frac{y_{i+1} − y_{i−1}}{2h} − \frac{h^{2}}{6} y^{(3)}(η_{i})) + q_{i}y_{i} + r_{i},  (9.23)

for i = 1, 2, …, n − 1. On the other hand, the discrete solution u_{i} satisfies the equations (9.17). If we subtract (9.17) from (9.23) and let e_{i} = y_{i} − u_{i}, the result is

\frac{e_{i+1} − 2e_{i} + e_{i−1}}{h^{2}} = p_{i} \frac{e_{i+1} − e_{i−1}}{2h} + q_{i}e_{i} + h^{2}g_{i}, i = 1, 2, …, n − 1,

where

g_{i} = \frac{1}{12}y^{(4)}(ξ_{i}) − \frac{1}{6}p_{i}y^{(3)}(η_{i}).

After collecting terms and multiplying by h^{2}. We have equations similar to (9.18), namely,

(1 + \frac{h}{2}p_{i}) e_{i−1} − (2 + h^{2}q_{i}) e_{i} + (1 − \frac{h}{2}p_{i}) e_{i+1} = h^{4}g_{i},  i = 1, 2, …, n − 1. (9.24)

Let e = [e_{1}, e_{2}, …, e_{n−1}]^{T} and k be the index such that |e_{k}| = ∥e∥_{∞}. Then from (9.24) we have

(2 + h^{2}q_{i}) e_{k} = (1 + \frac{h}{2}p_{i}) e_{k−1} + (1 − \frac{h}{2}p_{i}) e_{k+1} − h^{4}g_{k},

and, hence

|2 + h^{2}q_{k}| |e_{k}| ≤ |1 + \frac{h}{2}p_{k}| |e_{k−1}| + |1 − \frac{h}{2}p_{k}| |e_{k+1}| + h^{4}|g_{k}|
≤ |1 + \frac{h}{2}p_{k}| ∥e∥_{∞} + |1 − \frac{h}{2}p_{k}| ∥e∥_{∞} + h^{4}∥g∥_{∞}

When q(x) > 0 for all x ∈ [a, b] and h is chosen small enough so that |\frac{h}{2}p_{i}| < 1 for all i, then the condition (9.24) holds, and the above inequality induces

h^{2}q_{k}∥e∥_{∞} ≤ h^{4}∥g∥_{∞}.

Therefore, we derive an upper bound for ∥e∥_{∞}

∥e∥_{∞} ≤ h^{2} (\frac{∥g∥_{∞}}{inf q(x)}) .  (9.25)

By the definition of g_{i}, we have

∥g∥_{∞} ≤ \frac{1}{12}∥y^{(4)}(x)∥_{∞} + \frac{1}{6}∥p(x)∥_{∞}∥y^{(3)}(x)∥_{∞}.

Hence \frac{∥g∥_{∞}}{inf q(x)} is a bound independent of h. Thus we can conclude that ∥e∥_{∞} is O(h^{2}) as h → 0.

9.2.3 Higher Order Approximations

The finite difference method established in this section ensure that the truncation error has order O(h^{2}) provided that y ∈ C^{4}[a, b]. To obtain a finite difference method with greater accuracy, we may proceed in deriving higher order approximations for y′(x_{i}) and y′′(x_{i}) using high oder of Taylor series. However this requires more mesh points around x_{i} in the approximation formulas which lead to difficulties at the boundary points. Moreover, the resulting linear system is no longer in tridiagonal form, and the computational cost for the solution is much higher.

Instead of attempting to obtain a difference method with a higher-order truncation error, it is generally more satisfactory to consider a reduction in step size h. In addition, some extrapolation techniques can be applied effectively for this matter, provided that y is sufficiently differentiable.

9.3 Finite Difference Method For Nonlinear Problems

9.4 Shooting Methods

We consider solving the following 2-point boundary-value problem with shooting mehtod:

(1) \{ \begin{matrix}y′′ = f(x, y, y′) \\ y(a) = α, y(b) = β \end{matrix}

The idea of shooting method for the BVP (1) is to solve a related initial-value problem with a guess for y′(a), say z. The corresponding IVP

(2) \{ \begin{matrix}y′′ = f(x, y, y′) \\ y(a) = α, y′(a) = z \end{matrix}

can then be solved by, for example, Runge-Kutta method. We denote this approximate solution y_{z} and hope y_{z}(b) = β. If not, we use another guess for y′(a), and try to solve an altered IVP (2) again. This process is repeated and can be done systematically. The objective is to select z, so that y_{z}(b) = β. Let

ϕ(z) = y_{z}(b) − β.

Now our objective is simply to solve the equation ϕ(z) = 0. Hence secant method can be used. Note: ϕ(z) is very expensive to compute, since each value of ϕ(z) is obtained by numerically solving an IVP. Suppose we have solutions y_{z_{1}}, y_{z_{2}} with guesses z_{1}, z_{2} and obtain ϕ(z_{1}) and ϕ(z_{2}). If these guesses can not generate satisfactory solutions, we can obtain another guess z_{3} by the secant method

z_{3} = z_{2} − ϕ(z_{2})\frac{z_{2} − z_{1}}{ϕ(z_{2}) − ϕ(z_{1})}.

In general

z_{k+1} = z_{k} − ϕ(z_{k})\frac{z_{k} − z_{k−1}}{ϕ(z_{k}) − ϕ(z_{k−1})}.

The shooting method can be quite costly in computational effort. One may solve the IVP with a large step size in the first few tries, since high precision is essentially wasted in the first few tries. Only when ϕ(z) is close to zero should a small step size be used. When the BVP has the following special form

(3) \{ \begin{matrix}y′′ = u(x) + v(x)y + w(x)y′ \\ y(a) = α, y(b) = β \end{matrix}

where u(x), v(x), w(x) are continuous in [a, b]. Then (3) can be solved in at most two tries (one step). Suppose we have solved (3) twice with initial guesses z_{1} and z_{2}, and obtain approximate solutions y_{1} and y_{2}, hence

\{ \begin{matrix}y1^{′′} = u + vy_{1} + wy1^{′} \\ y_{1}(a) = α, y1^{′}(b) = z_{1} \end{matrix}

and

\{ \begin{matrix}y2^{′′} = u + vy_{2} + wy2^{′} \\ y_{2}(a) = α, y2^{′}(b) = z_{2} \end{matrix}

Now let

y(x) = λy_{1}(x) + (1 − λ)y_{2}(x)

for some parameter λ, we can show

y′′ = u + vy + wy′

and

y(a) = λy_{1}(a) + (1 − λ)y_{2}(a) = α

We can select λ so that y(b) = β.

β = y(b) = λy_{1}(b) + (1 − λ)y_{2}(b)
= λ(y_{1}(b) − y_{2}(b)) + y_{2}(b)
⇒ λ = \frac{β − y_{2}(b)}{(y_{1}(b) − y_{2}(b))}

In practice, we can solve the following two IVPs (in parallel)

\{ \begin{matrix}y′′ = u(x) + v(x)y + w(x)y′ \\ y(a) = α, y′(a) = 0 \end{matrix}

and

\{ \begin{matrix}y′′ = u(x) + v(x)y + w(x)y′ \\ y(a) = α, y′(a) = 1 \end{matrix}

to obtain approximate solutions y_{1} and y_{2}, then compute λ and form the solution y. Moreover, the problem can be transformed by solving a system of first-order ODE. Let

\{ \begin{matrix}y_{0}(x) = x \\ y_{3}(x) = y1^{′}(x) \\ y_{4}(x) = y2^{′}(x) \end{matrix}

Then

\{ \begin{matrix}y0^{′} = 1 \\ y1^{′} = y_{3} \\ y2^{′} = y_{4} \\ y3^{′} = y1^{′′} = u + vy_{1} + wy1^{′} = u + vy_{1} + wy_{3} \\ 44^{′} = u + vy_{2} + wy_{4} \end{matrix}

with I.C.

\{ \begin{matrix}y_{0}(a) = a \\ y_{1}(a) = α \\ y_{2}(a) = α \\ y_{3}(a) = y1^{′}(a) = 0 \\ 4_{4}(a) = y2^{′}(a) = 1 \end{matrix}