Scientific Computing, Spring 2002 (http://www.math.nyu.edu/faculty/goodman/teaching/SciComp2002/)

Assignment 5.

Given March 23, due April 6, corrected April 1.

Objective: To practice optimization methods and robust numerical software.

This assignment asks you to write good, robust numerical software for finding the minimum of a function of 2 variables. In the real world, you would minimize a function of n variables. This assignment keeps some of the flavor of the more general problem but leaves out some. The code has several components. In your writeup, hand in and explain tests of each component separately. Among these components are the line search program and the linear algebra program. In particular, the routines should have built in safeguards and you should hand in examples of the safeguards being activated and tested.

(1) Line Search: Write a program that attempts to minimize a function of a single variable, ϕ(t). The algorithm should have two phases. The first phase assumes that ϕ′(0) < 0 and searches for a positive t that is within a factor of 2 of a local minimum. Clearly this will be the case if

ϕ(t) < ϕ(0) and ϕ(t) > ϕ(2t) .  (1)

A binary search algorithm for locating a suitable t would start with an initial guess, say t = 1. Evaluate ϕ(t). If ϕ(t) > ϕ(0), then replace t bt t/2. If ϕ(t) < ϕ(0), evaluate ϕ(2t). If ϕ(2t) < ϕ(t), replace t with 2t. Repeat until t satisfies (1) or until an iteration bound is exceeded. For example, if ϕ(t) = −t then a binary search strategy will not find a local minimum.

The second phase is a Newton search for the minimum. This should have a safeguard against negative ϕ′′ and a safeguard against wild iterates (e.g., ones that violate the criteria of phase 1). Try your algorithm on the function ϕ(t) = e^{t} − 1/(1 + t^{2}) with various initial guesses. Large positive or negative initial guesses present challenges. Observe the action of phase one and the eventual quadratic convergence under phase two. Observe the iterates of unsafeguarded Newton’s method from a large positive initial guess.

(2) Write a program that performs unsafeguarded Newton’s method for a function of two variables. To do this, you will need a subroutine that solves the linear system Ax = b when A is a 2 × 2 symmetric matrix. When x is close to an optimum, A should be positive definite. Therefore, the Choleski factorization of A may be appropriate. Try your program on the function

V (x, y) = ϕ (x^{2} + a(y − b sin(cx))^{2})  (2)

with a = 2, b = 1, and c = 1. Try initial guesses close and far from the solution (x = y = 0). With a good initial guess you should see quadratic convergence to the minimizer. With a poor initial guess you should see wild behavior.

(3) Add two safeguards to the program from step (2). Do a phase one line search to make sure the step is not crazy and check that the Newton step is a descent direction. Use this to minimize the objective function (1) with a = 30, b = 1, and c = 6, starting from initial guess (x_{1}, x_{2}) = (3, 1). Make a contour plot of the objective function showing the minimization iterates.