Lecture plan

I will update the lecture plan as we go along

Date Topics Reading
Introduction, notion and existence of solutions
Week 2 Overview
Formulation of a minimization problem
Definitions of minima
Lower semi-continuity and coercivity
Existence theorems for minima
N&W, Chapters 1, 2.1
A note by M. Grasmair Minimisers of Optimisation Problems
Theory and methods for unconstrained optimization
Taylor's theorem in higher dimensions
1st & 2nd order necessary & sufficient conditions
N&W, Chapter 2.1
Week 3 Convex sets and functions N&W, Chapter 2.1
Note by M. Grasmair Basic Properties of Convex Functions
Line search methods
Armijo (backtracking) line search
The Wolfe conditions.
N&W, Chapter 3.1
Week 4 Convergence of backtracking methods
Zoutendijk's result
Algorithmic approach to the Wolfe conditions
Convergence rate of line search methods
N&W, Chapters 3.1-3.3 , note03.pdf, 2D example in Python: steep_desc.py
Week 5 Linear and non-linear conjugate gradient methods
(Fletcher-Reeves, Polack-Ribière, Hestenes-Stiefel)
N&W, Chapters 5.1-5.2. 2D example in Python: cg2d.py
Week 6 Quasi-Newton methods (DFP, BFGS, and SR1, L-BFGS). Newton-CG method. Derivative-free minimization (Nedler-Mead). N&W, 6.1-6.2, 7.1-7.2, 9.5 inexactnewton.pdf, rosenbrock_nedler.m
Week 7 Linear & non-linear least-squares problems. Gauss-Newton method N&W 10.1-10.3
Idea of trust-region methods. Cauchy point and dogleg method. N&W, 4.1-4.2, rosenbrock_extrust.m, rosenbrock_dogleg.m
Theory and methods for constrained optimization
Week 8 Introduction to optimality conditions for constrained problems:
optimization over convex sets and hyperplanes
Applications to trust-region subproblem and quasi-Newton formulas
Radial and normal cones
This note
Week 9 Tangent cones
Fundamental necessary condition
Constraint qualifications
Linearized feasible directions
N&W 12
Week 10 Separation of point and convex set (Hahn-Banach theorem)
Farkas' lemma
Second order conditions
N&W 12
Week 11 Quadratic programming
Active set methods
N&W 16.4-5
Penalty and augmented Lagrangian methods (overview) N&W 17
Week 12 Sequential quadratic programming
Merit functions
Line search-SQP method
N&W 18.1-18.4
Non-smooth merit function for Newton's methods
Sequential quadratic programming with inequality constraints
Maratos effect
N&W 18.1-18.4
Week 13 Easter break
Week 15 Linear programming (LP)
Standard form of LP
Optimality conditions and duality
N&W 13.1, this note
Interior point methods for LP
Central path
Primal-dual path-following methods
N&W 14.1
Week 16
Week 17 Repetition and questions The last lecture is held on Monday, April 23th.

N&W always refers to the textbook J. Nocedal and S. Wright: Numerical Optimization, 2nd Edition. Springer, 2006.

2018-03-08, Anton Evgrafov