Curriculum
Some general advice:
- Lengthy detailed proofs are not required. Focus on the essential.
- Understand the ideas and apply them to problems and exercises.
- Computer demonstrations and case studies are not part of the curriculum.
- Do not try to memorize difficult formulas (will be supplied at the exam).
The curriculum is found in the following material:
- J. Nocedal and S.J. Wright: Numerical Optimization, 2nd Ed., Springer, 2006
- J.L. Troutman: Variational Calculus and Optimal Control, 2nd Ed., Springer, 1996
- A subset of the notes (see below)
- Exercises with solutions
NB! Even if the list of central concepts below is fairly detailed, it is not complete!
Unconstrained Optimization
Nocedal & Wright and the supplementary notes.
Section 2.1, 2.2 (see also note 1 and 2)
- Taylor’s formula in n dimensions
- Convex sets and functions
- Convex problems
- Necessary and sufficient conditions for extrema
Section 3.1 - 3.3 (to p. 48) (see also note 1 and 3)
- What is a line search
- The quadratic model equation
- Wolfe conditions
- Steepest descent and (Quasi-) Newton
- The A-norm
- Convergence speed of the steepest descent
Section 4.1 (to p. 76) (see also note 4)
- The Trust Region-idea
- The Cauchy-point
- Understand Theorem 4.1 and (briefly) the solution of the trust region model problem
Section 5.1, 5.2 (to p. 122) (see also note 1 and 6)
- The projection theorem in Hilbert space (note 1, 2.8)
- The two main ideas behind the CG-algorithm for the quadratic problem
- Convergence (basic idea only)
- CG-algorithm for non-quadratic problems (no details)
Section 6.1, 6.2 (the idea)
- What is a quasi-Newton method
Section 9.5 (see also note 2)
- The Nelder-Mead algorithm (idea)
Section 10.1, 10.2, 10.3 (to p. 261) (see also note 8)
- Jacobian and Hessian matrices
- The linear problem
- SVD (note)
- Gauss-Newton’s method
- The Levenberg-Marquardt algorithm as a trust region method
Constrained optimization
Section 12.1, 12.2, 12.3, 12.5 (note 10 cover the curriculum)
- The KKT-conditions (Theorem 12.1)
- The LICQ condition
- Convexity and the KKT-theorem
- Second order conditions when the LICQ holds (the simplest aspects)
Section 13.1, 13.2, 13.3 and p. 378 – 382 (see also note 11)
- Reduction to standard form (slack and surplus variables)
- The duality theorem, application
- The simplex
- Basic feasible points
- The fundamental theorem for LP
- One SIMPLEX-step
- How to start the SIMPLEX algorithm.
Section 16.1, 16.4, 16.5 (to p. 476), 16.7 (see also note 13)
- Explicit solutions for equality constraints
- The null-space method
- Direct solution of the KKT-equations
- The Active Set method
- Gradient projection Method
Section 17.1, 17.2 (to p. 511) (see also note 14)
- Penalty-methods
- The eigenvalue distribution for the Hessian
- Barrier Methods
Variational Calculus
From Troutman.
Chapter 2.1, 2.2, 2.3 (only to p. 42, Application), 2.4.
- Functionals
- The Gâteaux derivative, and how to compute it
- Proposition 2.3
Chapter 3.1, 3.2, 3.3, 3.4 (not p. 69-71), 3.5
- Convex functionals
- Integral functional
- Convexity/strong convexity/strict convexity
- The Euler-Lagrange equation
- Theorem 3.5
- Theorem 3.16
- Boundary conditions (e.g. free end problems)