Curriculum

Some general advice:

  • Lengthy detailed proofs are not required. Focus on the essential.
  • Understand the ideas and apply them to problems and exercises.
  • Computer demonstrations and case studies are not part of the curriculum.
  • Do not try to memorize difficult formulas (will be supplied at the exam).

The curriculum is found in the following material:

  1. J. Nocedal and S.J. Wright: Numerical Optimization, 2nd Ed., Springer, 2006
  2. J.L. Troutman: Variational Calculus and Optimal Control, 2nd Ed., Springer, 1996
  3. A subset of the notes (see below)
  4. Exercises with solutions

NB! Even if the list of central concepts below is fairly detailed, it is not complete!

Unconstrained Optimization

Nocedal & Wright and the supplementary notes.

Section 2.1, 2.2 (see also note 1 and 2)
  • Taylor’s formula in n dimensions
  • Convex sets and functions
  • Convex problems
  • Necessary and sufficient conditions for extrema
Section 3.1 - 3.3 (to p. 48) (see also note 1 and 3)
  • What is a line search
  • The quadratic model equation
  • Wolfe conditions
  • Steepest descent and (Quasi-) Newton
  • The A-norm
  • Convergence speed of the steepest descent
Section 4.1 (to p. 76) (see also note 4)
  • The Trust Region-idea
  • The Cauchy-point
  • Understand Theorem 4.1 and (briefly) the solution of the trust region model problem
Section 5.1, 5.2 (to p. 122) (see also note 1 and 6)
  • The projection theorem in Hilbert space (note 1, 2.8)
  • The two main ideas behind the CG-algorithm for the quadratic problem
  • Convergence (basic idea only)
  • CG-algorithm for non-quadratic problems (no details)
Section 6.1, 6.2 (the idea)
  • What is a quasi-Newton method
Section 9.5 (see also note 2)
  • The Nelder-Mead algorithm (idea)
Section 10.1, 10.2, 10.3 (to p. 261) (see also note 8)
  • Jacobian and Hessian matrices
  • The linear problem
  • SVD (note)
  • Gauss-Newton’s method
  • The Levenberg-Marquardt algorithm as a trust region method

Constrained optimization

Section 12.1, 12.2, 12.3, 12.5 (note 10 cover the curriculum)
  • The KKT-conditions (Theorem 12.1)
  • The LICQ condition
  • Convexity and the KKT-theorem
  • Second order conditions when the LICQ holds (the simplest aspects)
Section 13.1, 13.2, 13.3 and p. 378 – 382 (see also note 11)
  • Reduction to standard form (slack and surplus variables)
  • The duality theorem, application
  • The simplex
  • Basic feasible points
  • The fundamental theorem for LP
  • One SIMPLEX-step
  • How to start the SIMPLEX algorithm.
Section 16.1, 16.4, 16.5 (to p. 476), 16.7 (see also note 13)
  • Explicit solutions for equality constraints
  • The null-space method
  • Direct solution of the KKT-equations
  • The Active Set method
  • Gradient projection Method
Section 17.1, 17.2 (to p. 511) (see also note 14)
  • Penalty-methods
  • The eigenvalue distribution for the Hessian
  • Barrier Methods

Variational Calculus

From Troutman.

Chapter 2.1, 2.2, 2.3 (only to p. 42, Application), 2.4.
  • Functionals
  • The Gâteaux derivative, and how to compute it
  • Proposition 2.3
Chapter 3.1, 3.2, 3.3, 3.4 (not p. 69-71), 3.5
  • Convex functionals
  • Integral functional
  • Convexity/strong convexity/strict convexity
  • The Euler-Lagrange equation
  • Theorem 3.5
  • Theorem 3.16
  • Boundary conditions (e.g. free end problems)
2013-04-29, Anne Kværnø