Lagrange multipliers

Suppose a function $f(\overrightarrow{x}):{\mathbb{R}}^n\to \mathbb{R}$ is to be optimized under constraints:

This can be solved by introducing new variables (termed Lagrange variables or slack variables) ${\lambda }_i$, one for each constraint and optimizing the function:

wrt $x$'s (now unconstrained) and $\lambda$'s.

Why does this work? The intuition is that the constraints carve out a $n-m$ subdimensional manifold of ${\mathbb{R}}^n$, corresponding to allowed configurations of $\overrightarrow{x}$. At each point of the submanifold, the normal space (the directions off the surface) is spanned by the gradients $\mathrm{\nabla }{g}_1,\dots ,\mathrm{\nabla }{g}_m$​, and the tangent space (the directions along the surface) is the orthogonal complement.

Furthermore, at the optimum on this manifold, the gradient ${\mathrm{\nabla }}_{x}f$ is necessarily normal to the manifold, meaning it is a linear combination of the gradients ${\mathrm{\nabla }}_x{g}_i$ of all the constraints. In other words:

These are exactly the equations (upto a sign change) produced by unconstrained optimization of $\tilde{f}$ above wrt $\overrightarrow{x}$. ${\mathrm{\nabla }}_{{\lambda }_i}\tilde{f}$ on the other hand yields back the original constraints. So solving this new unconstrained optimization problem is equivalent to the constrained one we set out to solve.