Least square method and normal equation

Subscribe Send me a message home page

Least square method is an optimization method. Suppose we have a function \(f\) and there is a target value is \(b\). The objective is to find a \(x\) such as it minimizes \(|f(x) - b|\). If \(f\) is a linear system (e.g. \(f(x) = Ax\)), then \(x\) is also the solution of the normal equation:

$$ A^TAx = A^Tb $$

\(Ax\) can be considered as a linear combination of column vectors in \(A\) and from a geometric point of view, the solution x should be the prodcution of vector \(b\) to the subspace generated from \({a_i}\). Therefore, for every \(i\), we should have

$$ \langle{}b - Ax, a_i\rangle{} = 0 $$

which gives us

$$ \begin{eqnarray} \langle{}a_1, Ax\rangle{} &=& \langle{}a_1, b\rangle{}\\ \langle{}a_2, Ax\rangle{} &=& \langle{}a_2, b\rangle{}\\ &\vdots&\\ \langle{}a_n, Ax\rangle{} &=& \langle{}a_n, b\rangle{} \end{eqnarray} $$

And the normal equation is the matrix form of these equations.

----- END -----

Send me a message Subscribe to blog updates

Want some fun stuff?