# Least square method and normal equation

Least square method is an optimization method. Suppose we have a function $$f$$ and there is a target value is $$b$$. The objective is to find a $$x$$ such as it minimizes $$|f(x) - b|$$. If $$f$$ is a linear system (e.g. $$f(x) = Ax$$), then $$x$$ is also the solution of the normal equation:

$$A^TAx = A^Tb$$

$$Ax$$ can be considered as a linear combination of column vectors in $$A$$ and from a geometric point of view, the solution x should be the prodcution of vector $$b$$ to the subspace generated from $${a_i}$$. Therefore, for every $$i$$, we should have

$$\langle{}b - Ax, a_i\rangle{} = 0$$

which gives us

$$\begin{eqnarray} \langle{}a_1, Ax\rangle{} &=& \langle{}a_1, b\rangle{}\\ \langle{}a_2, Ax\rangle{} &=& \langle{}a_2, b\rangle{}\\ &\vdots&\\ \langle{}a_n, Ax\rangle{} &=& \langle{}a_n, b\rangle{} \end{eqnarray}$$

And the normal equation is the matrix form of these equations.

----- END -----

Want some fun stuff?