Least square method is an optimization method. Suppose we have a function \(f\) and there is a target value is \(b\). The objective is to find a \(x\) such as it minimizes \(|f(x) - b|\). If \(f\) is a linear system (e.g. \(f(x) = Ax\)), then \(x\) is also the solution of the *normal equation*:

$$
A^TAx = A^Tb
$$

\(Ax\) can be considered as a linear combination of column vectors in \(A\) and from a geometric point of view, the solution x should be the prodcution of vector \(b\) to the subspace generated from \({a_i}\). Therefore, for every \(i\), we should have

$$
\langle{}b - Ax, a_i\rangle{} = 0
$$

which gives us

$$
\begin{eqnarray}
\langle{}a_1, Ax\rangle{} &=& \langle{}a_1, b\rangle{}\\
\langle{}a_2, Ax\rangle{} &=& \langle{}a_2, b\rangle{}\\
&\vdots&\\
\langle{}a_n, Ax\rangle{} &=& \langle{}a_n, b\rangle{}
\end{eqnarray}
$$

And the normal equation is the matrix form of these equations.

----- END -----

©2019 - 2022 all rights reserved