Least square method is an optimization method. Suppose we have a function \(f\) and there is a target value is \(b\). The objective is to find a \(x\) such as it minimizes \(|f(x) - b|\). If \(f\) is a linear system (e.g. \(f(x) = Ax\)), then \(x\) is also the solution of the normal equation:
$$
A^TAx = A^Tb
$$
\(Ax\) can be considered as a linear combination of column vectors in \(A\) and from a geometric point of view, the solution x should be the prodcution of vector \(b\) to the subspace generated from \({a_i}\). Therefore, for every \(i\), we should have
$$
\langle{}b - Ax, a_i\rangle{} = 0
$$
which gives us
$$
\begin{eqnarray}
\langle{}a_1, Ax\rangle{} &=& \langle{}a_1, b\rangle{}\\
\langle{}a_2, Ax\rangle{} &=& \langle{}a_2, b\rangle{}\\
&\vdots&\\
\langle{}a_n, Ax\rangle{} &=& \langle{}a_n, b\rangle{}
\end{eqnarray}
$$
And the normal equation is the matrix form of these equations.
----- END -----
Welcome to join reddit self-learning community.
©2019 - 2021 all rights reserved
Want some fun stuff?
