Model fitting by least squares

Normal equations

An important concept is that when energy is minimum, the residual is orthogonal to the fitting functions. The fitting functions are the column vectors $\bold f_1$, $\bold f_2$, and $\bold f_3$. Let us verify only that the dot product $\bold r \cdot \bold f_2$ vanishes; to do this, we'll show that those two vectors are orthogonal. Energy minimum is found by

$$0 \quad = \quad {\partial\over \partial x_2} \bold r \cdot ... ...\over \partial x_2} \quad = \quad 2\; \bold r \cdot \bold f_2$$ (44)

(To compute the derivative refer to equation (25).) Equation (44) shows that the residual is orthogonal to a fitting function. The fitting functions are the column vectors in the fitting matrix.

The basic least-squares equations are often called the ``normal'' equations. The word ``normal'' means perpendicular. We can rewrite equation (41) to emphasize the perpendicularity. Bring both terms to the left, and recall the definition of the residual $\bold r$ from equation (25):

$$\bold F\T ( \bold F \bold x - {\bf d}) = \bold 0$$ (45)

$$\bold F\T \bold r = \bold 0$$ (46)

Equation (46) says that the residual vector $\bold r$ is perpendicular to each row in the $\bold F\T$ matrix. These rows are the fitting functions. Therefore, the residual, after it has been minimized, is perpendicular to all the fitting functions.

Model fitting by least squares