Model fitting by least squares

Damped solution

Equation (2.1) is the solution to an optimization problem that arises in many applications. Now that we know the solution, let us formally define the problem. First, we will solve a simpler problem with real values: we will choose to minimize the quadratic function of $x$:

$$Q(x) \eq (f x-y)^2 + \epsilon^2 x^2$$ (2)

The second term is called a ``damping factor" because it prevents $x$ from going to $\pm \infty$ when $f\rightarrow 0$. Set $dQ/dx=0$, which gives

$$0 \eq f(f x-y) + \epsilon^2 x$$ (3)

This yields the earlier answer $x=fy/(f^2+\epsilon^2)$.

With Fourier transforms, the signal $X$ is a complex number at each frequency $\omega$. So we generalize equation (2.2) to

$$Q(\bar X, X) \eq (\overline{FX-Y}) (FX-Y) + \epsilon^2 \bar X X \eq (\bar X \bar F - \bar Y) (FX-Y) + \epsilon^2 \bar X X$$ (4)

To minimize $Q$ we could use a real-values approach, where we express $X=u+iv$ in terms of two real values $u$ and $v$ and then set $\partial Q/\partial u=0$ and $\partial Q/\partial v=0$. The approach we will take, however, is to use complex values, where we set $\partial Q/\partial X=0$ and $\partial Q/\partial \bar X=0$. Let us examine $\partial Q/\partial \bar X$:

$${\partial Q(\bar X, X)\over \partial \bar X} \eq \bar F (FX-Y) + \epsilon^2 X \eq 0$$ (5)

The derivative $\partial Q/\partial X$ is the complex conjugate of $\partial Q/\partial \bar X$. So if either is zero, the other is too. Thus we do not need to specify both $\partial Q/\partial X=0$ and $\partial Q/\partial \bar X=0$. I usually set $\partial Q/\partial \bar X$ equal to zero. Solving equation (2.5) for $X$ gives equation (2.1).

Equation (2.1) solves $Y=XF$ for $X$, giving the solution for what is called ``the deconvolution problem with a known wavelet $F$." Analogously we can use $Y=XF$ when the filter $F$ is unknown, but the input $X$ and output $Y$ are given. Simply interchange $X$ and $F$ in the derivation and result.

Model fitting by least squares