Exercise 11

We observed that if the columns of $A$ are linearly independent, then there is a unique least-squares approximate solution to the equation $A\xvec=\bvec$ because the equation $A\xhat=\bhat$ has a unique solution. We also said that $\xhat$ is the unique solution to the normal equation $A^TA\xhat = A^T\bvec$ without explaining why this equation has a unique solution. This exercise offers an explanation.

Assuming that the columns of $A$ are linearly independent, we would like to conclude that the equation $A^TA\xhat=A^T\bvec$ has a unique solution.

1. Suppose that $\xvec$ is a vector for which $A^TA\xvec = \zerovec$. Explain why the following argument is valid and allows us to conclude that $A\xvec = \zerovec$. $$\begin{aligned} A^TA\xvec & = \zerovec \\ \xvec\cdot A^TA\xvec & = \xvec\cdot\zerovec = 0 \\ (A\xvec)\cdot(A\xvec) & = 0 \\ \len{A\xvec}^2 & = 0. \\ \end{aligned}$$ In other words, if $A^TA\xvec = \zerovec$, we know that $A\xvec = \zerovec$.

2. If the columns of $A$ are linearly independent and $A\xvec = \zerovec$, what do we know about the vector $\xvec$?

3. Explain why $A^TA\xvec = \zerovec$ can only happen when $\xvec = \zerovec$.

4. Assuming that the columns of $A$ are linearly independent, explain why $A^TA\xhat=A^T\bvec$ has a unique solution.