Exercise 12

This problem is about the meaning of the coefficient of determination $R^2$ and its connection to variance, a topic that appears in the next section. Throughout this problem, we consider the linear system $A\xvec=\bvec$ and the approximate least-squares solution $\xhat$, where $A\xhat=\bhat$. We suppose that $A$ is an $m\times n$ matrix, and we will denote the $m$-dimensional vector $\onevec = \fourvec11{\vdots}1$.

1. Explain why $\bbar$, the mean of the components of $\bvec$, can be found as the dot product $$\bbar = \frac 1m \bvec\cdot\onevec.$$

2. In the examples we have seen in this section, explain why $\onevec$ is in $\col(A)$.

3. If we write $\bvec = \bhat + \bvec^\perp$, explain why $$\bvec^\perp\cdot\onevec = 0$$ and hence why the mean of the components of $\bvec^\perp$ is zero.

4. The variance of an $m$-dimensional vector $\vvec$ is $\var(\vvec) = \frac1m \len{\widetilde{\vvec}}^2$, where $\widetilde{\vvec}$ is the vector obtained by demeaning $\vvec$.

   Explain why $$\var(\bvec) = \var(\bhat) + \var(\bvec^\perp).$$

5. Explain why $$\frac{\len{\bvec - A\xhat}^2}{\len{\widetilde{\bvec}}^2} = \frac{\var(\bvec^\perp)}{\var(\bvec)}$$ and hence $$R^2 = \frac{\var(\bhat)}{\var(\bvec)} = \frac{\var(A\xhat)}{\var(\bvec)}.$$

   These expressions indicate why it is sometimes said that $R^2$ measures the “fraction of variance explained” by the function we are using to fit the data. As seen in the previous exercise, there may be other features that are not recorded in the dataset that influence the quantity we wish to predict.

6. Explain why $0\leq R^2 \leq 1$.