Projections and Projection Matrices

Projection of vector b onto vector a, with error vector e

As a simple example we project onto a line. First, calculate ee and pp.

0=e⃗∙a⃗e⃗⊥a⃗ so the dot product is zero0=(b⃗−x^a⃗)∙a⃗notice e⃗=b⃗−p⃗=b⃗−x^a⃗0=b⃗∙a⃗−a⃗x^∙a⃗distribute a⃗0=(b⃗∙a⃗)−x^(a⃗∙a⃗)notice x^ is a scalar, so the last term can be x^a⃗∙a⃗x^(a⃗∙a⃗)=(b⃗∙a⃗)x^=b⃗∙a⃗a⃗∙a⃗  ⟹  p⃗=b⃗∙a⃗a⃗∙a⃗a⃗because p⃗=x^a⃗ \begin{aligned} 0 &= \vec{e} \bullet{}\vec{a} && \vec{e} \perp \vec{a} \text{ so the dot product is zero}\\ 0 &= (\vec{b}-\hat{x}\vec{a})\bullet{}\vec{a} && \text{notice } \vec{e} = \vec{b}-\vec{p} = \vec{b}-\hat{x}\vec{a}\\ 0 &= \vec{b}\bullet \vec{a} - \vec{a} \hat{x}\bullet \vec{a} && \text{distribute } \vec{a}\\ 0 &= (\vec{b} \bullet \vec{a}) - \hat{x} (\vec{a} \bullet \vec{a}) && \text{notice } \hat{x} \text{ is a scalar, so the last term can be } \hat{x} \vec{a} \bullet \vec{a}\\ \hat{x} (\vec{a} \bullet \vec{a}) &= (\vec{b} \bullet \vec{a})\\ \hat{x} &= \dfrac{\vec{b} \bullet \vec{a}}{\vec{a} \bullet \vec{a}} \implies \vec{p} = \dfrac{\vec{b} \bullet \vec{a}}{\vec{a} \bullet \vec{a}} \vec{a} && \text{because } \vec{p} = \hat{x}\vec{a} \end{aligned}

We can summarize the projection as the matrix p⃗=Pb⃗\vec{p} = P\vec{b}. This is easy to find if we present the equation the other way around. Notice that with vectors the order is inconsequential, because a⃗∙b⃗=b⃗∙a⃗\vec{a} \bullet \vec{b} = \vec{b} \bullet \vec{a}

0=a⃗∙e⃗a⃗,e⃗ on opposite sides0=(a⃗∙b⃗)−x^(a⃗∙a⃗)x^(a⃗∙a⃗)=(a⃗∙b⃗)x^=a⃗∙b⃗a⃗∙b⃗=a⃗Tb⃗a⃗Ta⃗ \begin{aligned} 0 &= \vec{a} \bullet \vec{e} && \vec{a}, \vec{e} \text{ on opposite sides}\\ 0 &= (\vec{a} \bullet \vec{b}) - \hat{x}(\vec{a} \bullet \vec{a})\\ \hat{x}(\vec{a} \bullet \vec{a}) &= (\vec{a} \bullet \vec{b})\\ \hat{x} &= \dfrac{\vec{a} \bullet \vec{b}}{\vec{a} \bullet \vec{b}} = \dfrac{\vec{a}^T\vec{b}}{\vec{a}^T\vec{a}} \end{aligned}

With x^\hat{x} in this form, we can calculate the projection matrix, PP from p⃗=Pb⃗\vec{p} = P\vec{b}

p⃗=a⃗a⃗Tb⃗a⃗Ta⃗p⃗=a⃗a⃗Ta⃗Ta⃗b⃗  ⟹  P=a⃗a⃗Ta⃗Ta⃗because p⃗=Pb⃗ \begin{aligned} \vec{p} &= \vec{a}\dfrac{\vec{a}^T\vec{b}}{\vec{a}^T\vec{a}}\\ \vec{p} &= \dfrac{\vec{a}\vec{a}^T}{\vec{a}^T\vec{a}}\vec{b} \implies P = \dfrac{\vec{a}\vec{a}^T}{\vec{a}^T\vec{a}} && \text{because } \vec{p}= P\vec{b} \end{aligned}

Project Onto a Plane in R3\mathbb{R}^3

Note that a⃗Ta⃗\vec{a}^T\vec{a} in the denominator evaluates to a scalar. If we want to project onto a plane, the denominator becomes ATAA^TA, which evaluates to a matrix.

We can think of projecting onto a plane as projecting onto multiple vectors. To project b⃗\vec{b} onto AA, we are looking for the vector x^\hat{x}, such that p⃗=Ax^\vec{p}=A\hat{x}, where p⃗\vec{p} is the point on the plane closest to b⃗\vec{b}. The first step is to find the vector x^\hat{x}.

Like the first example, we define the error vector, e⃗\vec{e}, as the vector that goes from the plane to b⃗\vec{b}

e⃗=b⃗−Ax^\vec{e} = \vec{b} - A\hat{x}

Assume AA is a matrix made of two vectors, a1a_1 and a2a_2 in R3\mathbb{R}^3:

A=[a11a21a12a22a13a23] A = \begin{bmatrix} a_{11} & a_{21} \\ a_{12} & a_{22} \\ a_{13} & a_{23} \end{bmatrix}

Our error vector, e⃗\vec{e}, will be perpendicular to both a1a_1 and a2a_2. We want to find the closest point on AA to b⃗\vec{b}. Set e⃗\vec{e} and a⃗\vec{a} perpendicular: 0=a⃗nTe⃗0 = \vec{a}_n^T\vec{e}

0=a⃗1T(b⃗−Ax^)0=a⃗2T(b⃗−Ax^) \begin{aligned} 0 &= \vec{a}_1^T(\vec{b} - A\hat{x})\\ 0 &= \vec{a}_2^T(\vec{b} - A\hat{x}) \end{aligned}

There is a simple way to write the equation that captures all components at once.

AT(b⃗−Ax^)A^T(\vec{b}-A\hat{x})

Which can be written as

ATb⃗=ATAx^A^T\vec{b} = A^TA\hat{x}

Which can be solved for x^\hat{x}

(ATA)−1ATb⃗=x^(A^TA)^{-1}A^T\vec{b} = \hat{x}

Now that we have the x^\hat{x} vector, we can find the projection matrix. Remember that p⃗=Pb⃗\vec{p} = P\vec{b}. If we can arrange the equation above correctly, it gives us PP.

p⃗=Ax^\vec{p} = A\hat{x}

Substitute x^\hat{x}

p⃗=A(ATA)−1ATb⃗\vec{p} = A(A^TA)^{-1}A^T\vec{b}

Now the equation is in the form p⃗=Pb⃗\vec{p} = P\vec{b}, so

P=A(ATA)−1ATP = A(A^TA)^{-1}A^T