Eigenvalues and Eigenvectors

All nn by nn matrices have nn eigenvalues.

Assume:

Ax⃗=λx⃗Ax⃗−λx⃗=0(A−λI)x⃗=0 \begin{aligned} A\vec{x} &= \lambda\vec{x}\\ A\vec{x} - \lambda\vec{x} &= 0\\ (A - \lambda I)\vec{x} &= 0 \end{aligned}

Eigenvectors are defined as non-zero, so we are not interested in the case when x⃗=0\vec{x} = 0.

x⃗\vec{x} is some non-zero vector in the nullspace of (A−λI)(A - \lambda{}I). If (A−λI)(A - \lambda{}I) has vectors other than 00 in the nullspace, it must be singular.

We can find the singular matrices with

det(A−λI)=0det(A-\lambda I) = 0

Example of Finding the Eigenvectors

Start by finding the eigenvalues for A=[3113]A=\begin{bmatrix}3 & 1 \\1 & 3\end{bmatrix}

AA is set up so that the eigenvalues will be real numbers.

0=det(A−λI)0=∣3−λ113−λ∣0=(3−λ)2−10=9−6λ−λ2−10=λ2−6λ−80=(λ−4)(λ−2) \begin{aligned} 0 &= det(A-\lambda{}I)\\ 0 &= \begin{vmatrix} 3-\lambda & 1 \\ 1 & 3 - \lambda \end{vmatrix}\\ 0 &=(3-\lambda{})^2-1\\ 0 &= 9 - 6\lambda{} - \lambda{}^2 - 1\\ 0 &= \lambda{}^2 - 6\lambda -8\\ 0 &= (\lambda - 4)(\lambda - 2) \end{aligned}

So λ1=4\lambda{}_1=4 and λ2=2\lambda{}_2=2. Now we can plug both λ\lambda into (A−λI)(A-\lambda{}I)

[3113]−[2002]=[1111][3113]−[4004]=[−111−1] \begin{aligned} \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} - \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix} &= \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} \\ \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} - \begin{bmatrix} 4 & 0 \\ 0 & 4 \end{bmatrix} &= \begin{bmatrix} -1 & 1 \\ 1 & -1 \end{bmatrix} \\ \end{aligned}

And solve for (A−λI)x⃗=0(A-\lambda{}I)\vec{x}=0

[1111]x⃗1=0  ⟹  x⃗1=[1−1][−111−1]x⃗2=0  ⟹  x⃗2=[11] \begin{aligned} \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\vec{x}_1=0 &\implies \vec{x}_1= \begin{bmatrix}1\\-1\end{bmatrix}\\ \begin{bmatrix} -1 & 1 \\ 1 & -1 \end{bmatrix}\vec{x}_2=0 &\implies \vec{x}_2= \begin{bmatrix}1\\1\end{bmatrix} \end{aligned}

Example of a Degenerate Matrix

Notice that the eigenvectors in the first example are independent. Not all matrices have independent eigenvectors.

A=[3103] A = \begin{bmatrix}3 & 1 \\ 0 & 3\end{bmatrix}

We can read the eigenvalues directly off a triangular matrix. To see how, try finding det(A−λI)x⃗=0det(A-\lambda{}I)\vec{x}=0:

∣3−λ103−λ∣=0 \begin{vmatrix} 3-\lambda & 1 \\ 0 & 3-\lambda \end{vmatrix}=0

Remember, the determinant of a triangular matrix is the product down the diagonal.

(3−λ)(3−λ)=0  ⟹  λ1=3,λ2=3(3-\lambda )(3-\lambda )=0 \implies \lambda{}_1 = 3, \lambda{}_2 = 3

Repeated eigenvalues are not a problem. The problem comes when we try to solve (A−λI)x⃗=0(A-\lambda{}I)\vec{x}=0

[0100]x⃗=0 \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix} \vec{x} = 0

There is only one independent solution: x⃗=[10]\vec{x}=\begin{bmatrix}1\\0\end{bmatrix}. We cannot diagonalize AA.

Diagonalization S−1AS=ΛS^{-1}AS= \Lambda{}

Assume we take our nn linearly independent eigenvectors of AA

S=[x⃗1⋯x⃗n] S = \begin{bmatrix} & & \\ \vec{x}_1 & \cdots & \vec{x}_n \\ & & \end{bmatrix}

What happens when we take ASAS?

AS=A[x⃗1⋯x⃗n]=[λ1x⃗1⋯λnx⃗n] AS = A\begin{bmatrix} & & \\ \vec{x}_1 & \cdots & \vec{x}_n \\ & & \end{bmatrix} = \begin{bmatrix} & & \\ \lambda{}_1\vec{x}_1 & \cdots & \lambda{}_n\vec{x}_n \\ & & \end{bmatrix}

Remember that ASAS is a linear combination of the columns of AA. Because x⃗1\vec{x}_1 is an eigenvector, the first column of ASAS is going to be λ1x⃗1\lambda{}_1\vec{x}_1, and the subsequent columns follow the same pattern.

Now we want to factor out λ\lambda{} from ASAS.

AS=[λ1x⃗1⋯λnx⃗n]=[x⃗1⋯x⃗n][λ1⋱λn] AS = \begin{bmatrix} & & \\ \lambda{}_1\vec{x}_1 & \cdots & \lambda{}_n\vec{x}_n \\ & & \end{bmatrix} = \begin{bmatrix} & & \\ \vec{x}_1 && \cdots && \vec{x}_n \\ & & \end{bmatrix} \begin{bmatrix} \lambda{}_1 & & \\ & \ddots & \\ & & \lambda{}_n \end{bmatrix}

We will call this last diagonal matrix Λ\Lambda, and we can now say AS=SΛAS=S\Lambda, which gives us the following two equations:

S−1AS=ΛS^{-1}AS = \Lambda
SΛS−1=AS\Lambda{}S^{-1}=A

Remember: We can only invert SS if we have nn independent eigenvectors.

Which gives us the most sought after equation:

A2=SΛS−1SΛS−1=SΛ2S−1  ⟹  Ak=SΛkS−1 A^2= S\Lambda{}S^{-1}S\Lambda{}S^{-1}=S\Lambda{}^2S^{-1} \implies A^k=S\Lambda{}^kS^{-1}

Theorem: Ak→0A^k \rightarrow 0 as k→∞k \rightarrow \infty if all ∣λi∣<1|\lambda{}_i| < 1

Eigenvalues, Eigenvectors of A2A^2

If we have AA with eigenvalues λ1…λn\lambda{}_1 \ldots \lambda{}_n:

Said another way:

If Ax⃗=λx⃗A\vec{x} = \lambda \vec{x} then A2x⃗=λAx⃗=λ2x⃗A^2\vec{x} = \lambda A\vec{x} = \lambda{}^2\vec{x}

Facts

Facts About Diagonalizing