Eigenvalues & Eigenvectors

Let $V$ be a real vector space and $T \colon V \to V$ a linear map. A nonzero vector $\mathbf{v} \in V$ is an eigenvector of $T$ with eigenvalue $\lambda \in \mathbb{R}$ (or $\mathbb{C}$, when we work in the complex setting) if

$T(\mathbf{v}) = \lambda \mathbf{v}.$

The set

$E_\lambda = \{\mathbf{v} \in V \;\colon\; T(\mathbf{v}) = \lambda \mathbf{v}\}$

is the eigenspace of $T$ corresponding to $\lambda$. The set of all eigenvalues of $T$ is the spectrum, written $\sigma(T)$.

For any linear map $T \colon V \to V$ and any scalar $\lambda$, the eigenspace $E_\lambda$ is a subspace of $V$. In particular, $E_\lambda$ contains the zero vector and is closed under addition and scalar multiplication.

Rewrite the eigenspace as $E_\lambda = \ker(T - \lambda I)$, where $T - \lambda I \colon V \to V$ is the linear map $\mathbf{v} \mapsto T(\mathbf{v}) - \lambda \mathbf{v}$. To see the rewrite: $\mathbf{v} \in E_\lambda$ if and only if $T(\mathbf{v}) = \lambda \mathbf{v}$, which is equivalent to $T(\mathbf{v}) - \lambda \mathbf{v} = \mathbf{0}$, which is the statement $(T - \lambda I)(\mathbf{v}) = \mathbf{0}$. The kernel of any linear map is a subspace (Proposition 2 of Linear Algebra), so $E_\lambda$ is a subspace.

Concretely: the zero vector satisfies $T(\mathbf{0}) - \lambda \mathbf{0} = \mathbf{0} - \mathbf{0} = \mathbf{0}$, so $\mathbf{0} \in E_\lambda$. If $\mathbf{u}, \mathbf{v} \in E_\lambda$, then $T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v}) = \lambda \mathbf{u} + \lambda \mathbf{v} = \lambda(\mathbf{u} + \mathbf{v})$, so $\mathbf{u} + \mathbf{v} \in E_\lambda$. Scalar multiplication is analogous. $\blacksquare$

The geometric multiplicity of an eigenvalue $\lambda$ of $T$ is the dimension of its eigenspace:

$\operatorname{geo.mult}(\lambda) = \dim E_\lambda.$

For any actual eigenvalue $\lambda$, the eigenspace contains at least one nonzero eigenvector, so $\dim E_\lambda \geq 1$.

Let $D = \operatorname{diag}(d_1, d_2, \ldots, d_n)$ — the matrix with $d_i$ on the diagonal and zeros elsewhere. For each standard-basis vector $\mathbf{e}_i$, direct computation gives $D \mathbf{e}_i = d_i \mathbf{e}_i$. So $\mathbf{e}_i$ is an eigenvector with eigenvalue $d_i$ for every $i$. The eigenvalues are exactly the diagonal entries (with their multiplicities), and the eigenspaces are the coordinate axes — or sums of coordinate axes, when entries are repeated. A diagonal matrix is the easiest possible eigenvalue picture: the standard basis is already an eigenbasis.

Let $P \colon \mathbb{R}^2 \to \mathbb{R}^2$ be orthogonal projection onto the $x$-axis: $P(x, y) = (x, 0)$. Then $P \mathbf{e}_1 = (1, 0) = 1 \cdot \mathbf{e}_1$ and $P \mathbf{e}_2 = (0, 0) = 0 \cdot \mathbf{e}_2$. The eigenvectors are $\mathbf{e}_1$ with eigenvalue $1$ and $\mathbf{e}_2$ with eigenvalue $0$. The eigenspace $E_1$ is the $x$-axis (vectors that the projection leaves untouched), and $E_0$ is the $y$-axis (vectors that the projection collapses to the origin). The map is fully described by the data “stretch by $1$ along the $x$-axis, kill the $y$-axis” — and that data is exactly the eigenvalue / eigenspace pair. Example 15 of Linear Algebra used this projection to motivate change of basis; here it reappears as the prototypical eigenvalue example.

Let $R \colon \mathbb{R}^2 \to \mathbb{R}^2$ be rotation by $90°$ counterclockwise: $R(x, y) = (-y, x)$, with matrix $\begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$. No nonzero real vector is fixed up to scaling by a $90°$ rotation — every direction is mapped to the perpendicular direction, never to a scalar multiple of itself. So $R$ has no real eigenvalues at all.

Over the complex numbers, the situation changes. Try the candidate $\mathbf{v} = (1, -i)^\top$: $R \mathbf{v} = (-(-i), 1)^\top = (i, 1)^\top = i \cdot (1, -i)^\top$, so $\mathbf{v}$ is a complex eigenvector with eigenvalue $\lambda = i$. The conjugate $\overline{\mathbf{v}} = (1, i)^\top$ is an eigenvector with eigenvalue $\overline\lambda = -i$. Real matrices can have complex eigenvalues — and when they do, the eigenvalues come in complex conjugate pairs. The real-vs-complex distinction will return in §3 when we develop the characteristic polynomial.

Let $T \colon V \to V$ be a linear map and let $\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_k$ be eigenvectors of $T$ with eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_k$. If the eigenvalues are pairwise distinct, then the eigenvectors are linearly independent.

We induct on $k$. For $k = 1$, the claim is just $\mathbf{v}_1 \neq \mathbf{0}$, which is part of the definition of eigenvector.

For the induction step, assume the result for any $k - 1$ eigenvectors with distinct eigenvalues, and suppose

$c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + \cdots + c_k \mathbf{v}_k = \mathbf{0} \qquad (\star)$

for some scalars $c_1, \ldots, c_k$. Apply $T$ to both sides. Since $T$ is linear and each $\mathbf{v}_i$ is an eigenvector with eigenvalue $\lambda_i$,

$c_1 \lambda_1 \mathbf{v}_1 + c_2 \lambda_2 \mathbf{v}_2 + \cdots + c_k \lambda_k \mathbf{v}_k = \mathbf{0}. \qquad (\star\star)$

Now multiply $(\star)$ by $\lambda_k$ and subtract from $(\star\star)$ to eliminate the $\mathbf{v}_k$ term:

$c_1 (\lambda_1 - \lambda_k) \mathbf{v}_1 + c_2 (\lambda_2 - \lambda_k) \mathbf{v}_2 + \cdots + c_{k-1} (\lambda_{k-1} - \lambda_k) \mathbf{v}_{k-1} = \mathbf{0}.$

The first $k - 1$ eigenvectors have pairwise distinct eigenvalues (since all $k$ are pairwise distinct), so by the inductive hypothesis they are linearly independent. Each coefficient must therefore vanish:

$c_i (\lambda_i - \lambda_k) = 0 \quad \text{for every } i = 1, \ldots, k - 1.$

Since $\lambda_i \neq \lambda_k$, we conclude $c_i = 0$ for $i < k$. Plugging back into $(\star)$ leaves $c_k \mathbf{v}_k = \mathbf{0}$, and since $\mathbf{v}_k \neq \mathbf{0}$, $c_k = 0$ as well. So every $c_i$ vanishes — the eigenvectors are linearly independent. $\blacksquare$

A linear map $T \colon V \to V$ on a finite-dimensional vector space has at most $\dim V$ distinct eigenvalues.

A linearly independent set in $V$ has at most $\dim V$ elements (Proposition 5 of Linear Algebra). Choose one eigenvector from each distinct eigenvalue: by Proposition 2, the resulting list is linearly independent, so its length — the number of distinct eigenvalues — is at most $\dim V$. $\blacksquare$

The definition insists that $\mathbf{v} \neq \mathbf{0}$ for an eigenvector. The reason is bookkeeping: if we allowed $\mathbf{v} = \mathbf{0}$, then $T(\mathbf{0}) = \mathbf{0} = \lambda \cdot \mathbf{0}$ would hold for every scalar $\lambda$. The zero vector would be an “eigenvector for every eigenvalue,” and the notion of “the eigenvalue of $\mathbf{0}$” would be meaningless. By excluding it, each eigenvector pins down a unique eigenvalue, and the eigenvalue equation $T(\mathbf{v}) = \lambda \mathbf{v}$ becomes a constraint on the direction of $\mathbf{v}$ rather than a triviality. Note that the eigenspace $E_\lambda$ does contain $\mathbf{0}$ — otherwise it would not be a subspace — but $\mathbf{0}$ is not itself called an eigenvector.

For an $n \times n$ matrix $A$, the characteristic polynomial of $A$ is

$p_A(\lambda) = \det(\lambda I - A).$

It is a monic polynomial in $\lambda$ of degree $n$ with real coefficients (assuming $A$ is real). The characteristic equation is $p_A(\lambda) = 0$.

For any $n \times n$ matrix $A$ and any scalar $\lambda \in \mathbb{C}$:

$\lambda \text{ is an eigenvalue of } A \quad \Longleftrightarrow \quad p_A(\lambda) = 0.$

$\lambda$ is an eigenvalue of $A$ iff there exists a nonzero $\mathbf{v}$ with $(A - \lambda I)\mathbf{v} = \mathbf{0}$, i.e. $\ker(A - \lambda I) \neq \{\mathbf{0}\}$. By the invertibility characterization (Corollary 1 of §8 of Linear Algebra), this is equivalent to $A - \lambda I$ being non-invertible, which is equivalent to $\det(A - \lambda I) = 0$.

Finally, $\det(\lambda I - A) = (-1)^n \det(A - \lambda I)$ — multiplying every row of a matrix by $-1$ multiplies the determinant by $(-1)^n$ — so the two determinants vanish together. Therefore $p_A(\lambda) = \det(\lambda I - A) = 0$ if and only if $\lambda$ is an eigenvalue. $\blacksquare$

The algebraic multiplicity of an eigenvalue $\lambda$ of $A$ is its multiplicity as a root of the characteristic polynomial: the largest integer $m \geq 1$ such that $(\lambda - \lambda_0)^m$ divides $p_A(\lambda)$ when $\lambda_0$ is the eigenvalue in question. We write $m_a(\lambda)$ for this number.

For any eigenvalue $\lambda$ of $A$, the geometric multiplicity is at most the algebraic multiplicity:

$\dim E_\lambda = m_g(\lambda) \;\leq\; m_a(\lambda).$

Let $k = \dim E_\lambda$ be the geometric multiplicity. Choose a basis $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ of the eigenspace $E_\lambda$, and extend it to a basis $\mathcal{B} = \{\mathbf{v}_1, \ldots, \mathbf{v}_k, \mathbf{w}_1, \ldots, \mathbf{w}_{n - k}\}$ of $V$ (using the basis-extension lemma from §3 of Linear Algebra). In this basis, the matrix of $A$ has block form

$[A]_{\mathcal{B}} = \begin{pmatrix} \lambda I_k & B \\ 0 & C \end{pmatrix}$

for some $k \times (n-k)$ block $B$ and $(n-k) \times (n-k)$ block $C$. The reason: $A \mathbf{v}_i = \lambda \mathbf{v}_i$ for each $i \leq k$, so the first $k$ columns of $[A]_{\mathcal{B}}$ are $\lambda$ times the corresponding standard basis vectors. The bottom-left block is zero because the eigenvectors $\mathbf{v}_i$ have no $\mathbf{w}_j$ components. The remaining columns are arbitrary, filling out the blocks $B$ and $C$.

Similar matrices share the characteristic polynomial (Lemma 1 below), so

$p_A(\lambda') = \det(\lambda' I - [A]_{\mathcal{B}}) = \det\begin{pmatrix} (\lambda' - \lambda) I_k & -B \\ 0 & \lambda' I_{n-k} - C \end{pmatrix} = (\lambda' - \lambda)^k \cdot \det(\lambda' I_{n-k} - C),$

using the block-triangular determinant formula. The factor $(\lambda' - \lambda)^k$ contributes at least $k$ to the multiplicity of $\lambda$ as a root, so $m_a(\lambda) \geq k = m_g(\lambda)$. $\blacksquare$

If $A$ and $A'$ are similar — meaning $A' = P^{-1} A P$ for some invertible $P$ — then $p_{A'}(\lambda) = p_A(\lambda)$.

Using the multiplicativity of the determinant (Theorem 6 of Linear Algebra) and $\det(P^{-1}) \det(P) = 1$:

$p_{A'}(\lambda) = \det(\lambda I - P^{-1} A P) = \det\bigl(P^{-1}(\lambda I - A) P\bigr).$

Pulling the determinants apart and rearranging gives

$\det(P^{-1}) \det(\lambda I - A) \det(P) = \det(\lambda I - A) = p_A(\lambda).$

So $p_{A'}(\lambda) = p_A(\lambda)$. $\blacksquare$

For $A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$:

$p_A(\lambda) = \det\begin{pmatrix} \lambda - a & -b \\ -c & \lambda - d \end{pmatrix} = (\lambda - a)(\lambda - d) - bc = \lambda^2 - (a + d)\lambda + (ad - bc).$

The coefficients are familiar: $a + d = \operatorname{tr}(A)$ and $ad - bc = \det A$, so

$p_A(\lambda) = \lambda^2 - \operatorname{tr}(A)\,\lambda + \det A.$

Solving the quadratic gives the eigenvalues

$\lambda = \frac{\operatorname{tr}(A) \pm \sqrt{\operatorname{tr}(A)^2 - 4 \det A}}{2}.$

Apply to $A = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}$: $\operatorname{tr}(A) = 5$, $\det A = 6$, so $p_A(\lambda) = \lambda^2 - 5\lambda + 6 = (\lambda - 2)(\lambda - 3)$. Eigenvalues $\lambda_1 = 3$, $\lambda_2 = 2$.

For the 90° rotation $A = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$ from Example 3: $\operatorname{tr}(A) = 0$, $\det A = 1$, so $p_A(\lambda) = \lambda^2 + 1$. The roots are $\lambda = \pm i$ — no real eigenvalues. Over the complex numbers the eigenvectors are $\mathbf{v}_{+i} = (1, -i)^\top$ and $\mathbf{v}_{-i} = (1, i)^\top$. A real matrix can have complex eigenvalues; when it does, they come in complex conjugate pairs (because the characteristic polynomial has real coefficients, so non-real roots arise in conjugate pairs).

For $A = \begin{pmatrix} 2 & 1 & 0 \\ 0 & 2 & 1 \\ 0 & 0 & 3 \end{pmatrix}$ — upper-triangular, so the characteristic polynomial factors immediately:

$p_A(\lambda) = (\lambda - 2)(\lambda - 2)(\lambda - 3) = (\lambda - 2)^2 (\lambda - 3).$

Eigenvalues: $\lambda = 2$ with algebraic multiplicity 2 and $\lambda = 3$ with algebraic multiplicity 1. Whether $\lambda = 2$‘s geometric multiplicity is 1 or 2 depends on the off-diagonal entries — this matrix happens to be defective (geometric multiplicity 1), so it is not diagonalizable. We return to this in §5.

Over the real numbers, a polynomial of degree $n$ may have fewer than $n$ real roots: as Example 5 showed, $\lambda^2 + 1$ has no real roots even though its degree is 2. The Fundamental Theorem of Algebra guarantees that over the complex numbers, every degree-$n$ polynomial has exactly $n$ roots counted with multiplicity. So a real $n \times n$ matrix always has $n$ complex eigenvalues (with multiplicity), but possibly fewer real ones.

Real eigenvalues correspond to invariant directions in the geometric sense of §1 — directions the map stretches without rotating. Complex eigenvalues correspond to rotational behavior: when $\lambda = a + bi$ is complex, the action of $A$ on the real 2-plane spanned by the real and imaginary parts of the complex eigenvector is a rotation by angle $\arctan(b/a)$ combined with a scaling by $|\lambda| = \sqrt{a^2 + b^2}$. The 90° rotation in Example 5 has $|\lambda| = 1$ (pure rotation, no scaling) and angle $\pi/2$. Real symmetric matrices — the headline case of §6 — never have complex eigenvalues, because their characteristic polynomials factor over the reals.

Computing eigenvalues by writing down $p_A(\lambda)$ and factoring it works fine for $n = 2$ and $n = 3$ and is unusable for $n \geq 5$. Wilkinson’s polynomial — the characteristic polynomial of a deliberately constructed $20 \times 20$ matrix — is so ill-conditioned that floating-point errors of order $2^{-23}$ in the coefficients move the roots by order 1. Even for benign-looking matrices, the polynomial coefficients accumulate cancellation error and the roots become hopelessly inaccurate.

Production eigenvalue solvers use the QR algorithm (Francis, 1961) — an iterative orthogonal-similarity transformation that converges to a Schur form, from which eigenvalues read off the diagonal — and the Lanczos / Arnoldi iterations for sparse matrices when only a few extremal eigenvalues are needed. Both algorithms bypass the characteristic polynomial entirely. We cite this without developing the numerical theory; Trefethen & Bau, Numerical Linear Algebra (Lectures 24–30) is the right reference. The notebook for this topic uses np.linalg.eig and np.linalg.eigh for matrices larger than $3 \times 3$ — and so, in practice, do you.

An $n \times n$ matrix $A$ is diagonalizable (over the field $\mathbb{F}$, typically $\mathbb{R}$ or $\mathbb{C}$) if there exists an invertible $P$ and a diagonal matrix $D$ — both with entries in $\mathbb{F}$ — such that

$A = P D P^{-1}.$

Equivalently, $A$ is similar to a diagonal matrix.

An $n \times n$ matrix $A$ is diagonalizable if and only if there exists a basis of $\mathbb{R}^n$ (or $\mathbb{C}^n$, depending on the field) consisting of eigenvectors of $A$. In that case, the columns of $P$ are the eigenvectors, and the diagonal entries of $D$ are the corresponding eigenvalues.

($\Rightarrow$) Suppose $A = P D P^{-1}$ with $D = \operatorname{diag}(\lambda_1, \ldots, \lambda_n)$ and $P$ invertible with columns $\mathbf{p}_1, \ldots, \mathbf{p}_n$. Multiply both sides on the right by $P$ to clear the inverse: $A P = P D$. Reading column $j$ of both sides: $A \mathbf{p}_j = \lambda_j \mathbf{p}_j$. So each column of $P$ is an eigenvector with eigenvalue equal to the corresponding diagonal entry of $D$. Since $P$ is invertible, its columns are linearly independent, hence a basis of $\mathbb{R}^n$.

($\Leftarrow$) Suppose $\{\mathbf{p}_1, \ldots, \mathbf{p}_n\}$ is a basis of eigenvectors with eigenvalues $\lambda_1, \ldots, \lambda_n$. Form the matrix $P$ whose columns are these eigenvectors; $P$ is invertible because its columns are linearly independent. Form $D = \operatorname{diag}(\lambda_1, \ldots, \lambda_n)$. Then $A P = P D$ by column-by-column computation (each column of $A P$ is $A \mathbf{p}_j = \lambda_j \mathbf{p}_j$, and each column of $P D$ is $\lambda_j \mathbf{p}_j$). Multiply both sides on the right by $P^{-1}$ to conclude $A = P D P^{-1}$. $\blacksquare$

If an $n \times n$ matrix $A$ has $n$ distinct eigenvalues (each with algebraic multiplicity 1), then $A$ is diagonalizable.

Pick one eigenvector $\mathbf{v}_i$ from each eigenspace $E_{\lambda_i}$. By Proposition 2 (§2), eigenvectors for distinct eigenvalues are linearly independent. So $\{\mathbf{v}_1, \ldots, \mathbf{v}_n\}$ is a list of $n$ linearly independent vectors in $\mathbb{R}^n$, hence a basis. By Theorem 2, $A$ is diagonalizable. $\blacksquare$

An $n \times n$ matrix $A$ is diagonalizable (over a field containing all its eigenvalues) if and only if, for every eigenvalue $\lambda$, the geometric multiplicity equals the algebraic multiplicity:

$m_g(\lambda) = m_a(\lambda) \quad \text{for every } \lambda \in \sigma(A).$

($\Leftarrow$) Suppose $m_g(\lambda) = m_a(\lambda)$ for every eigenvalue. The eigenspaces $E_{\lambda_i}$ for distinct eigenvalues are linearly independent subspaces (extending Proposition 2 from single eigenvectors to whole eigenspaces — the same induction works with bases of $E_{\lambda_i}$ in place of single vectors). The total dimension of the direct sum is

$\sum_i m_g(\lambda_i) = \sum_i m_a(\lambda_i) = \deg p_A = n,$

so the eigenspaces span all of $\mathbb{R}^n$ (assuming the field contains all the eigenvalues). Concatenating bases of each eigenspace gives a basis of $\mathbb{R}^n$ consisting of eigenvectors; by Theorem 2, $A$ is diagonalizable.

($\Rightarrow$) Suppose $A = P D P^{-1}$ with $D = \operatorname{diag}(d_1, \ldots, d_n)$. The eigenspace $E_\lambda$ in this representation is the span of the standard basis vectors $\mathbf{e}_j$ for which $d_j = \lambda$ — so $m_g(\lambda)$ equals the count of diagonal entries equal to $\lambda$. The characteristic polynomial is $p_A(\lambda') = \prod_j (\lambda' - d_j)$ (the diagonal makes this immediate), so $m_a(\lambda)$ is also the count of diagonal entries equal to $\lambda$. The two counts agree. $\blacksquare$

Continue with $A = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}$. Eigenvalues $\lambda_1 = 3$ and $\lambda_2 = 2$ (Example 4) — distinct, so $A$ is diagonalizable by Corollary 2. Find eigenvectors:

• For $\lambda_1 = 3$: solve $(A - 3I)\mathbf{v} = \mathbf{0}$, i.e. $\begin{pmatrix} 1 & -2 \\ 1 & -2 \end{pmatrix}\mathbf{v} = \mathbf{0}$. The kernel is spanned by $\mathbf{v}_1 = (2, 1)^\top$.
• For $\lambda_2 = 2$: solve $(A - 2I)\mathbf{v} = \mathbf{0}$, i.e. $\begin{pmatrix} 2 & -2 \\ 1 & -1 \end{pmatrix}\mathbf{v} = \mathbf{0}$. The kernel is spanned by $\mathbf{v}_2 = (1, 1)^\top$.

Form $P = \begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix}$ and $D = \begin{pmatrix} 3 & 0 \\ 0 & 2 \end{pmatrix}$. Verify by direct multiplication that $A P = P D$. Then $A = P D P^{-1}$ where $P^{-1} = \begin{pmatrix} 1 & -1 \\ -1 & 2 \end{pmatrix}$. In the eigenbasis, $A$ is the pair of independent scalings — by 3 along $\mathbf{v}_1$ and by 2 along $\mathbf{v}_2$.

If $A$ and $A'$ are similar matrices, they share:

(a) the characteristic polynomial (and hence the eigenvalue multiset with algebraic multiplicities),

(b) the trace,

(c) the determinant,

(d) the rank,

(e) the nullity,

(f) the geometric multiplicity of each eigenvalue.

(a) is Lemma 1 (§3). (b) Trace equals minus the coefficient of $\lambda^{n-1}$ in $p_A(\lambda)$ (this is the elementary-symmetric-polynomial identity), so trace is determined by the characteristic polynomial, hence invariant. (c) Determinant equals $(-1)^n$ times the constant term of $p_A(\lambda)$, so it too is determined by the characteristic polynomial. (d) The column space of $P A P^{-1}$ has the same dimension as the column space of $A$: $P^{-1}$ on the right doesn’t change dimension (it’s a bijection from $\mathbb{R}^n$ to itself), and $P$ on the left maps the column space isomorphically. So rank is preserved. (e) Nullity equals $n - \operatorname{rank}$, so it follows from (d) by rank-nullity. (f) The geometric multiplicity of $\lambda$ is

$m_g(\lambda) = \dim \ker(A - \lambda I) = \dim \ker(P^{-1}(A - \lambda I) P) = \dim \ker(A' - \lambda I),$

the same in both bases. $\blacksquare$

For diagonalizable $A = P D P^{-1}$ with $D = \operatorname{diag}(\lambda_1, \ldots, \lambda_n)$:

$A^k = (P D P^{-1})^k = P D^k P^{-1} \quad \text{where} \quad D^k = \operatorname{diag}(\lambda_1^k, \ldots, \lambda_n^k).$

The $P^{-1} P$ pairs telescope. The same identity holds for any function $f$ defined on the spectrum:

$f(A) = P f(D) P^{-1} \quad \text{where} \quad f(D) = \operatorname{diag}(f(\lambda_1), \ldots, f(\lambda_n)).$

In particular,

$e^A = P \, e^D \, P^{-1} = P \operatorname{diag}(e^{\lambda_1}, \ldots, e^{\lambda_n}) P^{-1}.$

This is the spectral mapping observation: functions of a diagonalizable matrix are computed by applying the function to the diagonal of $D$, then sandwich-wrapping. Closed-form matrix exponentials, square roots of positive-definite matrices, and whitening transformations are all instances.

An $n \times n$ matrix $A$ is defective if it has an eigenvalue whose geometric multiplicity is strictly less than its algebraic multiplicity. Equivalently, $A$ has fewer than $n$ linearly independent eigenvectors, so it admits no eigenbasis and is not diagonalizable.

Let $J_2 = \begin{pmatrix} 2 & 1 \\ 0 & 2 \end{pmatrix}$. Characteristic polynomial: $p_{J_2}(\lambda) = (\lambda - 2)^2$. So $\lambda = 2$ is the only eigenvalue, with algebraic multiplicity 2.

Compute the eigenspace by solving $(J_2 - 2I)\mathbf{v} = \mathbf{0}$:

$(J_2 - 2I) = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}, \qquad \ker(J_2 - 2I) = \operatorname{span}\{(1, 0)^\top\}.$

The eigenspace is one-dimensional. Geometric multiplicity 1, algebraic multiplicity 2: $J_2$ is defective. There is no basis of $\mathbb{R}^2$ consisting of eigenvectors of $J_2$, so $J_2$ is not diagonalizable. The matrix is already in Jordan form — it is a Jordan block.

A Jordan block of size $k$ with eigenvalue $\lambda$ is the $k \times k$ matrix

$J_k(\lambda) = \begin{pmatrix} \lambda & 1 & & \\ & \lambda & 1 & \\ & & \ddots & \ddots \\ & & & \lambda & 1 \\ & & & & \lambda \end{pmatrix}$

with $\lambda$ on the diagonal, $1$s on the immediate superdiagonal, and $0$s elsewhere. The matrix $J_2$ in Example 8 is the Jordan block of size 2 with eigenvalue 2.

Every $n \times n$ matrix $A$ over the complex numbers is similar to a block-diagonal matrix $J$ whose blocks are Jordan blocks $J_{k_i}(\lambda_i)$. The list of Jordan blocks — their sizes and eigenvalues — is unique up to ordering, and is the Jordan canonical form of $A$. The Jordan form is the canonical generalization of diagonalization: diagonalizable matrices are exactly those whose Jordan form has all $1 \times 1$ blocks.

The proof of Theorem 5 (see Hoffman-Kunze Chapter 7 or Horn-Johnson Chapter 3) requires generalized eigenvectors, Jordan chains, and a careful induction on the algebraic multiplicity of each eigenvalue. The development is mathematically beautiful but long, and the practical payoff is modest: defective matrices are rare in clean applications, and when they do arise (linear-systems repeated eigenvalues, bifurcation analysis), the relevant techniques can be developed locally on a case-by-case basis.

We treat Linear Systems & Matrix Exponential as the place where case-by-case handling for $2 \times 2$ defective systems lives; the full Jordan theory belongs to a future topic on canonical forms or numerical linear algebra. For the rest of this topic, we proceed as if all matrices are diagonalizable — a working assumption that holds for the symmetric matrices §6 puts at the center, and for generic random matrices in general.

The set of $n \times n$ matrices with all distinct eigenvalues is open and dense in the space of all matrices $\mathbb{R}^{n \times n}$; its complement, where some eigenvalues coincide, is a set of measure zero (the discriminant of the characteristic polynomial vanishes on a hypersurface). So a random matrix is diagonalizable with probability 1.

But defective matrices live on the boundary between qualitatively different behaviors — e.g. the boundary between a stable spiral and a stable node in a 2D phase portrait, or the boundary between under-damped and over-damped oscillation in a second-order linear system. Bifurcation theory studies exactly what happens at such boundaries, and there defectiveness is the central object, not an exception. ML reality: repeated eigenvalues of the Hessian at a critical point are a measure-zero coincidence in a generic loss landscape, but at symmetric critical points (e.g. loss invariant under permutation of neurons in a wide network) they become structural rather than accidental.

A real $n \times n$ matrix $A$ is symmetric if $A^\top = A$, i.e. $a_{ij} = a_{ji}$ for every pair of indices $(i, j)$.

Every eigenvalue of a real symmetric matrix $A$ is real.

Let $\lambda \in \mathbb{C}$ be an eigenvalue of the real symmetric matrix $A$, with a corresponding (possibly complex) eigenvector $\mathbf{v} \in \mathbb{C}^n$: $A \mathbf{v} = \lambda \mathbf{v}$.

Take the conjugate transpose ($\mathbf{v}^* = \overline{\mathbf{v}}^\top$, $A^* = \overline{A}^\top$): $\mathbf{v}^* A^* = \bar\lambda \mathbf{v}^*$. Since $A$ is real and symmetric, $A^* = A^\top = A$. So $\mathbf{v}^* A = \bar\lambda \mathbf{v}^*$.

Multiply the eigenvalue equation $A \mathbf{v} = \lambda \mathbf{v}$ on the left by $\mathbf{v}^*$: $\mathbf{v}^* A \mathbf{v} = \lambda \mathbf{v}^* \mathbf{v}$. Multiply $\mathbf{v}^* A = \bar\lambda \mathbf{v}^*$ on the right by $\mathbf{v}$: $\mathbf{v}^* A \mathbf{v} = \bar\lambda \mathbf{v}^* \mathbf{v}$.

Equate the two expressions: $\lambda \mathbf{v}^* \mathbf{v} = \bar\lambda \mathbf{v}^* \mathbf{v}$. Since $\mathbf{v} \neq \mathbf{0}$, $\mathbf{v}^* \mathbf{v} = \|\mathbf{v}\|^2 > 0$. Divide both sides by $\|\mathbf{v}\|^2$ to get $\lambda = \bar\lambda$, hence $\lambda \in \mathbb{R}$. $\blacksquare$

If $A$ is real symmetric and $\mathbf{v}_1, \mathbf{v}_2$ are eigenvectors of $A$ corresponding to distinct eigenvalues $\lambda_1 \neq \lambda_2$, then $\mathbf{v}_1 \perp \mathbf{v}_2$.

Compute $\langle A \mathbf{v}_1, \mathbf{v}_2 \rangle$ two ways. First, using $A \mathbf{v}_1 = \lambda_1 \mathbf{v}_1$:

$\langle A \mathbf{v}_1, \mathbf{v}_2 \rangle = \langle \lambda_1 \mathbf{v}_1, \mathbf{v}_2 \rangle = \lambda_1 \langle \mathbf{v}_1, \mathbf{v}_2 \rangle.$

Second, pushing the $A$ across the inner product using $A^\top = A$ (a property of the symmetric case, see §10 of Linear Algebra on inner products and adjoints): $\langle A \mathbf{v}_1, \mathbf{v}_2 \rangle = \langle \mathbf{v}_1, A \mathbf{v}_2 \rangle = \langle \mathbf{v}_1, \lambda_2 \mathbf{v}_2 \rangle = \lambda_2 \langle \mathbf{v}_1, \mathbf{v}_2 \rangle.$

Equating: $\lambda_1 \langle \mathbf{v}_1, \mathbf{v}_2 \rangle = \lambda_2 \langle \mathbf{v}_1, \mathbf{v}_2 \rangle$, i.e. $(\lambda_1 - \lambda_2) \langle \mathbf{v}_1, \mathbf{v}_2 \rangle = 0$. Since $\lambda_1 \neq \lambda_2$, the inner product $\langle \mathbf{v}_1, \mathbf{v}_2 \rangle = 0$. $\blacksquare$

Let $A$ be a real $n \times n$ symmetric matrix. Then there exists an orthogonal matrix $Q$ (i.e. $Q^\top Q = I$) and a real diagonal matrix $\Lambda = \operatorname{diag}(\lambda_1, \ldots, \lambda_n)$ such that

$A = Q \, \Lambda \, Q^\top.$

The columns of $Q$ form an orthonormal basis of $\mathbb{R}^n$ consisting of eigenvectors of $A$; the diagonal entries of $\Lambda$ are the corresponding eigenvalues (with algebraic multiplicities, conventionally listed in non-increasing order $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n$).

Induction on $n$.

Base case ($n = 1$). A $1 \times 1$ matrix $A = (a_{11})$ is trivially symmetric and diagonal: take $Q = (1)$ and $\Lambda = (a_{11})$.

Inductive step. Assume the spectral theorem holds for $(n-1) \times (n-1)$ symmetric matrices. Let $A$ be an $n \times n$ real symmetric matrix.

By Lemma 2, the characteristic polynomial $p_A(\lambda)$ has at least one real root $\lambda_1$ (in fact, all its roots are real, but we only need one for the induction). By Theorem 1, $\lambda_1$ is a real eigenvalue. Pick a corresponding real unit eigenvector $\mathbf{q}_1 \in \mathbb{R}^n$ with $\|\mathbf{q}_1\| = 1$.

Extend $\{\mathbf{q}_1\}$ to an orthonormal basis $\{\mathbf{q}_1, \mathbf{w}_2, \ldots, \mathbf{w}_n\}$ of $\mathbb{R}^n$ via Gram-Schmidt (§10 of Linear Algebra). Form the orthogonal matrix $\tilde Q$ whose columns are these vectors; orthogonal because the columns are orthonormal.

Consider the matrix $\tilde Q^\top A \tilde Q$. Two things are true about it.

First, it is symmetric: $(\tilde Q^\top A \tilde Q)^\top = \tilde Q^\top A^\top \tilde Q = \tilde Q^\top A \tilde Q$ (using $A^\top = A$).

Second, its first column equals $\lambda_1 \mathbf{e}_1$: the $j$-th column of $\tilde Q^\top A \tilde Q$ is \tilde Q^\top A (\text{j-th column of } \tilde Q). For $j = 1$, this is $\tilde Q^\top A \mathbf{q}_1 = \tilde Q^\top (\lambda_1 \mathbf{q}_1) = \lambda_1 \tilde Q^\top \mathbf{q}_1 = \lambda_1 \mathbf{e}_1$, since $\tilde Q^\top \mathbf{q}_1 = \mathbf{e}_1$ by construction (the first column of $\tilde Q$ is $\mathbf{q}_1$).