6.6 Orthogonal Projections and the Spectral Theorem – Linear Algebra, 5th Edition

6.6 Orthogonal Projections and the Spectral Theorem

In this section, we rely heavily on Theorems 6.16 (p. 369) and 6.17 (p. 371) to develop an elegant representation of a normal (if F=C) or a self-adjoint (if F=R) operator T on a finite-dimensional inner product space. We prove that T can be written in the form λ1T1+λ2T2++λkTk, where λ1, λ2, , λk are the distinct eigenvalues of T T1, T2, , Tk are orthogonal projections. We must first develop some results about these special projections.

We assume that the reader is familiar with the results about direct sums developed at the end of Section 5.2. The special case where V is a direct sum of two subspaces is considered in the exercises of Section 1.3.

Recall from the exercises of Section 2.1 that if V=W1W2, then a linear operator T on V is the projection on W1 along W2 if, whenever x=x1+x2, with x1W1 and x2W2, we have T(x)=x1. By Exercise 27 of Section 2.1, we have

R ( T ) = W 1 = { x V :  T ( x ) = x } and N ( T ) = W 2 .

So V=R(T)N(T). Thus there is no ambiguity if we refer to T as a “projection on W1” or simply as a “projection.” In fact, it can be shown (see Exercise 17 of Section 2.3) that T is a projection if and only if T=T2. Because V=W1W2=W1W3 does not imply that W2=W3, we see that W1 does not uniquely determine T. For an orthogonal projection T, however, T is uniquely determined by its range.

Definition.

Let V be an inner product space, and let T: VV be a projection. We say that T is an orthogonal projection if R(T)=N(T) and N(T)=R(T).

Note that by Exercise 13(c) of Section 6.2, if V is finite-dimensional, we need only assume that one of the equalities in this definition holds. For example, if R(T)=N(T), then R(T)=R(T)=N(T).

An orthogonal projection is not the same as an orthogonal operator. In Figure 6.5, T is an orthogonal projection, but T is clearly not an orthogonal operator because ||T(v)||||v||.

Figure 6.5

Now assume that W is a finite-dimensional subspace of an inner product space V. In the notation of Theorem 6.6 (p. 347), we can define a function T: VV by T(y)=u. It is easy to show that T is an orthogonal projection on W. We can say even more—there exists exactly one orthogonal projection on W. For if T and U are orthogonal projections on W, then R(T)=W=R(U). Hence N(T)=R(T)=R(U)=N(U), and since every projection is uniquely determined by its range and null space, we have T=U. We call T the orthogonal projection of V on W.

To understand the geometric difference between an arbitrary projection on W and the orthogonal projection on W, let V=R2 and W=span{(1, 1)}. Define U and T as in Figure 6.5, where T(v) is the foot of a perpendicular from v on the line y=x and U(a1, a2)=(a1, a1). Then T is the orthogonal projection of V on W, and U is a different projection on W. Note that vT(v)W, whereas vU(v)W. From Figure 6.5, we see that T(v) is the “best approximation in W to v”; that is, if wW, then ||wv||||T(v)v||. In fact, this approximation property characterizes T. These results follow immediately from the corollary to Theorem 6.6 (p. 348).

As an application to Fourier analysis, recall the inner product space H and the orthonormal set S in Example 9 of Section 6.1. Define a trigonometric polynomial of degree n to be a function gH of the form

g ( t ) = j = n n a j f j ( t ) = j = n n a j e i j t ,

where an or an is nonzero.

Let fH. We show that the best approximation to f by a trigonometric polynomial of degree less than or equal to n is the trigonometric polynomial whose coefficients are the Fourier coefficients of f relative to the orthonormal set S. For this result, let W=span({fj: |j|n}), and let T be the orthogonal projection of H on W. The corollary to Theorem 6.6 (p. 348) tells us that the best approximation to f by a function in W is

T ( f ) = j = n n f ,    f j f j .

For an application of this material to electronic music, visit goo.gl/EN5Fai.

An algebraic characterization of orthogonal projections follows in the next theorem.

Theorem 6.24.

Let V be an inner product space, and let T be a linear operator on V. Then T is an orthogonal projection if and only if T has an adjoint T* and T2=T=T*.

Proof.

Suppose that T is an orthogonal projection. Since T2=T because T is a projection, we need only show that T* exists and T=T*. Now V=R(T)N(T) and R(T)=N(T). Let x, yV. Then we can write x=x1+x2 and y=y1+y2, where x1, y1R(T) and x2, y2N(T). Hence

x ,  T ( y ) = x 1 + x 2 ,   y 1 = x 1 ,   y 1 + x 2 ,   y 1 = x 1 ,   y 1

and

T ( x ) ,   y = x 1 ,   y 1 + y 2 = x 1 ,   y 1 + x 1 ,   y 2 = x 1 ,   y 1 .

So x, T(y) = T(x), y for all x, yV thus T* exists and T=T*.

Now suppose that T2=T=T*. Then T is a projection by Exercise 17 of Section 2.3, and hence we must show that R(T)=N(T) and R(T)=N(T). Let xR(T) and yN(T). Then x=T(x)=T*(x), and so

x ,   y = T* ( x ) ,   y = x ,  T ( y ) = x ,   0 = 0.

Therefore xN(T), from which it follows that R(T)N(T).

Let yN(T). We must show that yR(T), that is, T(y)=y. Now

| | y T ( y ) | | 2 = y T ( y ) ,   y T ( y ) = y ,   y T ( y ) T ( y ) ,   y T ( y ) .

Since yT(y)N(T), the first term must equal zero. But also

T ( y ) ,   y T ( y ) = y ,  T*( y T ( y ) ) = y ,  T ( y T ( y ) ) = y ,   0 = 0.

Thus yT(y)=0; that is, y=T(y)R(T). Hence R(T)=N(T).

Using the preceding results, we have R(T)=N(T)N(T) by Exercise 13(b) of Section 6.2. Now suppose that xR(T). For any yV, we have T(x), y = x, T*(y) = x, T(y) =0. So T(x)=0, and thus xN(T). Hence R(T)=N(T).

Let V be a finite-dimensional inner product space, W be a subspace of V, and T be the orthogonal projection of V on W. We may choose an orthonormal basis β={v1, v2, , vn} for V such that {v1, v2, , vk} is a basis for W. Then [T]β is a diagonal matrix with ones as the first k diagonal entries and zeros elsewhere. In fact, [T]β has the form

( I k O 1 O 2 O 3 ) .

If U is any projection on W, we may choose a basis γ for V such that [U]γ has the form above; however γ is not necessarily orthonormal.

We are now ready for the principal theorem of this section.

Theorem 6.25 (The Spectral Theorem).

Suppose that T is a linear operator on a finite-dimensional inner product space V over F with the distinct eigenvalues λ1, λ2, , λk. Assume that T is normal if F=C and that T is self-adjoint if F=R. For each i(1ik), let Wi be the eigenspace of T corresponding to the eigenvalue λi, and let Ti be the orthogonal projection of V on Wi. Then the following statements are true.

  1. (a) V=W1W2Wk.

  2. (b) If Wi denotes the direct sum of the subspaces Wj for ji, then Wi=Wi.

  3. (c) TiTj=δijTi for 1i, jk.

  4. (d) I=T1+T2++Tk.

  5. (e) T=λ1T1+λ2T2++λkTk.

Proof.

(a) By Theorems 6.16 (p. 369) and 6.17 (p. 371), T is diagonalizable;

so

V = W 1 W 2 W k

by Theorem 5.10 (p. 277).

(b) If xWi and yWj for some ij, then x, y =0 by Theorem 6.15(d) (p. 368). It follows easily from this result that WiWi. From (a), we have

dim ( W i ) = j i dim ( W j ) = dim ( V ) dim ( W i ) .

On the other hand, we have dim(Wi)=dim(V)dim(Wi) by Theorem 6.7(c) (p. 349). Hence Wi=Wi, proving (b).

(c) The proof of (c) is left as an exercise.

(d) Since Ti is the orthogonal projection of V on Wi, it follows from (b) that N(Ti)=R(Ti)=Wi=Wi. Hence, for xV, we have x=x1+x2++xk, where Ti(x)=xiWi, proving (d).

(e) For xV, write x=x1+x2++xk, where xiWi. Then

T ( x ) = T ( x 1 ) + T ( x 2 ) + + T ( x k ) = λ 1 x 1 + λ 2 x 2 + + λ k x k = λ 1 T 1 ( x ) + λ 2 T 2 ( x ) + + λ k T k ( x ) = ( λ 1 T 1 + λ 2 T 2 + + λ k T k ) ( x ) .

The set {λ1, λ2, , λk} of eigenvalues of T is called the spectrum of T, the sum I=T1+T2++Tk in (d) is called the resolution of the identity operator induced by T, and the sum T=λ1T1+λ2T2++λkTk in (e) is called the spectral decomposition of T. The spectral decomposition of T is unique up to the order of its eigenvalues.

With the preceding notation, let β be the union of orthonormal bases of the Wi’s and let mi=dim(Wi). (Thus mi is the multiplicity of λi.) Then [T]β has the form

( λ 1 I m 1 O O O λ 2 I m 2 O O O λ k I m k ) ;

that is, [T]β is a diagonal matrix in which the diagonal entries are the eigenvalues λi of T, and each λi is repeated mi times. If λ1T1+λ2T2++λkTk is the spectral decomposition of T, then it follows (from Exercise 7) that g(T)=g(λ1)T1+g(λ2)T2++g(λk)Tk for any polynomial g. This fact is used below.

We now list several interesting corollaries of the spectral theorem; many more results are found in the exercises. For what follows, we assume that T is a linear operator on a finite-dimensional inner product space V over F.

Corollary 1.

If F=C then T is normal if and only if T*=g(T) for some polynomial g.

Proof.

Suppose first that T is normal. Let T=λ1T1+λ2T2++λkTk be the spectral decomposition of T. Taking the adjoint of both sides of the preceding equation, we have T*=λ¯1T1+λ¯2T2++λ¯kTk since each Ti is self-adjoint. Using the Lagrange interpolation formula (see page 53), we may choose a polynomial g such that g(λi)=λ¯i for 1ik. Then

g ( T ) = g ( λ 1 ) T 1 + g ( λ 2 ) T 2 + + g ( λ k ) T k = λ ¯ 1 T 1 + λ ¯ 2 T 2 + + λ ¯ k T k = T* .

Conversely, if T*=g(T) for some polynomial g, then T commutes with T* since T commutes with every polynomial in T. So T is normal.

Corollary 2.

If F=C, then T is unitary if and only if T is normal and |λ|=1 for every eigenvalue λ of T.

Proof.

If T is unitary, then T is normal and every eigenvalue of T has absolute value 1 by Corollary 2 to Theorem 6.18 (p. 379).

Let T=λ1T1+λ2T2++λkTk be the spectral decomposition of T. If |λ|=1 for every eigenvalue λ of T, then by (c) of the spectral theorem,

TT* = ( λ 1 T 1 + λ 2 T 2 + + λ k T k ) ( λ ¯ 1 T 1 + λ ¯ 2 T 2 + + λ ¯ k T k ) = | λ 1 | 2 T 1 + | λ 2 | 2 T 2 + + | λ k | 2 T k = T 1 + T 2 + + T k = I .

Hence T is unitary.

Corollary 3.

If F=C, then T is self-adjoint if and only if T is normal and every eigenvalue of T is real.

Proof.

Suppose that T is normal and that its eigenvalues are real. Let T=λ1T1+λ2T2++λkTk be the spectral decomposition of T. Then

T* = λ ¯ 1 T 1 + λ ¯ 2 T 2 + + λ ¯ k T k = λ 1 T 1 + λ 2 T 2 + + λ k T k = T .

Hence T is self-adjoint.

Now suppose that T is self-adjoint and hence normal. That its eigenvalues are real has been proved in the lemma to Theorem 6.17 (p. 371).

Corollary 4.

Let T be as in the spectral theorem with spectral decomposition T=λ1T1+λ2T2++λkTk. Then each Tj is a polynomial in T.

Proof.

Choose a polynomial gj(1jk) such that gj(λi)=δij. Then

g j ( T ) = g j ( λ 1 ) T 1 + g j ( λ 2 ) T 2 + + g j ( λ k ) T k = δ 1 j T 1 + δ 2 j T 2 + + δ k j T k = T j .

Exercises

  1. Label the following statements as true or false. Assume that the underlying inner product spaces are finite-dimensional.

    1. (a) All projections are self-adjoint.

    2. (b) An orthogonal projection is uniquely determined by its range.

    3. (c) Every self-adjoint operator is a linear combination of orthogonal projections.

    4. (d) If T is a projection on W, then T(x) is the vector in W that is closest to x.

    5. (e) Every orthogonal projection is a unitary operator.

  2. Let V=R2, W=span({(1, 2)}), and β be the standard ordered basis for V. Compute [T]β, where T is the orthogonal projection of V on W. Do the same for V=R3 and W=span({(1, 0, 1)}).

  3. For each of the matrices A in Exercise 2 of Section 6.5:

    1. (1) Verify that LA possesses a spectral decomposition.

    2. (2) For each eigenvalue of LA, explicitly define the orthogonal projection on the corresponding eigenspace.

    3. (3) Verify your results using the spectral theorem.

  4. Let W be a finite-dimensional subspace of an inner product space V. Show that if T is the orthogonal projection of V on W, then IT is the orthogonal projection of V on W.

  5. Let T be a linear operator on a finite-dimensional inner product space V.

    1. (a) If T is an orthogonal projection, prove that ||T(x)||||x|| for all xV. Give an example of a projection for which this inequality does not hold. What can be concluded about a projection for which the inequality is actually an equality for all xV?

    2. (b) Suppose that T is a projection such that ||T(x)||||x|| for all xV. Prove that T is an orthogonal projection.

  6. Let T be a normal operator on a finite-dimensional inner product space. Prove that if T is a projection, then T is also an orthogonal projection.

  7. Let T be a normal operator on a finite-dimensional complex inner product space V. Use the spectral decomposition λ1T1+λ2T2++λkTk of T to prove the following results.

    1. (a) If g is a polynomial, then

      g ( T ) = i = 1 k g ( λ i ) T i .
    2. (b) If Tn=T0 for some n, then T=T0.

    3. (c) Let U be a linear operator on V. Then U commutes with T if and only if U commutes with each Ti.

    4. (d) There exists a normal operator U on V such that U2=T.

    5. (e) T is invertible if and only if λi0 for 1ik.

    6. (f) T is a projection if and only if every eigenvalue of T is 1 or 0.

    7. (g) T=T* if and only if every λi is an imaginary number.

  8. Use Corollary 1 of the spectral theorem to show that if T is a normal operator on a complex finite-dimensional inner product space and U is a linear operator that commutes with T, then U commutes with T*.

  9. Referring to Exercise 20 of Section 6.5, prove the following facts about a partial isometry U.

    1. (a) U*U is an orthogonal projection on W.

    2. (b) UU*U=U.

  10. Simultaneous diagonalization. Let U and T be normal operators on a finite-dimensional complex inner product space V such that TU=UT. Prove that there exists an orthonormal basis for V consisting of vectors that are eigenvectors of both T and U. Hint: Use the hint of Exercise 14 of Section 6.4 along with Exercise 8.

  11. Prove (c) of the spectral theorem. Visit goo.gl/utQ9Pb for a solution.