# 6.2 The Gram-Schmidt Orthogonalization Process and Orthogonal Complements

In previous chapters, we have seen the special role of the standard ordered bases for  and . The special properties of these bases stem from the fact that the basis vectors form an orthonormal set. Just as bases are the building blocks of vector spaces, bases that are also orthonormal sets are the building blocks of inner product spaces. We now name such bases.

# Definition.

Let V be an inner product space. A subset of V is an orthonormal basis for V if it is an ordered basis that is orthonormal.

# Example 1

The standard ordered basis for  is an orthonormal basis for .

# Example 2

The set



is an orthonormal basis for .

The next theorem and its corollaries illustrate why orthonormal sets and, in particular, orthonormal bases are so important.

# Theorem 6.3.

Let V be an inner product space and  be an orthogonal subset of V consisting of nonzero vectors. If , then



# Proof.

Write , where . Then, for ,

we have



So , and the result follows.

The next corollary follows immediately from Theorem 6.3.

# Corollary 1.

If, in addition to the hypotheses of Theorem 6.3, S is orthonormal and , then



If V possesses a finite orthonormal basis, then Corollary 1 allows us to compute the coefficients in a linear combination very easily. (See Example 3.)

# Corollary 2.

Let V be an inner product space, and let S be an orthogonal subset of V consisting of nonzero vectors. Then S is linearly independent.

# Proof.

Suppose that  and



As in the proof of Theorem 6.3 with , we have  for all j. So S is linearly independent.

# Example 3

By Corollary 2, the orthonormal set



obtained in Example 8 of Section 6.1 is an orthonormal basis for . Let . The coefficients given by Corollary 1 to Theorem 6.3 that express x as a linear combination of the basis vectors are



and



As a check, we have



Corollary 2 tells us that the vector space H in Section 6.1 contains an infinite linearly independent set, and hence H is not a finite-dimensional vector space.

Of course, we have not yet shown that every finite-dimensional inner product space possesses an orthonormal basis. The next theorem takes us most of the way in obtaining this result. It tells us how to construct an orthogonal set from a linearly independent set of vectors in such a way that both sets generate the same subspace.

Before stating this theorem, let us consider a simple case. Suppose that  is a linearly independent subset of an inner product space (and hence a basis for some two-dimensional subspace). We want to construct an orthogonal set from  that spans the same subspace. Figure 6.1 suggests that the set , where  and , has this property if c is chosen so that  is orthogonal to .

To find c, we need only solve the following equation:



So



Thus



The next theorem shows us that this process can be extended to any finite linearly independent subset.

# Theorem 6.4.

Let V be an inner product space and  be a linearly independent subset of V. Define , where  and

 (1)

Then  is an orthogonal set of nonzero vectors such that .

# Proof.

The proof is by mathematical induction on n, the number of vectors in S. For  let . If , then the theorem is proved by taking  i.e., . Assume then that the set  with the desired properties has been constructed by the repeated use of (1). We show that the set  also has the desired properties, where  is obtained from  by (1). If , then (1) implies that , which contradicts the assumption that  is linearly independent. For , it follows from (1) that



since  if  by the induction assumption that  is orthogonal. Hence  is an orthogonal set of nonzero vectors. Now, by (1), we have that . But by Corollary 2 to Theorem 6.3,  is linearly independent; so . Therefore .

The construction of  by the use of Theorem 6.4 is called the Gram—Schmidt process.

# Example 4

In , let  and . Then  is linearly independent. We use the Gram-Schmidt process to compute the orthogonal vectors  and , and then we normalize these vectors to obtain an orthonormal set.

Take . Then



Finally,



These vectors can be normalized to obtain the orthonormal basis , where



and



# Example 5

Let  with the inner product , and consider the subspace  with the standard ordered basis . We use the Gram-Schmidt process to replace  by an orthogonal basis  for , and then use this orthogonal basis to obtain an orthonormal basis for .

Take . Then , and . Thus



Furthermore,



Therefore



We conclude that  is an orthogonal basis for .

To obtain an orthonormal basis, we normalize , and  to obtain



and similarly,



Thus  is the desired orthonormal basis for .

Continuing to apply the Gram-Schmidt orthogonalization process to the basis  for P(R), we obtain an orthogonal basis . For each n, the polynomial  is called the kth Legendre polynomial. The first three Legendre polynomials are 1, x and . The set of Legendre polynomials is also an orthogonal basis for P(R).

The following result gives us a simple method of representing a vector as a linear combination of the vectors in an orthonormal basis.

# Theorem 6.5.

Let V be a nonzero finite-dimensional inner product space. Then V has an orthonormal basis . Furthermore, if  and , then



# Proof.

Let  be an ordered basis for V. Apply Theorem 6.4 to obtain an orthogonal set  of nonzero vectors with . By normalizing each vector in , we obtain an orthonormal set  that generates V. By Corollary 2 to Theorem 6.3,  is linearly independent; therefore  is an orthonormal basis for V. The remainder of the theorem follows from Corollary 1 to Theorem 6.3.

# Example 6

We use Theorem 6.5 to represent the polynomial  as a linear combination of the vectors in the orthonormal basis  for  obtained in Example 5. Observe that



and



Therefore .

Theorem 6.5 gives us a simple method for computing the entries of the matrix representation of a linear operator with respect to an orthonormal basis.

# Corollary.

Let V be a finite-dimensional inner product space with an orthonormal basis . Let T be a linear operator on V, and let . Then for any i and j, .

Proof. From Theorem 6.5, we have



Hence .

The scalars  given in Theorem 6.5 have been studied extensively for special inner product spaces. Although the vectors  were chosen from an orthonormal basis, we introduce a terminology associated with orthonormal sets  in more general inner product spaces.

# Definition.

Let  be an orthonormal subset (possibly infinite) of an inner product space V, and let . We define the Fourier coefficients of x relative to  to be the scalars , where .

In the first half of the 19th century, the French mathematician Jean Baptiste Fourier was associated with the study of the scalars



or in the complex case,



for a function f. In the context of Example 9 of Section 6.1, we see that , where  that is,  is the nth Fourier coefficient for a continuous function  relative to S. The coefficients  are the “classical” Fourier coefficients of a function, and the literature concerning their behavior is extensive. We learn more about Fourier coefficients in the remainder of this chapter.

# Example 7

Let . In Example 9 of Section 6.1, S was shown to be an orthonormal set in H. We compute the Fourier coefficients of  relative to S. Using integration by parts, we have, for ,



and, for ,



As a result of these computations, and using Exercise 16 of this section, we obtain an upper bound for the sum of a special infinite series as follows:



for every k. Now, using the fact that , we obtain



or



Because this inequality holds for all k, we may let  to obtain



Additional results may be produced by replacing f by other functions.

We are now ready to proceed with the concept of an orthogonal complement.

# Definition.

Let S be a nonempty subset of an inner product space V. We define  (read “S perp”) to be the set of all vectors in V that are orthogonal to every vector in S; that is, . The set  is called the orthogonal complement of S.

It is easily seen that  is a subspace of V for any subset S of V.

# Example 8

The reader should verify that  and  for any inner product space V.

# Example 9

If  and , then  equals the xy-plane (see Exercise 5).

Exercise 18 provides an interesting example of an orthogonal complement in an infinite-dimensional inner product space.

Consider the problem in  of finding the distance from a point P to a plane W. (See Figure 6.2.) Problems of this type arise in many settings. If we let y be the vector determined by 0 and P, we may restate the problem as follows: Determine the vector u in W that is “closest” to y. The desired distance is clearly given by . Notice from the figure that the vector  is orthogonal to every vector in W, and so .

The next result presents a practical method of finding u in the case that W is a finite-dimensional subspace of an inner product space.

# Theorem 6.6.

Let W be a finite-dimensional subspace of an inner product space V, and let . Then there exist unique vectors  and  such that . Furthermore, if  is an orthonormal basis for W, then



# Proof.

Let  be an orthonormal basis for W, let u be as defined in the preceding equation, and let . Clearly  and .

To show that , it suffices to show, by Exercise 7, that z is orthogonal to each . For any j, we have



To show uniqueness of u and z, suppose that , where  and . Then . Therefore  and .

# Corollary.

In the notation of Theorem 6.6, the vector u is the unique vector in W that is “closest” to y; that is, for any , and this inequality is an equality if and only if .

Proof.

As in Theorem 6.6, we have that , where . Let . Then  is orthogonal to z, so, by Exercise 10 of Section 6.1, we have



Now suppose that . Then the inequality above becomes an equality, and therefore . It follows that , and hence . The proof of the converse is obvious.

The vector u in the corollary is called the orthogonal projection of y on W. We will see the importance of orthogonal projections of vectors in the application to least squares in Section 6.3.

# Example 10

Let  with the inner product



We compute the orthogonal projection  of  on .

By Example 5,



is an orthonormal basis for . For these vectors, we have



and



Hence



It was shown (Corollary 2 to the replacement theorem, p. 48) that any linearly independent set in a finite-dimensional vector space can be extended to a basis. The next theorem provides an interesting analog for an orthonormal subset of a finite-dimensional inner product space.

# Theorem 6.7.

Suppose that  is an orthonormal set in an n-dimensional inner product space V. Then

1. (a) S can be extended to an orthonormal basis  for V.

2. (b) If , then  is an orthonormal basis for  (using the preceding notation).

3. (c) If W is any subspace of V, then .

# Proof.

(a) By Corollary 2 to the replacement theorem (p. 48), S can be extended to an ordered basis  for V. Now apply the Gram-Schmidt process to . The first k vectors resulting from this process are the vectors in S by Exercise 8, and this new set spans V. Normalizing the last  vectors of this set produces an orthonormal set that spans V. The result now follows.

(b) Because  is a subset of a basis, it is linearly independent. Since  is clearly a subset of , we need only show that it spans . Note that, for any , we have



If , then  for . Therefore



(c) Let W be a subspace of V. It is a finite-dimensional inner product space because V is, and so it has an orthonormal basis . By (a) and (b), we have



# Example 11

Let  in . Then  if and only if  and . So , and therefore . One can deduce the same result by noting that  and, from (c), that .

# Exercises

1. Label the following statements as true or false.

1. (a) The Gram-Schmidt orthogonalization process produces an orthonormal set from an arbitrary linearly independent set.

2. (b) Every nonzero finite-dimensional inner product space has an orthonormal basis.

3. (c) The orthogonal complement of any set is a subspace.

4. (d) If  is a basis for an inner product space V, then for any  the scalars  are the Fourier coefficients of x.

5. (e) An orthonormal basis must be an ordered basis.

6. (f) Every orthogonal set is linearly independent.

7. (g) Every orthonormal set is linearly independent.

2. In each part, apply the Gram–Schmidt process to the given subset S of the inner product space V to obtain an orthogonal basis for span(S). Then normalize the vectors in this basis to obtain an orthonormal basis  for span(S), and compute the Fourier coefficients of the given vector relative to . Finally, use Theorem 6.5 to verify your result.

1. (a)  and 

2. (b)  and 

3. (c)  with the inner product , and 

4. (d) , where  and 

5. (e)  and 

6. (f)  and 

7. (g)  and 

8. (h) , and 

9. (i)  with the inner product  and 

10. (j)  and 

11. (k)  and 

12. (l)  and 

13. (m)  and 

3. In , let



Find the Fourier coefficients of (3, 4) relative to  .

4. Let  in . Compute .

5. Let , where  is a nonzero vector in . Describe  geometrically. Now suppose that  is a linearly independent subset of . Describe  geometrically.

6. Let V be an inner product space, and let W be a finite-dimensional subspace of V. If , prove that there exists  such that , but . Hint: Use Theorem 6.6.

7. Let  be a basis for a subspace W of an inner product space V, and let . Prove that  if and only if  for every .

8. Prove that if  is an orthogonal set of nonzero vectors, then the vectors  derived from the Gram-Schmidt process satisfy  for . Hint: Use mathematical induction.

9. Let  in . Find orthonormal bases for W and .

10. Let W be a finite-dimensional subspace of an inner product space V. Prove that . Using the definition on page 76, prove that there exists a projection T on W along  that satisfies . In addition, prove that  for all . Hint: Use Theorem 6.6 and Exercise 10 of Section 6.1.

11. Let A be an  matrix with complex entries. Prove that  if and only if the rows of A form an orthonormal basis for . Visit goo.gl/iKcC4S for a solution.

12. Prove that for any matrix .

13. Let V be an inner product space, S and  be subsets of V, and W be a finite-dimensional subspace of V. Prove the following results.

1. (a)  implies that .

2. (b)  so .

3. (c) . Hint: Use Exercise 6.

4. (d) . (See the exercises of Section 1.3.)

14. Let  and  be subspaces of a finite-dimensional inner product space. Prove that  and . (See the definition of the sum of subsets of a vector space on page 22.) Hint for the second equation: Apply Exercise 13(c) to the first equation.

15. Let V be a finite-dimensional inner product space over F.

1. (a) Parseval’s Identity. Let  be an orthonormal basis for V. For any  prove that


2. (b) Use (a) to prove that if  is an orthonormal basis for V with inner product , then for any 



where  is the standard inner product on .

1. (a) Bessel’s Inequality. Let V be an inner product space, and let  be an orthonormal subset of V. Prove that for any  we have



Hint: Apply Theorem 6.6 to  and . Then use Exercise 10 of Section 6.1.

2. (b) In the context of (a), prove that Bessel’s inequality is an equality if and only if .

16. Let T be a linear operator on an inner product space V. If  for all , prove that . In fact, prove this result if the equality holds for all x and y in some basis for V.

17. Let . Suppose that  and  denote the subspaces of V consisting of the even and odd functions, respectively. (See Exercise 22 of Section 1.3.) Prove that , where the inner product on V is defined by


18. In each of the following parts, find the orthogonal projection of the given vector on the given subspace W of the inner product space V.

1. (a)  and 

2. (b) , and 

3. (c)  with the inner product  and 

19. In each part of Exercise 19, find the distance from the given vector to the subspace W.

20. Let  with the inner product , and let W be the subspace , viewed as a space of functions. Use the orthonormal basis obtained in Example 5 to compute the “best” (closest) second-degree polynomial approximation of the function  on the interval .

21. Let  with the inner product . Let W be the subspace spanned by the linearly independent set .

1. (a) Find an orthonormal basis for W.

2. (b) Let . Use the orthonormal basis obtained in (a) to obtain the “best” (closest) approximation of h in W.

22. Let V be the vector space defined in Example 5 of Section 1.2, the space of all sequences  in F (where  or ) such that  for only finitely many positive integers n. For , we define . Since all but a finite number of terms of the series are zero, the series converges.

1. (a) Prove that  is an inner product on V, and hence V is an inner product space.

2. (b) For each positive integer n, let  be the sequence defined by , where  is the Kronecker delta. Prove that  is an orthonormal basis for V.

3. (c) Let  and .

1. (i) Prove that , so .

2. (ii) Prove that , and conclude that .

Thus the assumption in Exercise 13(c) that W is finite-dimensional is essential.