Orthonormal Bases and the Projection Theorem

The use of orthonormal bases simplifies almost everything concerning inner product spaces, and for infinite-dimensional expansions, orthonormal bases are even more useful. Gram-Schmidt procedure starting from an arbitrary basis \({s_1, . . . , s_n}\) for an n-dimensional inner product subspace \(\mathcal S\), generates an orthonormal basis for \(\mathcal S\). The infinite dimensional projection theorem can provide simple and intuitive proofs and interpretations of limiting arguments and the approximations suggested by those limits.

Finite-dimensional Projections

Assume \(\mathcal V\) is an inner product space, A vector \(\phi \in \mathcal V\) is normalized if \(\|\phi\| = 1\).
The projection \(v_{|\phi} = \langle v, \phi \rangle \phi\) for \(\|\phi\| = 1\).
An orthonormal set \(\{\phi_j\}\) is a set such that \(\langle \phi_j, \phi_k \rangle = \delta_{jk}\).
If \(\{v_j\}\) is orthogonal set, then \(\{\phi_j\}\) is an orthonormal set where \(\phi_j = v_j/\|v_j\|\).

Projection theorem

Assume that \(\{\phi_1, . . . , \phi_n\}\) is an orthonormal basis for an n-dimensional subspace \(\mathcal S \subset \mathcal V\). For each \(v \in V\), there is a unique \(v_{| \mathcal S} \in \mathcal S\) such that \(\langle v - v_{| \mathcal S}, s \rangle = 0\) for all \(s \in \mathcal S\). Furthermore,

$$v_{|s} = \sum_j \langle v, \phi_j \rangle \phi_j$$

.
Proof outline: Let \(v_{|S} = \sum_i \alpha_i\phi_i\). Find the conditions on \(\alpha_1, . . . , \alpha_n\) such that \(v − v_{|S}\) is orthogonal to each \(\phi_i\).

$$0 = \langle v - \sum_i \alpha_i\phi_i, \phi_j \rangle = \langle v, \phi_j \rangle - \alpha_j$$

For \(v \in \mathcal S, v = \sum_j \alpha_j \phi_j, {\phi_j}\) orthonormal basis of \(\mathcal S\),

$$\|v\|^2 = \langle v, \sum_j \alpha_j\phi_j\rangle = \sum \alpha_j^* \langle v, \phi_j \rangle = \sum_j |\alpha_j|^2$$

Bessel’s inequality

Let \(\mathcal S \subseteq \mathcal V\) be the subspace spanned by the set of orthonormal vectors \({\phi_1, . . . , \phi_n}\). For any \(v \in V\)

$$0 \le \sum_{j=1}^n |\langle v, \phi_j \rangle|^2 \le \|v\|^2$$

is the key to understanding the convergence of orthonormal expansions.

LS error property

\(v_{|s}\) is the choice for s that yields the Least Square Error(LS) or Minimum Square Error(MSE).
The projection \(v_{|\mathcal S}\) is the unique closest vector in \(S\) to \(v\); i.e., for all \(s \in \mathcal S\),

$$\|v - v_{|\mathcal S}\|^2 \le \|v - s\|^2$$

The individual basis functions themselves have a trivial vector representation; namely \(\phi_n(t)\) is represented by \(\phi_n = [0 0 , ..., 1 , ..., 0]^T\), In effect, \(x_{in} = \langle x_i \phi_n \rangle\) is the projection of the ith modulated waveform on the \(n^{th}\) basis function.

Gram-Schmidt orthonormalization

Given basis \(s_1, . . . , s_n\) for an inner product subspace, find an orthonormal basis.
\(\phi_1 = s_1/\|s_1\|\) is an orthonormal basis for subspace \(\mathcal S_1\) generated by \(s_1\).
Given orthonormal basis \(\phi_1, . . . , \phi_k\) of subspace \(\mathcal S_k\) generated by \(s_1, . . . , s_k\), project \(s_{k+1}\) onto \(\mathcal S_k\).

$$\phi_{k+1} = {(s_{k+1})_{\bot \mathcal S_k} \over \|(s_{k+1})_{\bot \mathcal S_k}\| }$$

The Gram-Schmidt algorithm gives a simple construction of the q’s from the a’s in factorization of A = QR (orthonormal columns times upper triangular).

Orthonormal expansions in \(L^2\)

For \(L^2\), the projection theorem can be extended to a countably infinite dimension.

Infinite-dimensional projection

Let \(\{\phi_m, 1 \le m \lt \infty\}\) be a set of orthonormal functions, and let \(v\) be any \(L^2\) vector. Then there is a unique \(L^2\) vector \(u\) such that \(v − u\) is orthogonal to each \(\phi_m\) and

$$\lim\limits_{n \to \infty}\|u - \sum\limits_{m=1}^n\langle v, \phi_m \rangle \phi_m\| = 0$$

Outline of proof: Let \(\mathcal S_n\) be subspace spanned by \(\phi_1, . . . , \phi_n\).

$$v_{|\mathcal S_n} = \sum\limits_{k=1}^n \alpha_k \phi_k = \sum\limits_{k=1}^n \langle v, \phi_k \rangle \phi_k$$ $$\|v_{|\mathcal S_m} - v_{|\mathcal S_n}\|^2 = \sum\limits_{k=n}^m |\alpha_k|^2 \to 0$$

\(v_{|\mathcal S_n}\) forms a Cauchy sequence. By the Riesz-Fischer theorem(\(L^2\) waveforms has an \(L^2\) limit), \(l.i.m. v_{\mathcal |S_n} = u\) exists.
This shows that the fourier series converges in \(L^2\).

Reference:

1. Robert Gallager. (2006). 6.450 Digital Communication. MIT OpenCourseWare (http://ocw.mit.edu/)
2. Robert G.Gallager. (2009). Principles of Digital Communication. (New York: Cambridge University Press).
3. Gilbert Strang. (2007 ). Computational Science and Engineering. (Wellesley-Cambridge Press).

A good person! More about the author →