Principal Components Analysis (contd.)

Let A be a kxk matrix and l be an eigenvalue of A. If X is a non zero vector such that:

AX = l X

then X is an eigenvector of A associated with the eigenvalue l .

Consider A = {1 0, 1 3} then l ₁ = 1 and l ₂ = 3. For l ₁ = 1, X = {-2, 1} is an eigenvector and for l ₂ = 3, X = {0, 1} is an eigenvector.

Let A be a square symmetric matrix. A can be written as the product of two matrices P and L so that:

A = P L P^T

where P is a kxk, symmetric, orthogonal matrix containing the eigenvectors of A and L is a pxp diagonal matrix containing the eigenvalues of A. This is known as the Spectral Decomposition of A.

Principal Components of X:

Let X be a matrix of the instances of x₁,…,x_k. Let S be the kxk covariance matrix of X. Since S is a square symmetric then its Spectral Decomposition is:

S = P L P^T

Now, the principal components of X are given by the matrix C so that:

C = X P

Let S _c be the kxk covariance matrix of C then:

S _c = (X P)^T(X P) = P^T X^T X P = P^T S P

Since S = P L P^T then:

S _c = P^T P L P^T P = L

Thus the covariance matrix of C is a pxp diagonal matrix containing the eigenvalues of X. So, if c₁,…,c_k are the variables denoting the columns of C then they are uncorrelated with variances l ₁,…, l _k since S _c = L .

Now, the total variance of X is the same as the total variance of C. To see this, examine the trace of the covariance matrix S (i.e. tr(S )). Remember that tr(S ) is the sum of the diagonal elements of S where the diagonal elements are s _j; j=1,..,k.

Now:

tr(S ) = tr(P L P^T) = tr(P^T P L ) = tr(L )= tr(S _c)

Hence, the total variance of X is equal to the total variance of C.

Thus, to find the principal components of X we simply need to determine the Spectral Decomposition of the covariance matrix of X (i.e. S ) and then we can determine C by computing C = X P.