I'll elaborate a bit on DJ Patil's answer with some examples.

1. Suppose you have a real, symmetric matrix M. Then M transforms the unit sphere x T x=1 x^T x = 1 into an ellipsoid where:

M's eigenvectors point in the same direction as the axes of the ellipsoidM's eigenvalues squared give you the length of each radius[I assumed M was real and symmetric so that it has real eigenvectors and eigenvalues.]

2. Suppose you have a bunch of datapoints centered at the origin. Stick each datapoint into the column of a matrix M. Then the eigenvectors of the *covariance* matrix M T M M^T M tell you the shape of the data: the eigenvector with the largest eigenvalue tells you the "first principal component" of the data, i.e., the direction in which the datapoints have maximum variance/"maximum information"; the eigenvector with the second-largest eigenvalue tells you the second principal component, which is the direction among all directions orthogonal to the first principal component that maximizes variance; and so on. Moreover, the eigenvalues tell you the variance along each direction.

This picture might help:

http://en.wikipedia.org/wiki/Fil…

In the picture, the vector pointing to the northeast is the first principal component/eigenvector of the covariance matrix, and the other vector is the second principal component/eigenvector.

3. Suppose your matrix M is a ratings matrix, say where the entry in row i column j tells you how much user i liked movie j. Then the eigenvectors of MM T M M^T and M T M M^T M decompose ratings into linear combinations of user-movie features.

That is, the eigenvectors MM T M M^T correspond to features that explain user preferences (for example, one eigenvector might correspond to "horror" movies, and the ith component of this eigenvector would tell you how much user i likes horror movies), and the eigenvalues tell you how important each feature is. Similarly, the eigenvectors of M T M M^T M correspond to features that explain movies (again, one eigenvector might be a horror eigenvector, and the ith component of this eigenvector would say how horrific movie i is).

Answer by Edwin Chen:

I'll elaborate a bit on DJ Patil's answer with some examples.

1. Suppose you have a real, symmetric matrix M. Then M transforms the unit sphere [math]x^T x = 1[/math] into an ellipsoid where:

- M's eigenvectors point in the same direction as the axes of the ellipsoid
- M's eigenvalues squared give you the length of each radius
[I assumed M was real and symmetric so that it has real eigenvectors and eigenvalues.]

2. Suppose you have a bunch of datapoints centered at the origin. Stick each datapoint into the column of a matrix M. Then the eigenvectors of the *covariance* matrix [math]M^T M[/math] tell you the shape of the data: the eigenvector with the largest eigenvalue tells you the "first principal component" of the data, i.e., the direction in which the datapoints have maximum variance/"maximum information"; the eigenvector with the second-largest eigenvalue tells you the second principal component, which is the direction among all directions orthogonal to the first principal component that maximizes variance; and so on. Moreover, the eigenvalues tell you the variance along each direction.

This picture might help:

http://en.wikipedia.org/wiki/File:GaussianScatterPCA.png

In the picture, the vector pointing to the northeast is the first principal component/eigenvector of the covariance matrix, and the other vector is the second principal component/eigenvector.3. Suppose your matrix M is a ratings matrix, say where the entry in row i column j tells you how much user i liked movie j. Then the eigenvectors of [math]M M^T[/math] and [math]M^T M[/math] decompose ratings into linear combinations of user-movie features.

That is, the eigenvectors [math]M M^T[/math] correspond to features that explain user preferences (for example, one eigenvector might correspond to "horror" movies, and the ith component of this eigenvector would tell you how much user i likes horror movies), and the eigenvalues tell you how important each feature is. Similarly, the eigenvectors of [math]M^T M[/math] correspond to features that explain movies (again, one eigenvector might be a horror eigenvector, and the ith component of this eigenvector would say how horrific movie i is).

What do eigenvalues and eigenvectors represent intuitively?