What do eigenvalues and eigenvectors represent intuitively?

I'll elaborate a bit on DJ Patil's answer with some examples.

1. Suppose you have a real, symmetric matrix M. Then M transforms the unit sphere x T x=1  x^T x = 1  into an ellipsoid where:
M's eigenvectors point in the same direction as the axes of the ellipsoidM's eigenvalues squared give you the length of each radius[I assumed M was real and symmetric so that it has real eigenvectors and eigenvalues.]

2. Suppose you have a bunch of datapoints centered at the origin. Stick each datapoint into the column of a matrix M. Then the eigenvectors of the *covariance* matrix M T M  M^T M  tell you the shape of the data: the eigenvector with the largest eigenvalue tells you the "first principal component" of the data, i.e., the direction in which the datapoints have maximum variance/"maximum information"; the eigenvector with the second-largest eigenvalue tells you the second principal component, which is the direction among all directions orthogonal to the first principal component that maximizes variance; and so on. Moreover, the eigenvalues tell you the variance along each direction.

This picture might help:
http://en.wikipedia.org/wiki/Fil…
In the picture, the vector pointing to the northeast is the first principal component/eigenvector of the covariance matrix, and the other vector is the second principal component/eigenvector.

3. Suppose your matrix M is a ratings matrix, say where the entry in row i column j tells you how much user i liked movie j. Then the eigenvectors of MM T   M M^T  and M T M  M^T M  decompose ratings into linear combinations of user-movie features.

That is, the eigenvectors MM T   M M^T  correspond to features that explain user preferences (for example, one eigenvector might correspond to "horror" movies, and the ith component of this eigenvector would tell you how much user i likes horror movies), and the eigenvalues tell you how important each feature is. Similarly, the eigenvectors of M T M  M^T M  correspond to features that explain movies (again, one eigenvector might be a horror eigenvector, and the ith component of this eigenvector would say how horrific movie i is).

Answer by Edwin Chen:

I'll elaborate a bit on DJ Patil's answer with some examples.

1. Suppose you have a real, symmetric matrix M. Then M transforms the unit sphere [math]x^T x = 1[/math] into an ellipsoid where:

  • M's eigenvectors point in the same direction as the axes of the ellipsoid
  • M's eigenvalues squared give you the length of each radius

[I assumed M was real and symmetric so that it has real eigenvectors and eigenvalues.]

2. Suppose you have a bunch of datapoints centered at the origin. Stick each datapoint into the column of a matrix M. Then the eigenvectors of the *covariance* matrix [math]M^T M[/math] tell you the shape of the data: the eigenvector with the largest eigenvalue tells you the "first principal component" of the data, i.e., the direction in which the datapoints have maximum variance/"maximum information"; the eigenvector with the second-largest eigenvalue tells you the second principal component, which is the direction among all directions orthogonal to the first principal component that maximizes variance; and so on. Moreover, the eigenvalues tell you the variance along each direction.

This picture might help:
http://en.wikipedia.org/wiki/File:GaussianScatterPCA.png
In the picture, the vector pointing to the northeast is the first principal component/eigenvector of the covariance matrix, and the other vector is the second principal component/eigenvector.

3. Suppose your matrix M is a ratings matrix, say where the entry in row i column j tells you how much user i liked movie j. Then the eigenvectors of [math]M M^T[/math] and [math]M^T M[/math] decompose ratings into linear combinations of user-movie features.

That is, the eigenvectors [math]M M^T[/math] correspond to features that explain user preferences (for example, one eigenvector might correspond to "horror" movies, and the ith component of this eigenvector would tell you how much user i likes horror movies), and the eigenvalues tell you how important each feature is. Similarly, the eigenvectors of [math]M^T M[/math] correspond to features that explain movies (again, one eigenvector might be a horror eigenvector, and the ith component of this eigenvector would say how horrific movie i is).

What do eigenvalues and eigenvectors represent intuitively?

Advertisements

Leave a comment

Filed under Life

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s