What is the intuitive concept of the partial least square in regression/machine learning?

Answer by Abhilash Kantamneni:

Consider a problem where you have a bunch of factors that can predict some outcomes (or responses). For example, let us say you were trying to predict whether a particular set of wines are more likely be paired with meat, or dessert. You are given just two factors: Price and Sugar.

If these variables are independent, in other words, if the Price of a wine is not collinear with the amount of sugar in the wine, then you can use a simple multiple regression model (Figure from Page on unt.edu). You will probably end up with some model that effectively boils down to "higher the sugar, more likely to be used for dessert, price not likely to be a factor".

 Using this example Page on utdallas.edu as an illustration, what if the factors that determined wine pairings were Price, Acidity, Sugar and Alcohol?  As the number of factors increase when compared to the number of observations, the assumption that the factors are independent and non-collinear might not always hold.

In other words, you the wine shopper might be thinking "I'd probably use a sugary wine for dessert, as long as it is not too acidic, but maybe just a little alcoholic, but if it is too expensive, I'll just skip it etc….." etc.  As you build this predictor for wine pairings, you are not just concerned with how much each of these factors independently influence the outcomes, but what combination of these factors influence the outcomes and how.

This is where Principal Components Regression helps: an indirect modeling technique that maps the factors directly to outcomes by first performing a Principle Component Analysis directly to the factors, and then applying the Multilinear Regression.  (source : Page on ucla.edu)

PLS takes the idea further and maximises the factors that correlate to responses. PLS and PCR are almost similar, but you can think of PLS as a pruned PCR. (Read Partial Least Squares Program).

Again, from Page on unt.edu , the advantages of PLS/PCR when compared to Multi-Regression are:

  • Multiple outcomes
  • Dimension reduction
  • 'Closer to reality' modeling by getting rid of non-colinearity assumptions.

What is the intuitive concept of the partial least square in regression/machine learning?


Leave a comment

Filed under Life

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s