Dimension Reduction- Feature Selection and Feature Extraction

We often ask multiple questions to find insightful inputs when measuring fuzzy concepts such as “‘service quality”, “consumer trust” or “customer loyalty”. However, there are too many variables, including those that are unimportant or unrelated, that cause dimensionality problem. Therefore, data reduction is a necessary way in this kind of marketing research. It can be divided into two parts: feature selection and feature extraction.

Feature selection and feature extraction reduce the number of variables by obtaining a set of principal variables. The algorithm behind them helps us choose the relevant and significant variables automatically. This not only resolves overfitting issue but also avoids collinearity issue and improve model efficiency.

Feature Selection

Feature selection is the process of selecting a subset of relevant and important variables from the original variables for use in model construction.

It is often used in domains where the data contains some variables that are either redundant or irrelevant and comparatively few samples. In this condition, these variables can be removed without incurring too much loss of information.

Feature selection ranks the existing attributes according to their predictive significance, so we are able to know what the chosen features are. Its techniques include stepwise selection, regularization, etc.

Feature Extraction

Feature extraction projects a dataset with higher dimensionality onto a smaller number of dimensions by selecting and/ or combining the existing variables into a new set of features that captures most of the useful information.

It is used to identify key components in the data for coding by learning from the coding of the original dataset to derive new ones. Thus, the new variables created still accurately and completely describing the original dataset, however, sometimes they are not easily interpretable.

The transformed attributes by feature extraction are linear combinations of the original attributes. Its techniques include factor analysis (FA), principal component analysis (PCA), etc.

Note After Learning

Sometimes it's hard to decide whether using feature selection or feature extraction. Here’s what I think: if the only thing you want to achieve is dimensionality reduction in an existing dataset, it doesn't matter which one you adopt.

But if you need to know the identity, composition, and interpretation of the selected important features, only feature selection works.

Another situation is when you know nothing about the data, for example, there’s no data dictionary or the data is not in an understandable format, then applying feature extraction to the data is able to explain more.

Bonus

You can apply feature extraction on the given data to extract features and then apply feature selection with respect to the target variable to select the subset which is can help in making a good result — Stackexchange

About me

Hey, I’m Kelly, a business analytics graduate student with journalism and communication background who likes to share the life of exploring data and interesting findings. If you have any questions, feel free to contact me at kelly.szutu@gmail.com