Feature Selection in Machine Learning

Feature selection

Feature selection is the methodology of selecting some particular dataset instead of all the datasets, which is relevant to reducing the noise of machine learning models. In the machine learning model, we will select those features or datasets which can solve our problem or which can give the optimal solution we need.

What goes in comes out is a basic tenet of machine learning models. We anticipate that the outcome will be garbage if we input junk into our model. Garbage, in this context, refers to noise in our data.

We gather vast amounts of data to train a model in order to improve machine learning. Typically, a sizable amount of the data obtained is noise, and some of the columns in our dataset may not have a major impact on how well our model performs. In addition, training a model can take longer when there is a lot of data available. The model can become erroneous as a result of learning from this unimportant data.

Proper data scientists can be distinguished from the rest by their feature selection. Why do some people win competitions with models that are faster and more accurate when the model and computational resources are the same? Feature selection is the solution. In addition to selecting the appropriate model for our data, we also need to select the appropriate data for our model.

There are generally two types of feature selection methods:

1. Supervised models

In supervised models, we can choose the output labels as a reference to pick what features and which features should not be picked. Based on the output variables, we can reduce the dataset and choose only those features which will give the optimal solution.

There are the following ways of implementing the supervised models:

i) Filter method

The filter method is the way in which we pick and not pick the features based on the relationship with the output. The correlation function is used to determine the relation between output and input variables, and the positive and negative value of correlation determines which feature will be picked and which will be dropped.

Examples: chi-square test, Information gain, Fisher’s score, etc.

• Chi-square test: For categorical features in a dataset, the Chi-square test is employed. Each feature's relationship to the target is analyzed using Chi-square, and the desired number of features with the highest Chi-square scores is chosen. The variables must be categorical, sampled separately, and have values with an anticipated frequency larger than 5 in order to properly use the chi-squared to evaluate the relationship between different features in the dataset and the target variable.

• Information gain: The entropy decrease caused by a dataset modification is calculated using information gain. By assessing each variable's information gain in relation to the target variable, it can be used for feature selection.

• Fisher’s score: One of the most popular supervised feature selection techniques is the Fisher score. The algorithm which we will employ returns the ranks of the variables based on the fisher’s score in descending order. The variables can then be chosen based on the situation.

ii). Wrapper method

This method is based on a greedy approach. Basically, in a greedy approach, we try all possible ways and get the best solution that is optimal. In the same way, we divide our input dataset into many subsets, and we try to train the machine learning model from each subset. And for each subset, we try to add or subtract features to get the closest optimal solution .so. In this way, we get the subset that has a lesser number of variables and a highly optimized solution.

Example: backward elimination, forward selection, etc.

iii). Intrinsic method

This method is a combination of both filter methods and wrapper methods. It selects the subset which gives the optimal solution and which is highly related to the output.

Example: the lasso and ridge regression

2. Unsupervised models

Unsupervised models are the technique that does not use the output label for feature selection. It is just the opposite of supervised learning.