Dimensionality Reduction in machine learning

What is Dimensionality?

This word is generated from the word ‘dimension'. So as we know, machine learning is entirely dependent on a vast dataset, and there are a lot of variables as input into the dataset. So dimensionality describes the number of input variables into a dataset. If the number of input variables is high, then dimensionality will be more.

Dimensionality is directly proportional to the complexity of achieving a solution in machine learning. So we will try to reduce the number of input variables from our dataset without affecting the result or just maintaining the accuracy of the dataset, and this is called dimensionality reduction. There are a lot of methods used in machine learning to reduce the number of variables and redundant variables to make a complex problem simple.

Example:

In Gmail, we have a spam folder that will store the emails which are spam or irrelevant mail. Spam mail can be classified based on different variables like sender’s email address, sender’s location, the pattern of email, title, etc. these all properties are input variables for our machine learning model. Still, many of these properties are overlapping or redundant. So to reduce the redundancy without affecting the performance, we use the dimensionality reduction technique.

Another reason is that if any problem has 3D input variables then we will reduce them into 1D or 2D variables to visualize better, and we will get a more straightforward approach to solve it.

There are basically two components for dimensionality reduction:

1. Feature selection

In this part, we choose small subsets of the dataset and try to train the model with a smaller subset with a lesser number of variables.

2. Feature extraction

In this section, we try to rescue the dimension from higher to lower numbers to reduce the number of random variables in the dataset.

Reducing the number of random variables may be linear or nonlinear, and we have a lot of methods for dimensionality reduction. Some of them are described below:

1. Principal Component Analysis (PCA)

Karl Pearson proposed this approach. It works under the assumption that the variance of the data in the lower dimensional space should be at its highest while the data in a higher dimensional space is mapped to data in a lower dimensional space.

Following are the step-by-step points for this method:

• Create the data's covariance matrix.
• Establish the data's covariance matrix.
• A significant portion of the variance of the original data is reconstructed using eigenvectors corresponding to the greatest eigenvalues.

2. Manifold learning

Dimensionality reduction can also be accomplished using methods from high-dimensional statistics. These methods, which are also known as "manifold learning," are used to project high-dimensional data into a low-dimensional space, frequently for data visualization purposes. The projection is intended to preserve the most important relationships or structures in the data while also producing a low-dimensional representation of the dataset.

The projection's features frequently bear little resemblance to the original columns; for instance, they lack column names, which can be misleading to newcomers.

Examples:

• Multidimensional scaling(MDS)
• Kohonen self-organizing map(SOM)
• Sammons mapping
• T-distributed Stochastic neighbor embedding(t-SNE)

3. Autoencoder methods

Dimensionality reduction can be accomplished via deep learning neural networks. Autoencoders are a well-liked method. In order to accomplish this, one must frame a self-supervised learning problem where the model must accurately recreate the input.

A type of unsupervised neural network called an auto-encoder is employed for feature extraction and dimensionality reduction. An auto-encoder is a kind of feed forward neural network that has been trained to predict the input itself, to put it more exactly.

The output from the bottleneck is used directly as the input's reduced dimensionality after training, with the decoder being abandoned. Another model, not necessarily a neural network model, can be fed inputs altered by this encoder. It can be difficult to comprehend the encoder output because, like other projection methods, there is no direct connection between the bottleneck output and the initial input variables.

Although we use dimensional reduction methods to reduce the complexity but there are also some disadvantages of using this technique:

• There is always a risk of data loss in dimensionality reduction.
• Linear relationships between variables are frequently discovered by PCA, which is not always desired.
• PCA can be failed if the mean and variance can't determine the characterization of the dataset
• We may not be aware of the ideal number of principal components to maintain, but in actuality, several guidelines are used.