Gradient descent algorithm in machine learning
Machine learning
Machine learning algorithms are powerful methods and techniques which are high in terms of probability, and used to give computers high power to compute the solution for large numbers of datasets.
Computers generally use the previous dataset’s result and then compute for the huge, complex current dataset referencing previous datasets.
Gradient descent algorithm
The gradient descent algorithm is the method that gives the local minima or local maxima for a given curve. With the help of this algorithm, we can find out the points of local minima and maxima and use them in the field of machine learning or deep learning to minimize the value of any function.
The gradient Descent Algorithm scales the next point (by a learning rate), subtractions the resulting value from the current position, and repeatedly determines the next point with the gradient at the present position (makes a step). In order to minimize the function, it subtracts the value (to maximize it would be adding).
For a function, if it is differentiable and convex, then only the gradient descent algorithm will work; otherwise, it will not work.
A function to be differentiable means that in its domain, its derivative should be present. For a function to be convex, its double derivative should always be greater than zero.
For example:
Let
Now its first derivative would be:
And its second derivative would be:
For the above function, the second derivative is two, which is always greater than zero, so it is a strictly convex function.
As we know, in mathematics, we can find out the minima and maxima using differentiation. We find out those points where the first derivative of the curve is zero, and at those points, if the second derivative is positive, then there is a minima, and if the second derivative is negative, then there will be a maxima.
But if the second derivative becomes zero, then that point is neither minima nor maxima and these points are called saddle points. It is the disadvantage of the gradient descent algorithm.
For example:
Let
Now its first derivative is
The first derivative is zero, so it can be the point of local minima or maxima.
Now its second derivative would be:
At X = 0 second derivative is zero, so it is a saddle point.
Working of gradient descent algorithm
Gradient means the slope of the curve. We can evaluate the performance at any time, including the starting position. We will calculate the derivative (or slope) from that starting point, and then we may use a tangent line to determine how steep the slope is. Initially, the slope of the curve is quite steep, but after some time, when other parameters have been introduced, the angle of slope starts reducing until the lowest point of the curve, which is also called the point of convergence.
As in linear regression, our target is to get the line of best fit. The same in the gradient descent algorithm is to reduce the cost of any function as much as possible or reduce the error so much between expected output and actual output. It requires two parameters: the first one is the direction and the second one is the learning rate. These variables impact how future iterations' partial derivative computations are calculated, enabling it to gradually approach the local or global minimum (i.e., point of convergence).
There are the following types of gradient descent algorithms:
1. Batch gradient descent
In batch gradient descent in the training data, it adds the error of each point after evaluation. A training period is a name given to this procedure.
Even while this batching improves computing performance, it can still take a while to analyze big training datasets because all of the data must still be stored in memory. Although batch gradient descent frequently finds the local minimum as opposed to the global one, it also typically yields a steady error gradient and convergence.
2. Stochastic gradient descent
For each sample in the dataset, stochastic gradient descent (SGD) performs a training epoch and updates the parameters one at a time. They are simpler to store in memory since you only need to remember one training example. When compared to batch gradient descent, these frequent updates may provide greater detail and speed, but they may also reduce computational efficiency. Although its frequent updates might lead to noisy gradients, they can also be advantageous for avoiding local minima and locating global ones.
3. Mini-batch gradient descent
This gradient descent is generally combined with the above two batches and stochastic gradient descent. Basically, our input dataset on which we will train our machine learning model will be divided into batches, and on each batch, updates are run. This method creates a balance between stochastic gradient descent's speed and batch gradient descent's computing effectiveness.