Introduction

Rectified linear units (ReLUs) and leaky rectangular units (Leaky ReLUs) are two crucial activation functions that are frequently used in artificial neural networks (ANNs). Their main objective is to inject non-linearity into the network so that it can recognize complex patterns in the data. While both functions are useful, Leaky ReLU addresses a drawback that ReLU has by providing a few advantages over the conventional ReLU function.

Leaky ReLU

An activation function frequently employed in artificial neural networks (ANNs) is the leaky rectified linear unit (Leaky ReLU). It overcomes a drawback of the common Rectified Linear Unit (ReLU) activation function, which makes it a preferred option in some circumstances.

In a standard ReLU, the activation is determined by the function f(x) = max(0,x). Input x directly affects the output if it is positive; if it is negative, the output is 0. ReLU has a flaw known as the "dying ReLU" problem while being computationally efficient and aiding in training by preventing the vanishing gradient problem.
Neurons employing ReLU experience the dying ReLU problem when they stop responding to any input. Negative inputs that cause a neuron's output to continually be 0 during training effectively lead the neuron to stop learning because the gradient for negative inputs is also 0. This detrimentally affects the network's ability to learn and represent information efficiently.
This problem is addressed with Leaky ReLU. Adding a tiny slope for negative inputs significantly alters the ReLU function. This is how the function is defined:

f(x) = x if x > 0
f(x) = ax if x ≤ 0 (where a is a small positive constant)

Leaky ReLU guarantees that neurons with negative inputs continue to receive some learning signal during backpropagation by permitting a tiny, non-zero gradient for negative inputs.
This keeps the neurons from going completely dormant, as was the case with the dying ReLU issue. To balance treating the dying ReLU issue with preventing signal over-dampening, parameter a was chosen to allow control over the activation function's slope for negative inputs.

ReLU Function

Due to its simplicity, speed, and success in training deep neural networks, the Rectified Linear Unit (ReLU) is a frequently employed activation function in Artificial Neural Networks (ANNs). It gives the network non-linearity, enabling it to understand intricate data correlations.

The following is a definition of the ReLU activation function:

f(x) = max(0,x)

The function's output is the maximum of the inputs, x and 0. The output is x if the input x is larger than 0; otherwise, it is 0.

ReLU, however, has significant drawbacks as well:

Dying ReLU Problem: The "dying ReLU" problem is one of ReLU's drawbacks. Negative inputs during training can occasionally result in neurons consistently producing 0. The neuron ceases learning as a result of there being no gradient. Variants such as Leaky ReLU, Parametric ReLU, and Exponential Linear Unit (ELU) were created to remedy this issue.
Unbounded Positive Activation: ReLU has an unlimited positive activation, which permits neurons to generate outputs of any size. Occasionally, this might result in the "exploding gradient" problem during training, especially in deep networks.

Advantages of Leaky ReLU over ReLU:

Prevents Neuron Inactivity: Leaky ReLU's main benefit is that it contributes to solving the dying ReLU issue. In a conventional ReLU, the gradient flowing back during training is 0 if a neuron's output becomes consistently 0 for all inputs (i.e., it turns inactive). Negative inputs are given a slight slope using a leaky ReLU, which permits a slight gradient to flow even for negative values. This keeps neurons with negative inputs from going completely dormant by ensuring they still get a learning signal.
Flexibility in Slope: Leaky ReLU provides a parameter (often represented as a) that controls the function's slope for negative inputs, allowing for greater flexibility in slope. To prevent neuron inactivity and maintain network performance, this parameter maintains some control over the extent of the negative input.
Reduced Exploding Gradient Problem: The "exploding gradient" problem, when gradients grow enormously during training, can occasionally be brought on by ReLU's unbounded positive activation. Leaky ReLU can assist in mitigating the expanding gradient problem by permitting some gradient for negative inputs and thereby lowering the size of gradients.
Empirical Performance: Leaky ReLU has performed better empirically than regular ReLU in several deep-learning tests. In some situations, it has been discovered to converge more quickly and produce superior outcomes.

Conclusion

Compared to the traditional ReLU activation function in Artificial Neural Networks (ANNs), leaky ReLU offers several advantages. Leaky ReLU ensures that neurons are active and continue to learn, especially in deeper networks, by tackling the dying ReLU problem and introducing a modest slope for negative inputs. Improved performance and training are benefits of this ability to reduce neuronal inactivity. Leaky ReLU is favoured when the dying ReLU issue is an issue. Still, it's vital to understand that the choice of activation function depends on the particular issue, the design, and empirical testing. Exponential Linear Unit (ELU) and Parametric ReLU are two further versions that address related issues, highlighting the necessity of selecting an appropriate activation function to improve the performance of artificial neural networks.

ANN Tutorial