From ReLU to GeMU: Activation functions in the lens of cone projection

Area of Science:

• Machine Learning
• Neural Networks
• Optimization

Background:

• Activation functions introduce nonlinearity in neural networks.
• Rectified Linear Unit (ReLU) is a widely used activation function known for its simplicity and effectiveness.
• Feedforward Neural Networks (FNNs) share structural similarities with Projected Gradient Descent (PGD) algorithms.

Purpose of the Study:

• To generalize the ReLU activation function.
• To explore projection operators onto convex cones for neural network activation.
• To enhance the expressive power of neural networks.

Main Methods:

• Interpreting ReLU as a projection onto the nonnegative half-line.
• Developing the Generalized Multivariate projection Unit (GeMU) as a projection operator onto convex cones (e.g., Second-Order Cone).
• Analyzing the expressive power of FNNs utilizing GeMU.

Main Results:

• The expressive power of FNNs activated by GeMU is strictly greater than those activated by ReLU.
• GeMU demonstrates versatility across different neural network architectures and tasks.
• GeMU outperforms existing activation functions in experimental evaluations.

Conclusions:

• GeMU offers a powerful generalization of ReLU for neural network activation.
• The proposed GeMU enhances model expressiveness and empirical performance.
• GeMU represents a promising advancement in activation function design for deep learning.