Guide to Activation Functions in Neural Networks
How to choose the best activation function for your deep learning model
Activation functions play an essential part in artificial neural networks as they decide whether a neuron should be activated or not. Activating a neuron determines whether the information that the neuron is receiving is relevant and if it should be used or ignored.
Activation functions are mathematical equations that help us determine the output of the neural network. The purpose of an activation function is to derive an output value based on the input value that is fed to a neuron.
These activation functions are mainly used in the hidden and output layers of the artificial neural network. In this article, we will discuss why are activation functions required in neural networks and the types of activation functions.
Let’s get started!
Why do we need Activation Functions?
The purpose of the activation function in deep learning is to add non-linearity to the neural networks. It decides whether a neuron should be activated or not. Without an activation function, a neural network is just a linear regression model.
In a neural network with no activation function, each neuron will only perform a linear transformation on the input data using its weights and biases.
For each neuron in an artificial neural network, a respective weight is assigned to it, which is then multiplied by its input and then a static bias value is added. Bias values are arbitrary and unique to each neuron.
Example:
x = (w * input) + bWhere w is weight and b is bias.
Then, this value is passed to an activation function.
Y = Activation function(∑(w * input) + b)This is the scenario for a single neuron. The output achieved from the activation function is then passed to the neurons/nodes in the next layer and the same process is repeated.
Types of Activation Functions in Neural Networks
Binary Step Function
The binary step function is a threshold-based activation function. It is one of the simplest activation functions. In this activation function, a neuron is activated and deactivated based on the threshold value.
Here, we compare the input value with a certain threshold value. If the input value is greater than the threshold value, then the neuron gets activated, it gets deactivated if the input value is less than the threshold value. When the neuron is deactivated, its output is not passed to the next layer.
For example, let’s assume that the threshold value is 0. Here, if the input is > 0, then the output becomes 1. If the input value is less than 0 then the output will be 0.
It also can be represented as:
f(x) = 1, x>=0
= 0, x<0Example:
def binary_step_function(input_value):
if input_value < 0:
return 0
else:
return 1
binary_step_function(5)Output:
1
The limitation of this function is that it is not differentiable. If we try to calculate the derivation of this function f(x), it will be zero. Therefore, we cannot train it using gradient descent.
When to use?
This activation function is good to use as an activation function when creating a binary classification and the output result contains only two classes. For example, when we have to predict whether the image contains a dog or a cat.
This binary activation function is not very helpful when we have to deal with multi-class classification problems.
Did you know we’re on YouTube? Check out Office Hours, hear what customers have to say, and get inspired.
Linear Activation Function
A linear function is a simple straight-line activation function where the function is directly proportional to the input value. It is also known as “no identity” or “identity activation function.”
The problem with the binary step function is that its gradient becomes zero as there is no component of x in the function. Therefore, an alternative to the binary step function is the linear activation function. Unlike binary step functions, the linear function can handle more than one class.
Get Pralabh Saxena’s stories in your inbox
Join Medium for free to get updates from this writer.
This activation is can be used when we are dealing with regression problems. The range for this activation action is (-∞ to ∞). It simply adds the weighted total of the input and returns the result.
The equation of this activation function is the same as a straight line, i.e.:
y = axWhere a is a constant value.
Example:
def linear_function(input_value):
return 4 * input_valuelinear_function(4)Output:
16
It also can be represented as:
f(x) = xThe limitation of this activation function is that, if we differentiate this activation function to add non-linearity, the outcome for this function will be a constant value and will have no relation to the input value x.
And because it has no relation with input x, as a result during the backpropagation the rate of change of error remains constant and the performance of the neural network does not improve due to the constant gradient.
When to use?
The linear activation function can be used in the final output layer of a neural network. This activation function is used when we want to have actual values in our output (such as in the case of linear regression), instead of restricting the output to some range such as (0,1) or (-1,1).
Non-Linear Activation Function
Non-linear activation functions are the most used activation functions in neural networks. Here, the meaning of the non-linear is that output can not be reproduced from the linear sets of the input values.
A non-linear activation function is useful for stacking multiple layers of neurons that create a deep neural network and can be trained on complex data sets with high performance and accuracy.
There are various non-linear activation functions in deep learning, that are divided based on their range and curves. One common example of non-linear activation functions is the sigmoid activation function.
The formula for the sigmoid activation function is:
Example:
import numpy as npdef sigmoid_function(input_value):
output = (1/(1 + np.exp(-input_value)))
return outputx_data = 7
y_data = sigmoid_function(x_data)print(y_data)Output:
0.99908894
The non-linear activation functions are used in both the hidden and output layer of a neural network.
When to use?
Non-linear activation functions are differentiable and make it easy for the model to generalize with the variety of data. It is necessary for the activation function to differentiate the output as it helps the neuron weights to change accordingly and achieve the global minima.
The non-linear activation functions are used for non-linear transformations in the input data to make it capable of learning and performing more complex tasks.
Conclusion
That’s it from this article. In this article, we discussed the activation functions and their use in neural networks. We also discussed the types of the various activation functions with their mathematical formulas and examples.
Thanks for reading!
Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.
Editorially independent, Heartbeat is sponsored and published by Comet, an MLOps platform that enables data scientists & ML teams to track, compare, explain, & optimize their experiments. We pay our contributors, and we don’t sell ads.
If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletter (Deep Learning Weekly), check out the Comet blog, join us on Slack, and follow Comet on Twitter and LinkedIn for resources, events, and much more that will help you build better ML models, faster.
