The purpose of Activation functions in neural networks

figure 1 — source

Introduction

figure 2 — source

The above image(figure 2) describes each node in the neural network (figure 1). All hidden layers (nodes/units) typically use the same activation function. Except for the output layer, will use a different activation function for its units.

Why we use activation function?

To keep it simple, activation functions are what make the neural network learn by deciding which information is important to pass to the next neuron.

One more reason to use an activation function is because of the sum term,
this value can go very high in magnitude, especially in the case of very deep neural networks that have millions of parameters. Using an activation function will limit the output value of this sum.

Various Activation Functions

Binary

Binary Activation function — source

Left:

f(s) = 1 if s ≥ 0; f(s) = 0 if s < 0

Right:

f(s) = 1 if s > 0; f(s) = 0 if s ≤ 0 

Linear

Linear Activation function — source

f(x) = x

Range : ]-infinity ,infinity[

Sigmoid

A sigmoid function is a “S”-shaped curve or sigmoid curve. the term “sigmoid function” is used as an alias for the logistic function.

Sigmoid Activation function

Range= [0, 1]

Softmax

The softmax is a more generalized form of the sigmoid. It is used in multi-class classification problems. Similar to sigmoid, it produces values in the range of 0–1 therefore it is used as the final layer in classification models.

example of NN with softmax — source

The values produced are probabilities that's why we generally use a cross entropy loss function to get a one-hot labels.

Tanh

Tanh Activation function

Hyperbolic Tangent Function (tanh) function is mainly used classification between two classes.

f(x) = (e^x — e^-x) / (e^x + e^-x)

Range = [-1, 1]

ReLUT

The rectifier or ReLU (Rectified Linear Unit) activation function is an activation function defined as the positive part of its argument.

ReLU (Rectified Linear Unit) activation function

f(x) = max(0, x)

Range = [0, +infinity [

This blog was a brief introduction to different Activation function used in the artificial neural network and we choose to work with a specific function according to its property(Nonlinear, Range, Continuously differentiable ..).

Activation Functions in Neural Networks

Activation function - Wikipedia

Explaning the softmax activation function:

The Softmax activation function