ANN - Multi-Layer Perceptron (MLP)

 

UNIT – IV 

1. Multi-Layer Perceptron (MLP) with Two Hidden Layers

A Multi-Layer Perceptron with two hidden layers is a feed-forward artificial neural network that consists of:

• One input layer
• Two hidden layers
• One output layer

It is capable of learning complex non-linear relationships in data.

Architecture of MLP with Two Hidden Layers

Layers:

1. Input Layer
• Receives input features
• No computation is performed
2. Hidden Layer 1
• Receives input from input layer
• Performs weighted summation and activation
3. Hidden Layer 2
• Receives output from hidden layer 1
• Further extracts features
4. Output Layer
• Produces final output

Each neuron in a layer is connected to all neurons in the next layer.

Working of MLP with Two Hidden Layers

Activation Functions Used

• Hidden layers: ReLU / Tanh / Sigmoid
• Output layer:
• Sigmoid (binary classification)
• Softmax (multi-class)
• Linear (regression)

 Training of MLP (Backpropagation)

Advantages

• Learns highly complex patterns
• Better feature extraction than single hidden layer
• Higher accuracy

Limitations

• High computational cost
• Longer training time
• Risk of overfitting
• Needs large dataset

Applications

• Image recognition
• Speech processing
• Medical diagnosis
• Natural language processing
• Financial forecasting

---

2. Simple Layer of a Multi-Layer Perceptron

A simple layer of a Multi-Layer Perceptron (MLP) is a group of neurons that receives inputs from the previous layer, performs weighted summation, adds bias, applies an activation function, and forwards the output to the next layer.

 Components of a Simple Layer

A simple layer consists of:

• Input signals
• Weights
• Bias
• Neurons
• Activation function

 

Working of a Simple Layer

Step 1: Input Reception

The layer receives inputs (x_1, x_2,……, x_n) from the previous layer.

Step 2: Weighted Summation

Step 3: Activation Function

Mathematical Representation 

 Role of a Simple Layer in MLP

• Extracts features from input data
• Introduces non-linearity
• Improves learning capability
• Passes processed information to the next layer

 Example

For a simple layer with one neuron: 

 Advantages

• Easy to understand and implement
• Enables deep learning when stacked
• Improves model accuracy

---

 3. Delta Learning Rule of the Output Layer

The Delta learning rule is a supervised learning rule used to update the weights of the output layer in a Multi-Layer Perceptron (MLP) by minimizing the error between the desired output and the actual output.

 Purpose of Delta Learning Rule

• To reduce output error
• To adjust output layer weights
• To improve network accuracy
• To minimize mean squared error (MSE)

 Error Calculation at Output Layer

The error for a single output neuron is: 

Where:

• (t) = target output
• (y) = actual output

 Delta Term (Output Layer)

The delta value for the output neuron is:

Weight Update Rule (Output Layer)

Steps of Delta Learning Rule (Output Layer)

1. Apply inputs and perform forward pass
2. Compute output (y)
3. Calculate error ((t - y))
4. Compute delta value
5. Update output layer weights
6. Repeat until error is minimized

 Activation Function Derivatives 

Advantages

• Simple and effective
• Ensures gradual error reduction
• Works well with gradient descent

Limitations

• Requires differentiable activation function
• Slow convergence for complex problems

 Application

• Used in training MLP
• Used in backpropagation algorithm
• Applied in regression and classification tasks

---

 4. Multilayer Feed-Forward Neural Network with Continuous Perceptrons

A Multilayer Feed-Forward Neural Network with Continuous Perceptrons is an artificial neural network in which:

• Neurons are arranged in multiple layers
• Information flows only in the forward direction
• Each neuron uses a continuous (differentiable) activation function

This network is capable of learning complex non-linear mappings.

 Need for Continuous Perceptrons

• Step (hard-limit) perceptrons are not differentiable
• Backpropagation requires differentiable functions
• Continuous perceptrons allow gradient-based learning

 Architecture

Layers:

1. Input Layer
• Receives input features
• No computation is performed
2. Hidden Layer(s)
• Uses continuous activation functions
• Performs feature extraction
3. Output Layer
• Produces final output
• Uses suitable continuous activation function

All connections are unidirectional (feed-forward).

 Continuous Perceptron Model

 Common Continuous Activation Functions

Working of Multilayer Feed-Forward Network

1. Input is applied to the input layer
2. Signals propagate forward through hidden layers
3. Each neuron performs weighted sum and activation
4. Output layer produces final result

No feedback or loops are present.

Training Method – Backpropagation

Error Function

[
E = \frac{1}{2}(t - y)^2
]

Delta Term (Output Layer)

[
\delta = (t - y) f'(net)
]

Weight Update Rule

[
w_{new} = w_{old} + \eta \delta x
]

Where:

• (\eta) = learning rate

 Advantages

• Can learn complex non-linear problems
• Differentiable activation enables efficient training
• High accuracy for real-world applications

 Limitations

• Computationally expensive
• Slow training for deep networks
• Risk of overfitting

 Applications

• Pattern recognition
• Speech and image processing
• Medical diagnosis
• Function approximation

---