GRNN : Generalized Regression Neural Networks

GRNN stands for Generalized Regression neural network.

1. What does GRNN do ??

This is basically a neural network based function approximation or function estimation algorithm. It predicts the output of a given input data.

2. How does it work ??

As per the basic principle of neural network it needs a training data to train itself. Training data should contain input-output mapping. Now if we train the network with the training data set and  we feed a new testing data set, it will accordingly gives the output or predict the result.

In case of GRNN, output is estimated  using weighted average of the outputs of training dataset, where the weight  is calculated using the euclidean distance between the training data  and   test data. If the weight or distance is large then the weight will be very less and if the distance is small it will put more weight to the output.

3. Network Architecture

Network architecture contains four basic layers. Input layer,Pattern layer,Summation layer, Output layer.

Input layer:

Input layer feeds the input to the next layer.

Pattern layer:

Pattern layer calculates the Euclidean distance and activation function.

Summation layer: 

Summation layer has two subparts one is Numerator part and another one is Denominator part. Numerator part contains summation of the multiplication of training output data and  activation function. Denominator is the summation of all activation function. This layer feeds both the Numerator & Denominator to the next output layer.

Output layer:

Output layer contains one neuron which calculate the output by dividing the numerator part of the Summation layer by the denominator part.

4. Main Principle

GRNN stands on the below equation:

Y(x)=∑Y_ie^{-(d_i2∕2σ2)} ∕ ∑e^{-(d_i2∕2σ2)}

Where, d_i²=(x-x_i)^T(x-x_i)

Here x is the input sample and x_i is the training sample.  Output of the input sample i

 is Y_i. d_i² is the Euclidean distance from the x and x_i. e^{-(d_i2∕2σ2)} is the activation function. Basically this activation function theoretically is the weight for that input .

Now if you look closely, The value of    d_i²   signifies how much the training sample can contribute to the output of that test particular test sample. 

If  d_i²  has small value that means it will contribute more value to the output but if it is a big value that means it will contribute very less to the output.

The term  e^{-(d_i2∕2σ2)} is deciding that how much weight the training sample will contribute.

If d_i² is small value, the term e^{-(d_i2∕2σ2)}  returns a relatively large value.

If d_i² is large value, the term e^{-(d_i2∕2σ2)}  returns a relatively small value.

If d_i² is zero the term e^{-(d_i2∕2σ2)}  returns one that means test data = training sample and the output of test data will be the output of the training sample.

Here we have only one unknown parameter, spread constant σ. That can be tuned by training process to an optimum value where the error will be very small.

5. Training Procedure

Training procedure is to find out the optimum value of σ. Best practice is that find the position where the MSE (Mean Squared Error) is minimum.

First divide the whole training sample into two parts. Training sample and test sample. Apply GRNN on the test data based on training data and find out the MSE for different σ. Now find the minimum MSE and corresponding value of σ. 

6. Advantages of GRNN

1) The main advantage of GRNN is to speed up the training process which helps the network to be trained faster.

2) The network is able to learning from the training data by “1-pass” training in a fraction of the time it takes to train standard feed forward networks.

 3) The spread, Sigma (σ), is the only free parameter in the network, which often can be identified by the V-fold or Split-Sample cross validation.

4) Unlike standard feed forward networks, GRNN estimation is always able to converge to a global solution and won’t be trapped by a local minimum. 

7. Example

input    output

2             3

4             5

6             7

8             9   

What will be the output of 5??

Step 1

Calculate distances d1 = (5-2)^2 = 9 , d2= (5-4)^2 = 1, d3 =(5-6)^2=1, d4 = (5-8)^2 = 9.

Step 2

Calculate weights using the activation function:   e^{-(d_i2∕2σ2)}

Lets say σ = 1.

so weights are, 

w1 =  0.01

w2 =  0.6

w3=   0.6

w4=   0.01

Step 3

Summation of  w's  W = w1+w2+w3+w4 =  1.22

So denominator is 1.22.

Now numerator is  YW = w1*y1 + w2*y2+w3*y3+ w4*y4

                                    =0 .01*3+0.6*5+0.6*7+0.01*9 

                                    = 7.32

Step 4

So the output is: (Neumerator/Denominator )

output = YW/W = 7.32/1.22 = 6.

So predicted output is 6.