-
Notifications
You must be signed in to change notification settings - Fork 4
Normalization and Regularization
In image processing, normalization is a process that changes the range of pixel intensity values. When we normalize we are changing our scale from one system to another. when our data is distributed in small domain the kernels do not have enough feature distribution wrt amplitude to be able to find that particular feature and when we stretch our range we get the normalization back i.e the amplitude changes back and it becomes easier for the kernel to learn.
Batch Normalization is dependent on number of channels. In case of BN, for each of the channel we have 1 mu each, so if we have 4 channels we have 4 mu's. Mu is nothing but the average of channel1 from img1 + channel1 from img2 + channel1 from img3 respectively. Then, we have 4 sigma's and this is calculated using the variance for each channel. Adding BN helps reduce overfit, because it changes the images.
Layer Normalization is for one image we would be normalizing all the channels in a particular layer. In LN all the channels of the layer is been averaged but for 1 image only, so the total number of mu's are dependent on the images we have.
Group Normalization is something in between BN and LN, in GN we would be normalizing the data or values among some of the channels. Lets say, we have a layer with 10 channels and if we make a group of two, then we would divide this 10 channels and divide into 5 & 5. Then we would be doing layer normalization among these 5.
Regularization is about the image that we have never seen, it is basically a image augmentation technique, but here we are talking about the images in our channels not the beginning of the images, whenever we are using the regularization we are saying that we are reducing the gap between test and train acc, and not increasing test acc, or increasing train acc or increasing test acc while reducing train acc.
There are two kinds of regularization L1 (Lasso) and L2 (Ridge)