1
W1(j=1,i=1) W1(j=2,i=1) W1(j=9,i=1)
A1 Neuron i=1 A2 Bias=b1(i=1) A1(i=1) A5 A1(i=1)
W2(i=1,k=1) W2(i=2,k=1) W2(i=5,k=1) Neuron k=1 Bias=b2(k=1)
A2(k=2)
Convolution Neural Network CNN ver. 4.11a
3
Part 1
Fully Connected Back Propagation (BP) neural net
Convolution Neural Network CNN ver. 4.11a
4
Theory Fully connected Back Propagation Neural Net (BPNN)
Convolution Neural Network CNN
A tutorial KH Wong
Convolution Neural Network CNN ver. 4.11a
1
Introduction
• Very Popular:
– Toolboxes: cuda-convnet and caffe (user friendlier)
Convolution Neural Network CNN ver. 4.11a
1 , is t heparamterfor slope u 1 e
/chapter/10.1007%2F3-540-59497-3_175#page-1
9
A single neuron
Convolution Neural Network CNN ver. 4.11a 6
Part 1A
Forward pass of Back Propagation Neural Net (BPNN) Recall: Forward pass for each output neuron:
-Use training samples: Xclass_t : feed forward to find y. -Err=error_function(y-t)
•
f (u ) Hence 1 d u u df ( u ) df ( e ) 1 e ' f (u ) du d (1 e u ) du u u f ' (u ) e e (1 e u ) 2 (1 e u ) 2 1 e u f (u )1 f (u ) u u (1 e ) (1 e ) For simplicit y , paramterfor t he slope 1 f ' (u ) f (u )1 f (u )
Convolution Neural Network CNN ver. 4.11a 7
Feed forward of Back Propagation Neural Net (BPNN)
• In side each neuron: such thatx x , u u ,W
l l l l
x l f (u l ) with u l W l x l bl ,
• Use many samples to train the weights, so it can be used to classify an unknown input into different classes • Will explain
– How to use it after training: forward pass – How to train it: how to train the weights and biases (using forward and backward passes)
Convolution Neural Network CNN ver. 4.11a
5
Training
• How to train it: how to train the weights (W) and biases (b) (use forward, backward passes) • Initialize W and b randomly • Iter=1: all_epocks (each is called an epcok)
• A high performance Classifier (multi-class) • Successful in handwritten optical character OCR recognition, speech recognition, image noise removal etc. • Easy to implementation
Output neurons=3 neurons, indexed by k W2=5x3 b2=3x1
•
14
Answer (exercise: write values for A1(i=4) and A2(k=3)
• P=[ 0.7656 0.7344 0.9609 0.9961 0.9141 0.9063 0.0977 0.0938 0.0859] • W1=[ 0.2112 0.1540 -0.0687 -0.0289 0.0720 -0.1666 0.2938 -0.0169 -0.1127] • -b1= 0.1441 • %Find A1(i=4) • A1_i_is_4=1/(1+exp[-(W1*P+b1))] • =0.49
– Slow in learning – Fast in classification
Convolution Neural Network CNN ver. 4.11a 2
Overview of this note
• Part 1: Fully connected Back Propagation Neural Networks (BPNN)
l
W l , bl bl T ypicallyf is a logistic (sigmod) function,i.e. 1 , therefore u 1 e 1 x l f (u l ) (W l x l b l ) 1 e f (u )
x l 1
x l 2
N c 1 n Overallerror E N tkn yk 2 n 1 k 1
c 1 1 n n n n Errorfor each neuron: E tk yk t y n 2 k 1 2
2
2 2
2 s norm; tkn T hegiven true class of then th trainingsample
– Part 1A: feed forward processing – Part 1A: feed backward processing
• Part 2: Convolution neural networks (CNN)
– Part 2A: feed forward of CNN – Part 2B: feed backward of CNN
n yk T heoutputclass of then th trainingsample
at theouput of t hefeed forward net work
Convolution Neural Network CNN ver. 4.11a
12
Structure of a BP neural network
BPNN Forward pass
• Forward pass is to find output when an input is given. For example: • Assume we have used N=60,000 images to train a network to recognize c=10 numerals. • When an unknown image is input, the output neuron corresponds to the correct answer will give the highest output level.
xl f (ul ) with ul W l xl 1 bl

ul
f ul

xl
x l 1 inputs at layer l 1 x x, W l weights , w W
xl input at layerl
Convolution Neural Network CNN ver. 4.11a 10
Input image
10 output neurons for 0,1,2,..,9
Convolu. 4.11a 11
The criteria to train a network
• Is based on the overall error function • 2
W1(j=1,i=1) W1(j=2,i=1)
W2(i=1,k=1) A1(i=2) W2(i=2,k=1) W2(i=2,k=2)
P(j=3)
W1(j=3,i=4) : :
W2(i=5,k=3)
W1(j=9,i=5)
P(j=9) Input: P=9x1 Indexed by j
A1(i=5) Hidden layer =5 neurons, indexed by i W1=9x5 Convolution Neural Network CNN ver. 4.11a b1=5x1
x
l 3
xl N
wl 2
Inputs