13
其实是有例外的——CNN

Neocognitron 1980
Institute of Computing Technology, Chinese Academy of Sciences
Local Connection
K. Fukushima, ―Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,‖ Biological Cybernetics, vol. 36, pp. 193–202, 1980 14
例外：CNN用于数字识别
Institute of Computing Technology, Chinese Academy of Sciences
15
例外：CNN用于目标检测与识别
Institute of Computing Technology, Chinese Academy of Sciences
DBN Science
Description Deep learning Hand-crafted features and learning models. Bottleneck.
Institute of Computing Technology, Chinese Academy of Sciences
Output layer Feedforward operation Hidden layers Input layer
David E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning representations by back-propagating errors". Nature 323 (6088): 533–536
深度学习：快速推进中的 机器学习与人工智能前沿
山世光
中科院计算所
提纲
深度学习(DL)及其应用前沿 DL在CV领域应用的启示 关键算法介绍

Perceptron及学习算法
Institute of Computing Technology, Chinese Academy of Sciences
12
其实是有例外的——CNN

卷积神经网络CNN


Institute of Computing Technology, Chinese Academy of Sciences

K. Fukushima, ―Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,‖ Biological Cybernetics, vol. 36, pp. 193–202, 1980 Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, ―Backpropagation applied to handwritten zip code recognition,‖ Neural Computation, vol. 1, no. 4, pp. 541–551, 1989 Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner, ―Gradientbased learning applied to document recognition,‖ Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998
Vector (Estimation)
Object recognition
{dog, cat, horse,, …}
Super resolution
Low-resolution image
High-resolution image
3
源起——生物神经系统的启示

神经元之间通过突触(synapse)连接
9
It
can get stuck in poor local optima

1990-2006更流行…

Specific methods for specific tasks
Hand-crafted ML

features (SIFT, LBP, HOG)
methods

SVM
Kernel tricks AdaBoost

Hinton, G. E., Osindero, S. and Teh, Y., A fast learning algorithm for deep belief nets. Neural Computation 18:1527-1554, 2006 Hinton, G. E. and Salakhutdinov, R. R. (2006) Reducing the dimensionality of data with neural networks. Science, Vol. 313. no. 5786, pp. 504 - 507, 28 July 2006 Yoshua Bengio, Pascal Lamblin, Dan Popovici and Hugo Larochelle, Greedy Layer-Wise Training of Deep Networks, Advances in Neural Information Processing Systems 19 (NIPS 2006) Marc’Aurelio Ranzato, Christopher Poultney, Sumit Chopra and Yann LeCun. Efficient Learning of Sparse Representations with an Energy-Based Model, Advances in Neural Information Processing Systems (NIPS 2006)
& Papert的专著Perceptron(1969) 只能对线性可分的模式进行分类 解决不了异或问题 几乎宣判了这类模型的死刑，导致了随后多 年NN研究的低潮
Minsky
Institute of Computing Technology, Chinese Academy of Sciences
16
而且，东风同样重要
大数据 大数据 大数据

语音图像视频
Institute of Computing Technology, Chinese Academy of Sciences

计算能力
并行计算平台 GPU大量部署

开放的社区
开源，开放数据
17
Speech
DBN Science
Institute of Computing Technology, Chinese Academy of Sciences
11
A Breakthrough Back to 2006



Institute of Computing Technology, Chinese Academy of Sciences
MLP及其BP算法
Auto-Encoder CNN及其主要变种

关于DL的思考与讨论ຫໍສະໝຸດ 2机器学习的基本任务
x
Institute of Computing Technology, Chinese Academy of Sciences
F ( x)
y

Class label (Classification)
It
requires labeled training data
Almost all data is unlabeled.

The

learning time does not scale well
It is very slow in networks with multiple hidden layers. These are often quite good, but for deep nets they are far from optimal.
6
2nd Generation Neural Networks

多层感知机(Multi-layer Perceptron, MLP)
超过1层的hidden Compute
layers（正确输出未知的层）
BP算法 [Rumelhart et al., 1986]
error signal; Then, back-propagate error signal to get derivatives for learning
层级感受野，学习使突触连接增强或变弱甚至消失
Institute of Computing Technology, Chinese Academy of Sciences
Hubel, D. H. & Wiesel, T. N. (1962)
4
第一代神经网络

Institute of Computing Technology, Chinese Academy of Sciences
Institute of Computing Technology, Chinese Academy of Sciences