LeNet

LeNet is a series of convolutional neural network architectures created by a research group at AT&T Bell Laboratories between of the period of 1988 to 1998, centered around Yann LeCun. They were designed for reading small grayscale images of handwritten digits and letters, and were used in ATMs for reading cheques. Convolutional neural networks are a kind of feed-forward neural network whose artificial neurons can respond to a part of the surrounding cells in the coverage range and perform well in large-scale image processing. LeNet-5 was one of the earliest convolutional neural networks and was historically important during the development of deep learning. In general, when LeNet is referred to without a number, it refers to the 1998 version, LeNet-5, which is the most well-known version.

LeNet is a series of convolutional neural network architectures created by a research group at AT&T Bell Laboratories between of the period of 1988 to 1998, centered around Yann LeCun. They were designed for reading small grayscale images of handwritten digits and letters, and were used in ATMs for reading cheques. Convolutional neural networks are a kind of feed-forward neural network whose artificial neurons can respond to a part of the surrounding cells in the coverage range and perform well in large-scale image processing. LeNet-5 was one of the earliest convolutional neural networks and was historically important during the development of deep learning. In general, when LeNet is referred to without a number, it refers to the 1998 version, LeNet-5, which is the most well-known version. Each of the "subsampling" layers is not precisely an average pooling, but with trainable parameters. Specifically, consider a single cell in S2. It takes as input 4 cells in C1. Let these have values x_1, \dots, x_4 , then the cell in S2 has value \sigma \left(w \left(\sum_{i=1}^4 x_i \right) + b \right) , where w, b \in \R are trainable parameters, and \sigma is a sigmoidal activation function, sometimes called "LeCun's tanh". It is a scaled version of the hyperbolic tangent activation function: 1.7159 \tanh(2x/3) . It was designed so that it maps the interval [-1, +1] to itself, thus ensuring that the overall gain is around 1 in "normal operating conditions", and that |f(x)| is at maximum when x = -1, +1 , which improves convergence at the end of training. In the following table, each column indicates which of the 6 feature maps in S2 are combined by the units in each of the 15 feature maps of C3. Even though C5 has output shape 120 \times 1 \times 1 , it is not a fully connected layer, because the network is designed to be able to take in input shapes of arbitrary height and width, much larger than the 1 \times 32 \times 32 that the network is trained on. In those cases, the output shape of C5 would be larger. Similarly, the output of F6 would also be larger than 84 \times 1 \times 1 . Indeed, in modern language, the layer F6 is better described as a 1 \times 1 convolution. The output of the convolutional part of the network has 84 neurons, and this is not a coincidence. It was designed such, because 84 = 7×12, meaning that the output of the network can be viewed as a small 7×12 grayscale image. The output layer has RBF units, similar to RBF networks. Each of the 10 units has 84 parameters, which might either be hand-designed and fixed, or trained. When hand-designed, it was designed so that, when viewed as a 7×12 grayscale image, it looks like the digit to be recognized. 1998 LeNet was trained with stochastic Levenberg–Marquardt algorithm with diagonal approximation of the Hessian. It was trained for about 20 epoches over MNIST. It took 2 to 3 days of CPU time on a Silicon Graphics Origin 2000 server, using a single 200 MHz R10000 processor.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1372994034 under CC BY-SA 4.0; modifications have been made.