Deep learning is about neural networks, which are an interconnected web of nodes whose edges are connected. Neural networks are used for many applications.
Computers are very good at repetitive tasks but they are very poor at recognizing patterns. Thanks to deep learning, this is about to change. For example, imagine that we want to categorize pictures based on tags; this is just not possible with standard computing.
Neural networks have three layers: input layers, hidden layers and output layers.
The generated output is the modified data from the output layers.
The output is the actual known class.
The gradient is the rate at which cost changes with respect to weight or bias.
The cost is the generated output minus the actual output.
Forward-propagation is the process in which an input is passed and classified through the hidden layers until it reaches the output layers and is classified.
Deep networks have been around for almost 50 years, so why have they only become popular recently? Well, it’s because they are very hard to train and it can take a very long time. Proper and cost-effective training wasn’t possible until the late 2000s, when the restricted Boltzmann machine was invented.
Training is the process through which we increase neuronal network accuracy using the back-propagation process. In this process, the outcome of forward-propagation between two subsequent layers is compared to known classes and the cost is the difference between the two. The purpose of training is to run a high number of training passes, in which adjustments in each iteration are made with the goal of progressively reducing the cost.
Without going into detail, keep in mind that the early layers are the ones that are the most difficult to train, and this brings a fundamental issue since the first layers are the most important in directing the network towards the right solution. This means that if the early layers are wrong, the result built up by the system will also be wrong.