Summary
A neural network is a series of nodes, or artificial neurons, connected to each other in such a way that they can take an input signal, and convert it to output by passing it through the network. Neural networks are useful in pattern recognition, as the creator does not need to write algorithms to recognize every pattern; the network learns how to recognize the patterns by going through many examples. In this way a neural network can find patterns and make predictions that the programmer knew nothing about.
The purpose of this project is to better explain how exactly neural networks function, and to create a simple example of a neural network that performs the function of an XOR gate.
Introduction/Background
Neural networks were first imagined in 1943 by Warren McCulloch and Walter Pitts. Since then, they have progressed greatly, with the most noteworthy advancement being the creation of the backpropagation algorithm in 1974. While critics say that neural networks are inefficient, as they take up a great amount of resources to store all of the neurons and connections, and take a lot of processing power to send signals across the connections, they have proved very useful in making predictions, as they can find patterns by themselves given training data.
Application
Various researchers have made great progress using neural networks in the field of pattern recognition. Neural networks were the first artificial systems to achieve a level of pattern recognition as good as human recognition, and now there are even some systems with better-than-human pattern recognition. Alex Graves et al. created a neural network that was able to learn to recognize handwriting in 3 different languages. At the International Joint Conference on Neural Networks in 2012, neural networks were able to compete with humans in recognizing traffic signs. Neural networks are useful in the field of deep learning.
Previous Work
A lot of work has been done recently with neural networks. Notably, Alex Graves and Jurgen Schmidhuber created a Multi-Dimensional Recurrent Neural Network (MDRNN), which can recognize handwriting one character at a time, and is capable of being trained in any language. Their network also uses memory storage to contain information about the network’s function. The network was successfully able to learn to recognize Arabic when tested with handwritten Tunisian post codes.
Also, Gert Westermann of the University of Edinburgh led a team that made a network to conjugate English past-tense verbs. According to the report:
After 912 epochs, the network produced 100% of the
irregular and 99.8% (all but two) of the regular past
tense forms correctly.
These networks are much more complicated and sophisticated than the one proposed in this project. They are simply meant as a demonstration of the power of neural networks.
Current Problems in the Area
Currently, a big problem and source of criticism in the field of neural networks is the amount of resources that a neural network uses up. The brain has evolved to perform computations in this manner, but computers were not designed to do so, and thus neural networks lead to a sub-optimal allocation of resources.
Also, neural networks are looked down upon as being black boxes. A neural network can be trained to behave in any way, but it is hard to alter its behavior without creating thousands of further testing scenarios, and the weights that the network ends up with at the end do not help researchers better understand the workings of the network. They may as well be random numbers.
Proposed Solutions
Given the rapid increase in computer power that is constantly going on, the first objection is not very strong. Even if neural networks do not use resources optimally, their useful function makes up for this failure, and as computing power increases, so does the power of neural networks. It would be an interesting experiment to try to create a computer architecture more suited to neural networks, with physical connections between nodes, but it is not a necessary project for the field to advance. Also, work into biological neural networks could eventual lead to an optimized solution.
The second problem is not so easy to brush aside. It is a significant flaw that neural networks cannot be easily adjusted. One way to find out what parts of a network would have to be adjusted would be to run through one trail with a new scenario in which the desired behavior was required and keep track of which nodes had their weights changed most and then change those weights by hand. Of course, this also has the problem that it might affect the outputs from other scenarios, which is probably why it is not done.
Conclusion
Neural networks are an interesting field very different from other approaches to AI. This has its benefits, like their usefulness in pattern recognition, but it also has drawbacks. Neural networks are an interesting roundabout way of solving problems in AI like modelling and prediction, and while they may not be the most efficient way to solve problems, they are fascinating in themselves.
Future Work
In future, I propose constructing a more sophisticated neural network that moves beyond being a simple proof of concept and actually performs a useful function, like predicting weather.
References
A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, J. Schmidhuber. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, 2009.
D. C. Ciresan, U. Meier, J. Masci, J. Schmidhuber>. Multi-Column Deep Neural Network for Traffic Sign Classification. Neural Networks, 2012.
Graves, Alex, and Jurgen Schmidhuber. “Offine Handwriting Recognition with
Multidimensional Recurrent Neural Networks.” 2009.
McCulloch, Warren; Walter Pitts (1943). “A Logical Calculus of Ideas Immanent in Nervous Activity”.Bulletin of Mathematical Biophysics
Werbos, P.J. (1975). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences
