Why Research Matters

The brain has long perplexed scientists, researchers, philosophers and thinkers on the mechanisms of cognition and consciousness.

Neural networks and the brain
When AI started to gain popularity decades ago, there was debate about making a machine “learn,” since developers still had little idea how humans learned. One approach was to have machines mimic the way that the human brain learns.

Since the brain is primarily a collection of interconnected neurons, AI researchers sought inspiration from the brain by re-creating the brain’s structure through artificial neurons. They could connect simple artificial neurons in complex ways, and the connections of those neurons in artificial neural networks would create more complicated outcomes.

This is how the idea of artificial neural networks emerged.

The appeal of neural networks has waxed and waned over the decades. Neural nets, and specifically a form of neural nets called deep learning, have caught the industry’s attention with their remarkable capabilities when fed with large quantities of data and computing horsepower. The emergence of deep learning combined with big data and computing has brought about the recent renaissance in AI. However, there have been some challenges to the neural networking approach.

We’re starting to face limitations on the abilities of neural networks and still are major innovations away from solving the hard problems of cognition. Despite decades of research on AI and neuroscience, we’re still not sure how exactly the brain works, raising the question as to whether creating AI based on human brains will be successful.

Brief history of artificial neural networks
In 1943, neuroscientists came up with the concept of an artificial neural network as a proof-of-concept attempt to mimic how biological neurons worked. Ten years later, psychologist Frank Rosenblatt further evolved the idea by creating a single-layer neural network for supervised learning called the Perceptron.

It could “learn” from good examples of data that would train the network, and then the trained network could apply that learning to new data it had never seen before.

However, the Perceptron started to quickly show its limitations to handle certain kinds of problems, most notably “nonlinear functions.” In 1986, AI researchers published research that detailed that “hidden layers” of neurons can be used to solve the fundamental problems faced by earlier iterations of Perceptrons, especially when trained on large amounts of data. By 2006, researchers found a formula that worked: deep learning neural nets.

The evolution of big data, GPU processors and cloud computing rekindled interest in neural networks. Neural networks power many of the major AI-enabled systems such as translation services, facial and image recognition, and voice assistants.

How to create and train a neural network

At a fundamental level, artificial neural networks are approaches to machine learning modeled after the biological activity of the human brain. Neural nets are made up of an input layer that receives data, one or more hidden layers that process the data and an output layer that provides one or more data points based on the function of the network.

Neurons are connected in various configurations. Each connection has a weight that contributes to the “activation” of another neuron, which is also adjusted by a bias and activation function to trigger further neurons in the network.

The entire network is then trained using input data of known value (labeled data), with the various errors in the network corrected after many interactions. After training sufficiently, the network has “learned” the relationship between inputs and outputs that can then be applied to future, unknown data.

At a foundational level, neural nets start from some untrained or pretrained state and the weights are then adjusted by training the network to make the output more accurate. While this may sound simple, adjusting the weights can take lots of time and compute power to get correct.

Each hidden layer in the neural net detects a specific class of features. For example, if we take a neural net built to detect cats, the first layer might detect some level of abstraction in the image. The next layers then detect a further level of abstraction. With enough training data, the neural net will adjust its weights to be able to detect if the image presented is a cat or not.

Now, you’ve built a model that identifies cats; a binary classifier that will give you a probability of whether the image is a cat or not. However, since no one trained the model on bird images, or car images, or flower images, the entire process needs to start over to detect each different object accurately. The model, in other words, can only tell if an image is a cat or not a cat. It can’t determine what the “not a cat” image actually is.

There are approaches called transfer learning that take a neural network that has learned one thing and apply it to another similar training task, but neural networks are generally narrow in what they have learned.

Popular neural network training methods
While it may seem that neural networks are just one thing, they are a family of algorithms optimized for different kinds of patterns. In fact, there are many ways to connect neurons together to form neural networks.

Feedforward neural network (FFNN)
One of the most common forms is the feedforward neural network (FFNN), in which a neural network starts from inputs and works its way to outputs without any loops or other interesting convolutions. FFNNs are simple and can’t handle more complex needs.

Since information only moves from the input layer to the output layer, hence the name “feedforward,” its uses are largely limited to simple supervised learning applications. This includes image recognition, image classification and speech recognition use cases — essentially, applications that don’t require the model to look at information outside a linear time series.