In this article by Prateek Joshi, author of book Artificial Intelligence with Python, we are going to learn about artificial neural networks. We will start with an introduction to artificial neural networks and the installation of the relevant library. We will discuss perceptron and how to build a classifier based on that. We will learn about single layer neural networks and multilayer neural networks.
(For more resources related to this topic, see here.)
Introduction to artificial neural networks
One of the fundamental premises of Artificial Intelligence is to build machines that can perform tasks that require human intelligence. The human brain is amazing at learning new things. Why not use the model of the human brain to build a machine? An artificial neural network is a model designed to simulate the learning process of the human brain.
Artificial neural networks are designed such that they can identify the underlying patterns in data and learn from them. They can be used for various tasks such as classification, regression, segmentation, and so on. We need to convert any given data into the numerical form before feeding it into the neural network. For example, we deal with many different types of data including visual, textual, time-series, and so on. We need to figure out how to represent problems in a way that can be understood by artificial neural networks.
Building a neural network
The human learning process is hierarchical. We have various stages in our brain’s neural network and each stage corresponds to a different granularity. Some stages learn simple things and some stages learn more complex things. Let’s consider an example of visually recognizing an object. When we look at a box, the first stage identifies simple things like corners and edges. The next stage identifies the generic shape and the stage after that identifies what kind of object it is. This process differs for different tasks, but you get the idea! By building this hierarchy, our human brain quickly separates the concepts and identifies the given object.
To simulate the learning process of the human brain, an artificial neural network is built using layers of neurons. These neurons are inspired from the biological neurons we discussed in the previous paragraph. Each layer in an artificial neural network is a set of independent neurons. Each neuron in a layer is connected to neurons in the adjacent layer.
Training a neural network
If we are dealing with N-dimensional input data, then the input layer will consist of N neurons. If we have M distinct classes in our training data, then the output layer will consist of M neurons. The layers between the input and output layers are called hidden layers. A simple neural network will consist of a couple of layers and a deep neural network will consist of many layers.
Consider the case where we want to use a neural network to classify the given data. The first step is to collect the appropriate training data and label it. Each neuron acts as a simple function and the neural network trains itself until the error goes below a certain value. The error is basically the difference between the predicted output and the actual output. Based on how big the error is, the neural network adjusts itself and retrains until it gets closer to the solution. You can learn more about neural networks here: http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html.
We will be using a library called NeuroLab . You can find more about it here: https://pythonhosted.org/neurolab. You can install it by running the following command on your Terminal:
$ pip3 install neurolabOnce you have installed it, you can proceed to the next section.
Building a perceptron based classifier
Perceptron is the building block of an artificial neural network. It is a single neuron that takes inputs, performs computation on them, and then produces an output. It uses a simple linear function to make the decision. Let’s say we are dealing with an N-dimension input datapoint. A perceptron computes the weighted summation of those N numbers and it then adds a constant to produce the output. The constant is called the bias of the neuron. It is remarkable to note that these simple perceptrons are used to design very complex deep neural networks. Let’s see how to build a perceptron based classifier using NeuroLab.
Create a new python file and import the following packages:
importnumpy as np
importmatplotlib.pyplot as plt
importneurolab as nlLoad the input data from the text file data_perceptron.txt provided to you. Each line contains space separated numbers where the first two numbers are the features and the last number is the label:
# Load input data
text = np.loadtxt(‘data_perceptron.txt’)Separate the text into datapoints and labels:
# Separate datapoints and labels
data = text[:, :2]
labels = text[:, 2].reshape((text.shape[0], 1))Plot the datapoints:
# Plot input data
plt.figure()
plt.scatter(data[:,0], data[:,1])
plt.xlabel(‘Dimension 1’)
plt.ylabel(‘Dimension 2’)
plt.title(‘Input data’)Define the maximum and minimum values that each dimension can take:
# Define minimum and maximum values for each dimension
dim1_min, dim1_max, dim2_min, dim2_max = 0, 1, 0, 1Since the data is separated into two classes, we just need one bit to represent the output. So the output layer will contain a single neuron.
# Number of neurons in the output layer
num_output = labels.shape[1]We have a dataset where the datapoints are 2-dimensional. Let’s define a perceptron with 2 input neurons where we assign one neuron for each dimension.
# Define a perceptron with 2 input neurons (because we
# have 2 dimensions in the input data)
dim1 = [dim1_min, dim1_max]
dim2 = [dim2_min, dim2_max]
perceptron = nl.net.newp([dim1, dim2], num_output)Train the perceptron with the training data:
# Train the perceptron using the data
error_progress = perceptron.train(data, labels, epochs=100, show=20, lr=0.03)Plot the training progress using the error metric:
# Plot the training progress
plt.figure()
plt.plot(error_progress)
plt.xlabel(‘Number of epochs’)
plt.ylabel(‘Training error’)
plt.title(‘Training error progress’)
plt.grid()
plt.show()The full code is given in the file perceptron_classifier.py. If you run the code, you will get two output figures. The first figure indicates the input datapoints:
The second figure represents the training progress using the error metric:
As we can observe from the preceding figure, the error goes down to 0 at the end of fourth epoch.
Constructing a single layer neural network
A perceptron is a good start, but it cannot do much. The next step is to have a set of neurons act as a unit to see what we can achieve. Let’s create a single neural network that consists of independent neurons acting on input data to produce the output.
Create a new python file and import the following packages:
importnumpy as np
importmatplotlib.pyplot as plt
importneurolab as nlLoad the input data from the file data_simple_nn.txt provided to you. Each line in this file contains 4 numbers. The first two numbers form the datapoint and the last two numbers are the labels. Why do we need to assign two numbers for labels? Because we have 4 distinct classes in our dataset, so we need two bits represent them.
# Load input data
text = np.loadtxt(‘data_simple_nn.txt’)Separate the data into datapoints and labels:
# Separate it into datapoints and labels
data = text[:, 0:2]
labels = text[:, 2:]Plot the input data:
# Plot input data
plt.figure()
plt.scatter(data[:,0], data[:,1])
plt.xlabel(‘Dimension 1’)
plt.ylabel(‘Dimension 2’)
plt.title(‘Input data’)Extract the minimum and maximum values for each dimension (we don’t need to hardcode it like we did in the previous section):
# Minimum and maximum values for each dimension
dim1_min, dim1_max = data[:,0].min(), data[:,0].max()
dim2_min, dim2_max = data[:,1].min(), data[:,1].max()Define the number of neurons in the output layer:
# Define the number of neurons in the output layer
num_output = labels.shape[1]Define a single layer neural network using the above parameters:
# Define a single-layer neural network
dim1 = [dim1_min, dim1_max]
dim2 = [dim2_min, dim2_max]
nn = nl.net.newp([dim1, dim2], num_output)
Train the neural network using training data:
# Train the neural network
error_progress = nn.train(data, labels, epochs=100, show=20, lr=0.03)Plot the training progress:
# Plot the training progress
plt.figure()
plt.plot(error_progress)
plt.xlabel(‘Number of epochs’)
plt.ylabel(‘Training error’)
plt.title(‘Training error progress’)
plt.grid()
plt.show()Define some sample test datapoints and run the network on those points:
The full code is given in the file simple_neural_network.py. If you run the code, you will get two figures. The first figure represents the input datapoints:
# Run the classifier on test datapoints
print(‘nTest results:’)
data_test = [[0.4, 4.3], [4.4, 0.6], [4.7, 8.1]]
for item in data_test:
print(item, ‘-->‘, nn.sim([item])[0])
The second figure shows the training progress:
You will see the following printed on your Terminal:
If you locate those test datapoints on a 2D graph, you can visually verify that the predicted outputs are correct.
Constructing a multilayer neural network
In order to enable higher accuracy, we need to give more freedom the neural network. This means that a neural network needs more than one layer to extract the underlying patterns in the training data. Let’s create a multilayer neural network to achieve that.
Create a new python file and import the following packages:
importnumpy as np
importmatplotlib.pyplot as plt
importneurolab as nlIn the previous two sections, we saw how to use a neural network as a classifier. In this section, we will see how to use a multilayer neural network as a regressor. Generate some sample datapoints based on the equation y = 3x^2 + 5 and then normalize the points:
# Generate some training data
min_val = -15
max_val = 15
num_points = 130
x = np.linspace(min_val, max_val, num_points)
y = 3 * np.square(x) + 5
y /= np.linalg.norm(y)Reshape the above variables to create a training dataset:
# Create data and labels
data = x.reshape(num_points, 1)
labels = y.reshape(num_points, 1)Plot the input data:
# Plot input data
plt.figure()
plt.scatter(data, labels)
plt.xlabel(‘Dimension 1’)
plt.ylabel(‘Dimension 2’)
plt.title(‘Input data’)Define a multilayer neural network with 2 hidden layers. You are free to design a neural network any way you want. For this case, let’s have 10 neurons in the first layer and 6 neurons in the second layer. Our task is to predict the value, so the output layer will contain a single neuron.
# Define a multilayer neural network with 2 hidden layers;
# First hidden layer consists of 10 neurons
# Second hidden layer consists of 6 neurons
# Output layer consists of 1 neuron
nn = nl.net.newff([[min_val, max_val]], [10, 6, 1])Set the training algorithm to gradient descent:
# Set the training algorithm to gradient descent
nn.trainf = nl.train.train_gdTrain the neural network using the training data that was generated:
# Train the neural network
error_progress = nn.train(data, labels, epochs=2000, show=100, goal=0.01)Run the neural network on the training datapoints:
# Run the neural network on training datapoints
output = nn.sim(data)
y_pred = output.reshape(num_points)Plot the training progress:
# Plot training error
plt.figure()
plt.plot(error_progress)
plt.xlabel(‘Number of epochs’)
plt.ylabel(‘Error’)
plt.title(‘Training error progress’)Plot the predicted output:
# Plot the output
x_dense = np.linspace(min_val, max_val, num_points * 2)
y_dense_pred = nn.sim(x_dense.reshape(x_dense.size,1)).reshape(x_dense.size)
plt.figure()
plt.plot(x_dense, y_dense_pred, ‘-’, x, y, ‘.’, x, y_pred, ‘p’)
plt.title(‘Actual vs predicted’)
plt.show()The full code is given in the file multilayer_neural_network.py. If you run the code, you will get three figures. The first figure shows the input data:
The second figure shows the training progress:
The third figure shows the predicted output overlaid on top of input data:
The predicted output seems to follow the general trend. If you continue to train the network and reduce the error, you will see that the predicted output will match the input curve even more accurately.
You will see the following printed on your Terminal:
Summary
In this article, we learnt more about artificial neural networks. We discussed how to build and train neural networks. We also talked about perceptron and built a classifier based on that. We also learnt about single layer neural networks as well as multilayer neural networks.
Resources for Article:
Further resources on this subject:
- Training and Visualizing a neural network with R [article]
- Implementing Artificial Neural Networks with TensorFlow [article]
- How to do Machine Learning with Python [article]