2 Mlp Xor Problem Solutions
MultiLayer perceptron (MLP)
In this practical we will implement a 2 layer perceptron following the tutorial from James Loy, Author of Neural Network Projects with Python to solve the XOR problem. You can find the github repository with the code here.
We will create a class to represent our model and we will store the weights and biases as attributes. Additionally, our class will have methods to compute the forward and back propagation.
To start, we need a couple of helper functions such as the sigmoid function and the loss function that we will define outside of the class.
import matplotlib.pyplot as plt
import numpy as npExercise 1 - Define a function called sigmoid where you compute the sigmoid function \(\sigma(x)=\frac{1}{1+e^{-x}}\) and another function called mean_squared_error where you compute the mean squared error: \(MSE=\frac{1}{n}\sum_{i=0}^n(y_i-\hat{y}_i)^2\)
def sigmoid(x):
return 1/(1 + np.exp(-x))
def mean_squared_error(y, y_pred):
return np.mean((y - y_pred)**2)Exercise 2 - Create a class called NeuralNetwork. Your init method should have three arguments: n_input_features, hidden_size and output_shape, all of them integers. The first argumnent defines the number of input features to your neural network, which depends on the data that we use to train the network. In this case we have two inputs, both of which can be either 0 or 1 (see the XOR table below). The second argument is the number of hidden layers, in this case 2. Finally, the third argument the shape of the output, in this case 1 since it is a single number (0 or 1). In addition, you should define 6 attributes in the constructor: weights_1, a n_input_featuresxhidden_size matrix representing the connections from a 2 feature input layer to a 2 neuron hidden layer; bias_1, a hidden_sizex1 vector representing the bias of the two neurons in the hidden layer; weights_2, a hidden_sizexoutput_shape matrix that represents the two connections from the hidden layer to the output; and bias_2, a output_shapex1 vector that represents the bias of the single neuron output layer. The last two attributes, loss_per_epoch and epochs will be used to monitor the training process of the neural network.
class NeuralNetwork:
def __init__(self, n_input_features = 2, hidden_size = 2, output_shape = 1):
self.weights_1 = np.random.random((n_input_features, hidden_size))
self.bias_1 = np.random.random((hidden_size, 1))
self.weights_2 = np.random.random((hidden_size, output_shape))
self.bias_2 = np.random.rand(output_shape)
self.loss_per_epoch = []
self.epochs = 0
def forward(self, X):
layer_1_output = sigmoid(np.dot(self.weights_1.T, X.T) + self.bias_1)
output = sigmoid(np.dot(self.weights_2.T, layer_1_output) + self.bias_2)
return output
def update_weights(self, X, y, learning_rate):
layer_1_output = sigmoid(np.dot(self.weights_1.T, X.T) + self.bias_1)
output = sigmoid(np.dot(self.weights_2.T, layer_1_output) + self.bias_2)
grad_2 = -2 * (y - output) * output * (1 - output)
self.bias_2 -= np.mean(grad_2) * learning_rate
self.weights_2 -= np.dot(layer_1_output, grad_2.T) * learning_rate
grad_1 = grad_2 * self.weights_2 * layer_1_output * (1 - layer_1_output)
self.bias_1 -= np.mean(grad_1, axis=1).reshape(-1,1) * learning_rate
self.weights_1 -= np.dot(X.T, grad_1.T)
def train(self, X, y, epochs, learning_rate = 0.01):
for epoch in range(epochs):
y_pred = self.forward(X)
self.loss_per_epoch.append(mean_squared_error(y, y_pred))
self.update_weights(X, y, learning_rate)
self.epochs += epochs
def predict(self, X):
return np.round(self.forward(X))
def plot_loss(self):
plt.plot(self.loss_per_epoch)
plt.xlabel('Epoch', fontsize=14)
plt.ylabel('Loss (MSE)', fontsize=14)Exercise 3 - Fill the forward, compute_gradients, train & predict methods. The forward computation is defined by
\[\begin{equation} \hat{y} = \sigma(W_2^T(\sigma(W_1^TX+b_1))+b_2), \end{equation}\]
where \(W_1\) is the previously defined attribute weights_1, \(b_1\) is bias_1, \(W_2\) is weights_2, \(b_2\) is bias_2 and \(\sigma\) is the sigmoid function.
The compute_gradients method should return 4 values, which are the partial derivatives of the loss function with respect to the parameters of the model represented by the attribute matrices. Compute the gradient \(\nabla L\) in four steps:
\[\begin{equation} \frac{\partial L}{\partial W_2} = -2[(y - \hat{y})\hat{y}(1 - \hat{y})]X_0^T \end{equation}\]
\[\begin{equation} \frac{\partial L}{\partial b_2} = -2(y - \hat{y})·\hat{y}(1 - \hat{y}) \end{equation}\]
\[\begin{equation} \frac{\partial L}{\partial W_1} = -2(y - \hat{y})·\hat{y}(1 - \hat{y})·W_2X^T\odot(X_0(1-X_0)) \end{equation}\]
\[\begin{equation} \frac{\partial L}{\partial b_1} = -2(y - \hat{y})·\hat{y}(1 - \hat{y})·W_2\odot(X_0(1-X_0)) \end{equation}\]
where \(X_0=\sigma(W_1^TX+b_1)\) is the output of the first layer and \(\odot\) is the Hadamard (element-wise) product.
The train method should use the input data to calculate the gradients and update the parameters of the model. In addition to X and y, it has two more arguments: the number of epochs and the learning rate. You should add the number of epochs to the value stored in the attribute epochs, and at each training step you shoud add the current loss to the attribute loss_per_epoch.
Finally, the predict method should call the forward method and round the output to either 0 or 1. You should round the output such that if \(\hat{y}<0.5 \rightarrow 0\), \(\hat{y}\ge0.5 \rightarrow 1\)
We are just missing the training data, which in this case is simply given by the XOR table:
| Input 1 | Input 2 | Output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
Exercise 4 - Train the model for 2500 epochs and learning rate 1 with the given data and plot the loss per epoch.
np.random.seed(42)
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0, 1, 1, 0])
epochs = 2500
learning_rate = 1
model = NeuralNetwork()
model.train(X, y, epochs, learning_rate)
model.plot_loss()Exercise 5 - Call the forward method to produce a raw output (without rounding) and print it
print(model.forward(X))Exercise 6 - Print the final predictions by calling the predict function and compare them to the values on the table.
y_pred = model.predict(X)
print('inputs:', X)
print('outputs:', y_pred)Bonus: Sleep Disorder Data
Load the data from the sleep_disorder_data.csv file with Pandas. This time we will use all of the non-categorical variables in the dataset. Select the columns ‘Age’, ‘Sleep Duration’, ‘Quality of Sleep’, ‘Physical Activity Level’, ‘Stress Level’, ‘Heart Rate’ and ‘Dailty Steps’. Standardize the columns. Select the ‘Sleep Disorder’ column and convert the None values to the vector [1, 0, 0], Sleep Apnea to [0, 1, 0] and Insomnia to [0, 0, 1]. Build a neural network using the NeuralNetwork class you defined above with 7 input features, 8 hidden neurons in the hidden layer and 3 neurons in the output. Train it and show the results as in exercises 4 and 5
import pandas as pd
# your code goes here