3 Convolutional Neural Networks

Convolutional Neural Networks: Image classification of brain tumor MRI data

In this practical session we will use the following dataset: https://www.kaggle.com/datasets/sartajbhuvaji/brain-tumor-classification-mri/data. This is a dataset of magnetic resonance images of the brain of patients with the following (or absence thereof) conditions: - glioma - meningioma - pituitary tumor - healthy brain.

Exercise 1

Write the code that you will use to load the training and test data and build the two datasets. The images are found in the brain_mri_data folder and they are already organized in train and test subfolders. Use python’s os or pathlib libraries to iterate over the folders and access the data and use the open function from Python Imaging Library (PIL) to open the images.

Each dataset must be a list of tuples, with each tuple having two values: an image in torch tensor format (type float32), and a label, which must be a 4-dimensional torch tensor of type float32 where [1,0,0,0] is the first class (glioma), [0,1,0,0] is the second class (meningioma), [0,0,1,0] is the third class (pitutary tumor) and [0,0,0,1] is the fourth class (healthy brain). Since the images are of different sizes, you need to use the .resize() method from PIL to resize them so that they are all of shape (128, 128).

Now convert the images to numpy arrays. If you print the shape you will see that it is 128x128x3, the 3 corresponding to the 3 channels in an RGB image. In PyTorch we need the channels to be the first dimension. Before converting the image to the final torch tensor format, use np.rollaxis(img, -1) to roll the last axis to the front and obtain an image of 3x128x128, then convert it to a torch tensor of type float32.

Finally, standardize each image by subtracting a mean of 63.64 and dividing by a standard deviation of 54.89 calculated from the images in the training set. Store the training and test sets in variables called train_set and test_set.

import os

import numpy as np
import torch
from PIL import Image


tumor_types = ['glioma_tumor', 'no_tumor', 'pituitary_tumor', 'meningioma_tumor']
splits = ['Testing', 'Training']
# dataset = ([], []) / uncomment this line if you want to use tuple unpacking to build your datasets
train_set, test_set = [], []

for tumor in tumor_types:
    for split in splits:
        dir_path = f'brain_mri/{split}/{tumor}'
        for img in os.listdir(dir_path):
            if img.endswith('.jpg'):
                img_pil = Image.open(f'{dir_path}/{img}').resize((128, 128))
                image = (torch.tensor(np.rollaxis(np.array(img_pil), -1), dtype=torch.float32) - 63.64) / 54.89
                label = torch.zeros(4, dtype=torch.float32)
                label[tumor_types.index(tumor)] = 1
                if split == 'Testing':
                    test_set.append((image, label))
                elif split == 'Training':
                    train_set.append((image, label))
      # write your code here
      # loading the images might somewhere between 30 seconds and 3 minutes

# train_set, test_set = dataset / uncomment this line if you want to use tuple unpacking to build your datasets

Exercise 2

Build your model. It must have the following structure:

Use ReLU as the activation function (F.relu(x)) after each convolutional and linear layer except the last one, where you must use SoftMax. You can read how to use batch normalization in PyTorch’s documentation: https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html

After applying the adaptive average pooling in the forward method, you must reshape the tensor to have shape (-1, 1024). Using -1 when you reshape tensor will infer the first dimension from the batch size. The output should be a tensor of shape (batch size, 4) that you can compare with the label, where 4 corresponds to the number of classes.

Pytorch has a tutorial on how to train a classifier with convolutional layers:

https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

If at any moment you are in doubt about how the information flows, print the dimensions of the vector x at key steps in the forward method with print(x.shape). This will help you understand the dimensions of your data as you apply the transformations.

trainloader = DataLoader(train_set, batch_size=16, shuffle=True)
testloader = DataLoader(test_set)
len(train_set)
nn.Conv2d(input_channels, output_channels, kernel_size, stride, padding)
x = torch.rand(100, 3, 128, 128)
conv1 = nn.Conv2d(3, 16, kernel_size=5, padding=2)
conv2 = nn.Conv2d(16, 32, 3, padding=1)
maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
conv3 = nn.Conv2d(32, 64, 3, padding=1)
conv4 = nn.Conv2d(64, 64, 3, padding=1)
avg_pool = nn.AdaptiveAvgPool2d(4)
linear_1 = nn.Linear(1024, 32)
linear_2 = nn.Linear(32, 4)


print(x.shape)
x = F.relu(conv1(x))
print(x.shape)
x = F.relu(conv2(x))
print(x.shape)
x = maxpool(x)
print(x.shape)
x = F.relu(conv3(x))
print(x.shape)
x = F.relu(conv4(x))
print(x.shape)
x = maxpool(x)
print(x.shape)
x = avg_pool(x)
print(x.shape)
x = x.reshape(100, 1024)
print(x.shape)
x = linear_1(x)
print(x.shape)
x = linear_2(x)
print(x.shape)
64 * 4 * 4
x = torch.rand(64, 3, 128, 128)
print(x.shape)





conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)
conv2 = nn.Conv2d(16, 32, kernel_size=5, padding=2)
maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
conv3 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
linear1 = nn.Linear(256, 64)
linear_2 = nn.Linear(64, 4)
avgpool = nn.AdaptiveAvgPool2d(2)
batch_norm_1 = nn.BatchNorm2d(16)
batch_norm_2 = nn.BatchNorm2d(32)
batch_norm_3 = nn.BatchNorm2d(64)
x = F.relu(conv1(x))
x = batch_norm_1(x)
x = F.relu(conv2(x))
x = batch_norm_2(x)
x = maxpool(x)
x = F.relu(conv3(x))
x = batch_norm_3(x)
x = avgpool(x)
x = x.reshape(-1, 256)
x = F.relu(linear1(x))
x = linear_2(x)
x.shape
my_array.reshape(4,6)
import torch
import torch.nn as nn
import torch.nn.functional as F

class ConvolutionalNN(nn.Module):
    def __init__(self):
        super(ConvolutionalNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, padding=2)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        self.conv4 = nn.Conv2d(64, 64, 3, padding=1)
        self.avg_pool = nn.AdaptiveAvgPool2d(4)
        self.linear_1 = nn.Linear(1024, 32)
        self.linear_2 = nn.Linear(32, 4)

    def forward(self, x):
        x = F.relu(conv1(x))
        x = F.relu(conv2(x))
        x = maxpool(x)

        x = F.relu(conv3(x))
        x = F.relu(conv4(x))
        x = maxpool(x)

        x = avg_pool(x)
        x = x.reshape(-1, 1024)

        x = F.relu(linear_1(x))
        x = F.softmax(linear_2(x), dim=1)

        return x

    def get_number_of_parameters(self):
        return sum(p.numel() for p in self.parameters())

Exercise 3

Initialize your model by creating an instance of the ConvolutionalNN class and print the number of parameters.

device = 'cuda' if torch.cuda.is_available() else 'cpu'

model = ConvolutionalNN().to(device)

print('model number of parameters:') # write the code to print the number of parameters of the model
model(torch.randn(1,3,128,128)).shape

Exercise 4

Define the DataLoader, optimizer and loss function. Use a batch size of 32 for the training loader with shuffle=True and a batch size of 1 for the test loader with shuffle=False. Name the dataloader variables train_loader and test_loader. We will ignore PyTorch’s Dataset class for now.

Use cross entropy as the loss function and Adam as optimizer with a learning rate of 5e-4 and weight decay of 1e-5.

from torch.utils.data import DataLoader
from torch.nn import CrossEntropyLoss
from torch.optim import Adam

# define the train loader and test loader

loss_function = CrossEntropyLoss()

optimizer = Adam(model.parameters(), lr=5e-4, weight_decay=1e-5)

Exercise 5

Train the model for 60 epochs. What is the best accuracy that you obtain in the test set?

epochs = 60
n_samples_train = len(train_loader)
n_samples_test = len(test_loader)
training_loss_per_epoch, test_loss_per_epoch, test_accuracy_per_epoch = [], [], []
best_accuracy = - torch.inf
for epoch in range(epochs):
    training_loss = 0
    for data, labels in train_loader:
        predict = model(data.to(device))
        loss = loss_function(labels.to(device), predict)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        training_loss += (loss.item() / n_samples_train)
    test_loss, correct, total = (0, 0, 0)
    for data, label in test_loader:
        predict = model(data.to(device))
        loss = loss_function(label.to(device), predict)
        test_loss += (loss.item() / n_samples_test)
        if torch.argmax(predict) == torch.argmax(label):
            correct += 1
        total += 1
    accuracy = correct/total*100
    if accuracy > best_accuracy:
        best_accuracy = accuracy
    training_loss_per_epoch.append(training_loss)
    test_loss_per_epoch.append(test_loss)
    test_accuracy_per_epoch.append(accuracy)
    print('''Epoch {}: training loss: {:.3f}, test loss: {:.3f}, test accuracy: {:.2f}%
          '''.format(epoch+1, training_loss, test_loss, accuracy))
print(f'\n Best test accuracy: {best_accuracy}')

Exercise 6

Print the training loss per epoch and the test loss per epoch. What do you observe?

import matplotlib.pyplot as plt

fig, ax = plt.subplots(1, 2, figsize=(12,5))
ax[0].plot(test_loss_per_epoch, c='r')
ax[0].set_xlabel('Epochs', fontsize=16)
ax[0].set_ylabel('Loss', fontsize=16)
ax[1].plot(test_accuracy_per_epoch, c='g')
ax[1].set_xlabel('Epochs', fontsize=16)
ax[1].set_ylabel('Accuracy %', fontsize=16)

Exercise 7

Go back to the neural network that you designed in exercise 2. Using a model with a maximum of 50.000 parameters, train a neural network to obtain the maximum possible accuracy in the test set. Use a maximum of 150 epochs with a learning rate of 5e-4 and a batch size of 32. You can also tune the weight_decay hyperparameter in the Adam optimizer to have the value of your choice. Additionally, you can try to resize the images to have different height and width and see if this improves the performance of the model. Who will achieve the maximum accuracy? I have witnessed test accuracies of up to ~70% with this setup.

You can use Dropout, Batch Normalization and Maxpooling operations which don’t add any parameters to your model and might improve generalization and therefore the accuracy in the test set.

https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html

https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html

https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html