Training OverlapSegmentationNet with GPU acceleration on google-colaboratory

 OverlapSegmentationNet is a python implementation of Unet intented to resolve pairs of overlapping chromosomes by semantic segmentation (See previous post).


OverlapSegmentationNet can now be trained on the google colaboratory platform with GPU acceleration.

The whole files and data are available from google drive. Wouldn't have been able to set-up without the following article.

Training OverlapSegmentationNet with gpu acceleration on a Ubuntu powered laptop.

Set-up

Harnessing the gpu 1 : nvidia driver

It is  supposed that a fresh ubuntu 16.04 is set-up.

Install Nvidia proprietary driver from the system parametres menu:

The NVidia installed files, with synaptic package manager, are:

Nvidia acceleration can be switched on graphically after having installed nvidia-settings :

 Check driver installation from a terminal by starting glxgears:

When NVIDIA mode is selected, glxgears performances should increase.

Harnessing the gpu 2: install CUDA 8, cudnn 5.1

CUDA8 was installed from a .deb file available from nvidia's site:

Check cuda installation using the "./deviceQuerry" method (see the end of the post).

The Cudnn 5.1 library was installed by copying the files at the right places (Installing cudnn 6 from .deb archive wasn't a good idea. )

 

Install Tensorflow and Keras 

There's different ways to install tensorflow. Here, both tensorflow with gpu support and keras were installed  in a virtual environnement with pip.

Tensorflow 1.2.1 and Keras 2.0.6 were installed .

h5py was installed from pip too.

Don't forget at the end from a terminal to :

$source .bashrc

Installation of OverlapSegmentationNet:

OverlapSegmentationNet is an implementation of UNet by Dr Hu .

Download image-segmentation-chromosomes or clone it from repository with git.

The image-segmentation-chromosomes directory contains:

$ ls
code  images    README.md

Dataset:

The first version of the low resolution dataset was used. Download it and move it into the code directory:

$ ls

create_histogram_visualisation.ipynb  processInputImages.py~

Explore-data.ipynb                    __pycache__

Explore-model2.ipynb                  sub-trainModel.py

histogram.png                         trainModel.py

LowRes_13434_overlapping_pairs.h5     trainModel.py~

models                                utilities.py

OverlapSegmentationNet.py             utilities.py~

OverlapSegmentationNet.pyc            utilities.pyc

preprocessing-jp.ipynb                xdata_88x88.npy

processInputImages.py                 ydata_88x88_0123_onehot.npy

In a terminal,  run processInputImages.py as follow:

$python processInputImages.py

The code will generate the two files highlighted in green above.

Training

Switch into gpu mode with (then logout / login):

 

Activate the proper virtual environnement. Mine was called tfgpu, so for my computer, it is:

$source VirtualEnv/tfgpu/bin/activate

Edit the code to set-up the desired number of epoch

From a terminal, run trainModel.py. For example in my computer, it is:

(tfgpu) jeanpat@WA50SHQ:~/image_segmentation_chromosomes/code$ python trainModel.py

If the gpu is properly used by tensorflow the terminal should yield something like:

The usage of the gpu is mentionned, furthermore the cpu usage is not stucked on 100%.

On a modest nvidia GT740M gpu, 14 epochs were run for 8~9 hours.