Friday, August 25, 2017

Training OverlapSegmentationNet with gpu acceleration on a Ubuntu powered laptop.

Set-up

Harnessing the gpu 1 : nvidia driver

It is  supposed that a fresh ubuntu 16.04 is set-up.
Install Nvidia proprietary driver from the system parametres menu:
The NVidia installed files, with synaptic package manager, are:


Nvidia acceleration can be switched on graphically after having installed nvidia-settings :

 Check driver installation from a terminal by starting glxgears:

When NVIDIA mode is selected, glxgears performances should increase.

Harnessing the gpu 2: install CUDA 8, cudnn 5.1

CUDA8 was installed from a .deb file available from nvidia's site:


Check cuda installation using the "./deviceQuerry" method (see the end of the post).

The Cudnn 5.1 library was installed by copying the files at the right places (Installing cudnn 6 from .deb archive wasn't a good idea. )

 

Install Tensorflow and Keras 

There's different ways to install tensorflow. Here, both tensorflow with gpu support and keras were installed  in a virtual environnement with pip.

Tensorflow 1.2.1 and Keras 2.0.6 were installed .

h5py was installed from pip too.
Don't forget at the end from a terminal to :

$source .bashrc

Installation of OverlapSegmentationNet:

OverlapSegmentationNet is an implementation of UNet by Dr Hu .

The image-segmentation-chromosomes directory contains:
$ ls
code  images    README.md

Dataset:

The first version of the low resolution dataset was used. Download it and move it into the code directory:
$ ls
create_histogram_visualisation.ipynb  processInputImages.py~
Explore-data.ipynb                    __pycache__
Explore-model2.ipynb                  sub-trainModel.py
histogram.png                         trainModel.py
LowRes_13434_overlapping_pairs.h5     trainModel.py~
models                                utilities.py
OverlapSegmentationNet.py             utilities.py~
OverlapSegmentationNet.pyc            utilities.pyc
preprocessing-jp.ipynb                xdata_88x88.npy
processInputImages.py                 ydata_88x88_0123_onehot.npy

In a terminal,  run processInputImages.py as follow:
$python processInputImages.py
The code will generate the two files highlighted in green above.

Training

Switch into gpu mode with (then logout / login):


Activate the proper virtual environnement. Mine was called tfgpu, so for my computer, it is:
$source VirtualEnv/tfgpu/bin/activate
Edit the code to set-up the desired number of epoch
From a terminal, run trainModel.py. For example in my computer, it is:

(tfgpu) jeanpat@WA50SHQ:~/image_segmentation_chromosomes/code$ python trainModel.py
If the gpu is properly used by tensorflow the terminal should yield something like:


The usage of the gpu is mentionned, furthermore the cpu usage is not stucked on 100%.

On a modest nvidia GT740M gpu, 14 epochs were run for 8~9 hours.