Building tensorflow 1.10 with cuda 9.2 support on ubuntu 16.04

From version 1.7, tensorflow binary available from anaconda repository, is build with AVX support. To run tensorflow on old cpu missing AVX instructions set, such Xeon E5520, tensorflow must be build from source.

Tensorflow can be build on ubuntu 18.04. Here tensorflow 1.10 will be build for ubuntu 16.04 with CUDA 9.2/cuDNN 7.14 support. Building tensorflow from source relies on the installation of several softwares. Once installed, one can try to run the configure script:

./configure

After some trials, the following configuration seems to succeed for building with CUDA support. From a terminal launch the configure script (here in a virtual environment):

$ ./configure
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.15.0 installed.
Please specify the location of python. [Default is /home/jeanpat/anaconda3/envs/DeepFish/bin/python]:


Found possible Python library paths:
  /home/jeanpat/anaconda3/envs/DeepFish/lib/python3.6/site-packages
Please input the desired Python library path to use.  Default is [/home/jeanpat/anaconda3/envs/DeepFish/lib/python3.6/site-packages]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: y
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon AWS Platform support? [Y/n]: n
No Amazon AWS Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Apache Kafka Platform support? [Y/n]: n
No Apache Kafka Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: y
XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: y
GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: y
VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n
No OpenCL SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 9.0]: 9.2


Please specify the location where CUDA 9.2 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:


Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]: 7.14


Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:


Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.

Please specify the location where TensorRT is installed. [Default is /usr/lib/x86_64-linux-gnu]:


Please specify the NCCL version you want to use. If NCCL 2.2 is not installed, then you can use version 1.3 that can be fetched automatically but it may have worse performance with multiple GPUs. [Default is 2.2]: 1.3.5


Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 5.2]:


Do you want to use clang as CUDA compiler? [y/N]: n
nvcc will be used as CUDA compiler.

Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:


Do you wish to build TensorFlow with MPI support? [y/N]: n
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
    --config=mkl             # Build with MKL support.
    --config=monolithic      # Config for mostly static monolithic build.
Configuration finished

Building with bazel failed:

According to documentation, the building step was started by typing in a terminal:

$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Unfortunately, after some hours, the build failed with the error:


./tensorflow/core/util/tensor_format.h:420:3: note: in expansion of macro 'CHECK'
   CHECK(index >= 0 && index < dimension_attributes.size())
   ^
./tensorflow/core/util/tensor_format.h: In instantiation of 'T tensorflow::GetFilterDim(tensorflow::gtl::ArraySlice<T>, tensorflow::FilterTensorFormat, char) [with T = long long int]':
./tensorflow/core/util/tensor_format.h:461:54:   required from here
./tensorflow/core/util/tensor_format.h:435:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   CHECK(index >= 0 && index < dimension_attribute.size())
                             ^
./tensorflow/core/platform/macros.h:87:47: note: in definition of macro 'TF_PREDICT_FALSE'
 #define TF_PREDICT_FALSE(x) (__builtin_expect(x, 0))
                                               ^
./tensorflow/core/util/tensor_format.h:435:3: note: in expansion of macro 'CHECK'
   CHECK(index >= 0 && index < dimension_attribute.size())
   ^
ERROR: /home/jeanpat/Developpement/Arch-TensorFlow/tensorflow/tensorflow/BUILD:576:1: Executing genrule //tensorflow:tensorflow_python_api_gen failed (Aborted): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
2018-08-22 21:44:49.248926: F tensorflow/core/framework/allocator_registry.cc:52] New registration for AllocatorFactory with name=BFCRdmaAllocator priority=101 at location tensorflow/contrib/gdr/gdr_memory_manager.cc:204 conflicts with previous registration at location tensorflow/contrib/verbs/rdma_mgr.cc:277
/bin/bash: line 1: 17921 Aborted                 (core dumped) bazel-out/host/bin/tensorflow/create_tensorflow.python_api --root_init_template=tensorflow/api_template.__init__.py --apidir=bazel-out/host/genfiles/tensorflow --apiname=tensorflow --apiversion=1 --package=tensorflow.python --output_package=tensorflow bazel-out/host/genfiles/tensorflow/__init__.py bazel-out/host/genfiles/tensorflow/app/__init__.py bazel-out/host/genfiles/tensorflow/bitwise/__init__.py bazel-out/host/genfiles/tensorflow/compat/__init__.py bazel-out/host/genfiles/tensorflow/data/__init__.py bazel-out/host/genfiles/tensorflow/debugging/__init__.py bazel-out/host/genfiles/tensorflow/distributions/__init__.py bazel-out/host/genfiles/tensorflow/dtypes/__init__.py bazel-out/host/genfiles/tensorflow/errors/__init__.py bazel-out/host/genfiles/tensorflow/feature_column/__init__.py bazel-out/host/genfiles/tensorflow/gfile/__init__.py bazel-out/host/genfiles/tensorflow/graph_util/__init__.py bazel-out/host/genfiles/tensorflow/image/__init__.py bazel-out/host/genfiles/tensorflow/io/__init__.py bazel-out/host/genfiles/tensorflow/initializers/__init__.py bazel-out/host/genfiles/tensorflow/keras/__init__.py bazel-out/host/genfiles/tensorflow/keras/activations/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/densenet/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/inception_resnet_v2/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/inception_v3/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/mobilenet/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/nasnet/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/resnet50/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/vgg16/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/vgg19/__init__.py bazel-out/host/genfiles/tensorflow/keras/applications/xception/__init__.py bazel-out/host/genfiles/tensorflow/keras/backend/__init__.py bazel-out/host/genfiles/tensorflow/keras/callbacks/__init__.py bazel-out/host/genfiles/tensorflow/keras/constraints/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/boston_housing/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/cifar10/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/cifar100/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/fashion_mnist/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/imdb/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/mnist/__init__.py bazel-out/host/genfiles/tensorflow/keras/datasets/reuters/__init__.py bazel-out/host/genfiles/tensorflow/keras/estimator/__init__.py bazel-out/host/genfiles/tensorflow/keras/initializers/__init__.py bazel-out/host/genfiles/tensorflow/keras/layers/__init__.py bazel-out/host/genfiles/tensorflow/keras/losses/__init__.py bazel-out/host/genfiles/tensorflow/keras/metrics/__init__.py bazel-out/host/genfiles/tensorflow/keras/models/__init__.py bazel-out/host/genfiles/tensorflow/keras/optimizers/__init__.py bazel-out/host/genfiles/tensorflow/keras/preprocessing/__init__.py bazel-out/host/genfiles/tensorflow/keras/preprocessing/image/__init__.py bazel-out/host/genfiles/tensorflow/keras/preprocessing/sequence/__init__.py bazel-out/host/genfiles/tensorflow/keras/preprocessing/text/__init__.py bazel-out/host/genfiles/tensorflow/keras/regularizers/__init__.py bazel-out/host/genfiles/tensorflow/keras/utils/__init__.py bazel-out/host/genfiles/tensorflow/keras/wrappers/__init__.py bazel-out/host/genfiles/tensorflow/keras/wrappers/scikit_learn/__init__.py bazel-out/host/genfiles/tensorflow/layers/__init__.py bazel-out/host/genfiles/tensorflow/linalg/__init__.py bazel-out/host/genfiles/tensorflow/logging/__init__.py bazel-out/host/genfiles/tensorflow/losses/__init__.py bazel-out/host/genfiles/tensorflow/manip/__init__.py bazel-out/host/genfiles/tensorflow/math/__init__.py bazel-out/host/genfiles/tensorflow/metrics/__init__.py bazel-out/host/genfiles/tensorflow/nn/__init__.py bazel-out/host/genfiles/tensorflow/nn/rnn_cell/__init__.py bazel-out/host/genfiles/tensorflow/profiler/__init__.py bazel-out/host/genfiles/tensorflow/python_io/__init__.py bazel-out/host/genfiles/tensorflow/quantization/__init__.py bazel-out/host/genfiles/tensorflow/resource_loader/__init__.py bazel-out/host/genfiles/tensorflow/strings/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/builder/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/constants/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/loader/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/main_op/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/signature_constants/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/signature_def_utils/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/tag_constants/__init__.py bazel-out/host/genfiles/tensorflow/saved_model/utils/__init__.py bazel-out/host/genfiles/tensorflow/sets/__init__.py bazel-out/host/genfiles/tensorflow/sparse/__init__.py bazel-out/host/genfiles/tensorflow/spectral/__init__.py bazel-out/host/genfiles/tensorflow/summary/__init__.py bazel-out/host/genfiles/tensorflow/sysconfig/__init__.py bazel-out/host/genfiles/tensorflow/test/__init__.py bazel-out/host/genfiles/tensorflow/train/__init__.py bazel-out/host/genfiles/tensorflow/train/queue_runner/__init__.py bazel-out/host/genfiles/tensorflow/user_ops/__init__.py
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 7962.287s, Critical Path: 283.59s
INFO: 7346 processes: 7346 local.
FAILED: Build did NOT complete successfully

What to do? Trying to rebuild with a minimalist configuration (keeping cuda acceleration)

./configure again

This  time with less optimizations (keep XLA and CUDA):

$ ./configure
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.15.0 installed.
Please specify the location of python. [Default is /home/jeanpat/anaconda3/envs/DeepFish/bin/python]:


Found possible Python library paths:
  /home/jeanpat/anaconda3/envs/DeepFish/lib/python3.6/site-packages
Please input the desired Python library path to use.  Default is [/home/jeanpat/anaconda3/envs/DeepFish/lib/python3.6/site-packages]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: y
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon AWS Platform support? [Y/n]: n
No Amazon AWS Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Apache Kafka Platform support? [Y/n]: n
No Apache Kafka Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: y
XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: n
GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: n
VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n
No OpenCL SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 9.0]: 9.2

Please specify the location where CUDA 9.2 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:

Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]: 7.14

Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:

Do you wish to build TensorFlow with TensorRT support? [y/N]: n
TensorRT support will be enabled for TensorFlow.

Please specify the location where TensorRT is installed. [Default is /usr/lib/x86_64-linux-gnu]:

Please specify the NCCL version you want to use. If NCCL 2.2 is not installed, then you can use version 1.3 that can be fetched automatically but it may have worse performance with multiple GPUs. [Default is 2.2]: 1.3.5

Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 5.2]:

Do you want to use clang as CUDA compiler? [y/N]: n
nvcc will be used as CUDA compiler.

Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:

Do you wish to build TensorFlow with MPI support? [y/N]: n
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:

Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
    --config=mkl             # Build with MKL support.
    --config=monolithic      # Config for mostly static monolithic build.
Configuration finished

Build

Build first failed but installing additional keras elements allowed finally to succeed

build package

pip install /tmp/tensorflow_pkg/tensorflow*.whl

Check tensorflow

From a terminal, call an ipython console (here in DeepFish virtualenv) and type:

(DeepFish) jeanpat@Dell-T5500:~/Developpement$ ipython
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:39:56)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tensorflow as tf

In [2]: tf.__version__
Out[2]: '1.10.0'
In [3]: hello = tf.constant('Hello, Tensorflow!')

In [4]: sess = tf.Session()
2018-08-23 15:37:45.588128: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-08-23 15:37:45.588830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Found device 0 with properties:
name: GeForce GTX 960 major: 5 minor: 2 memoryClockRate(GHz): 1.253
pciBusID: 0000:03:00.0
totalMemory: 3.95GiB freeMemory: 3.68GiB
2018-08-23 15:37:45.588860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1485] Adding visible gpu devices: 0
2018-08-23 15:37:45.997955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:966] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-23 15:37:45.998007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:972]      0
2018-08-23 15:37:45.998035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:985] 0:   N
2018-08-23 15:37:45.998306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1098] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3402 MB memory) -> physical GPU (device: 0, name: GeForce GTX 960, pci bus id: 0000:03:00.0, compute capability: 5.2)

In [5]: print(sess.run(hello))
b'Hello, Tensorflow!'

Modeling the Telomere length distribution with Chromosomal Segregation along Synchronous Cellular Divisions

Telomeres: structure and functions

If it is necessary to explain what the telomeres are, let's watch some lectures:

First from Elisabeth Blackburn:

 

Or from Titia de Lange:

 

Modeling Telomere Dynamic:

Levy et al. in Telomere End-replication Problem and Cell Aging, modeled the distribution of telomeric erosion with cells divisions at a single telomere. The model accounts for the decrease in somatic cell division capabilities with in-vitro passages of fibroblasts (Hayflick's limit).

Here, the telomere length of several chromosomes is modeled numerically with segregation of chromosomes in daughter cells. The model takes the 5' degradation of the CCCTAA strand into account. The aim of the simulation is to compare the distributions with QFISH data. The model shows that the telomeres length at pter and qter are correlated for a given homologous chromosome. It can be used to derive statistical tests to detect telomere length difference between homologous chromosomes.

Code Installation

Download several python modules and the jupyter notebook in the same directory:

• WatsonCrick.py
• ClassChromosome.py
• Genome.py
• Cellular.py
• telomere-length-distribution-in-synchronous-dividing-cells-ipynb

Some unit tests can be run, a jupyter notebook is available (see the link to gist), they yield:

Model structure:

 The  model consists in a class Cell which have a Genome, which have chromosomes, which have two complementary single strands DNA (Watson/Crick). Each strand has two telomeres.

A single strand object can be instantiated as follow:

A chromosome object is instantiated with two complementary Watson/Crick strands:

For example, two chromosomes, rosalind and franklin can be instantiated as follow:

A single chromatid chromosome, in the G1 state of the cell cycle, can be triggered to the G2 state:

After a round of DNA replication, 5' CCCTAA motifs are incompletely replicated and TTAGGG 3' can be randomly degraded, leading to shorter telomeres on metaphasic chromosomes on both chromatids at pter and qter. Iin the G2 state a chromosome has two chromatids, the length of a telomere on a given arm (pter ou qter) is given by two values:

The mitosis is triggered by chromosome segregation:

The ros chromosome object keeps its signature before and after the mitosis:

Signature of the ros chromosome in the G2 state before mitosis

Definition of a Genome object:

It is convenient to define a genome object, it is instantiated for example with two chromosomes (2N=2):

The genome state (G1/G2), the chromosomes of the genome can be accessed to read the telomere length (CCCTAA or TTAGGG motifs), the length difference between homologous telomeres:

Cell object has a genome:

Unit test results for Cell object

Once a single genome object is instantiated, a single cell object can be instantiated. Let's build a cell with 2N=10:

The length of the telomeres (bp) in that single cell can be read from a pandas dataframe:

Let's allow 10 synchronous cellular divisions from 1 cell to 1024 cells. Then as for fibroblasts cultures, let's make passages, that is one allows only half randomly chosen cells to divide once:

Results

The envelope of the telomere length distribution at a single telomere after 14 divisions (10 divisions + 4 passages) seems to be gaussian:

Correlation between the length at pter and qter on the same homolog

The length of the telomeres (pter, qter) of the same homologous chromosome (the maternal one for example) are correlated:

It is possible to plot the telomeres length of two homologous chromosomes:

The length of the telomeres belonging to different homologous are not correlated.

Let's plot the length of the telomere from the paternal chromosome 1 at qter as a function of the length of the telomere at 1qter on the maternal homolog:

Mean telomere length in synchronous dividing cells population:

In telomerase negative cells as modeled here, the telomere length decreases with cells divisions.

The decrease of telomere length depends on the 5' exonuclease activity. The amount of degraded DNA is modeled here by random variable, X, following a binomial law (N=400, p=0.4) :

Distribution of X: P(130<X<200)=0.999

to fit the data of Makarov et al. (1997) as follow:

 The expected value of X, the deletion length, is :

taking the RNA primer deletion (20 bp), the mean length of the 3' s G strand is:

160 +20 bp

As published in 1992, the standard deviation of the telomere length increase at each cell division.

So the mean telomere length, $mean+/-std$, was plotted as a function of the cell divisions after an expansion of ten cellular divisions and four passages.

The mean telomere length decreases by 90 bp/div

and the heterogeneity increases:

At each passage the length of each telomere (column) of each cell (row) of the simulation is copied in a pandas data-frame, for example at passage P4:

 Then the mean telomere length in the whole cells population can be calculated:

Using seaborn, the telomere length of some homologs can be compared:

Population of Mixed Cells

Cells from different passages can be mixed, for example in equal proportions. The correlation between pter and qter of a given homolog should start to vanish:

Senescence in-silico:

Short telomeres trigger an irreversible transition to $G_0$ state of the cell cycle (senescence). Let's take a cells population, make successive passages. Initial cells population is expended from one cell (telomerase off) after eight  divisions. The genome of the initial cell is instantiated with 2N=4 and with the length of the shortest telomere set to 2000 bp.

 

At each passage, it's possible to count the cells in the $G_0$ state or to calculate the confluence of the cells population. The threshold telomere length triggering the G0 state is set arbitrarily to 200 bp.  The initial population is passed 20 times:

The confluence is calculated as the number of cells at a given passage by the initial number of cells. With the shortest initial telomere length set to 2000 bp, the confluence reach 50% after 18 doublings:

Senescence in silico: confluence  (100 N cell/ N initial cells) decreases with cells divisions

Python modules and jupyter notebook

Frankencomputer for deep learning: running ubuntu 16.04 on a Dell Precision T5500

Choosing a second hand computer:

As my previous laptop died of heat, I was looking for a new machine with a tight budget. Having read the post from Denys Katerenchuck on a budget friendly PC for deep learning it was tempting find an old machine capable of powering a gpu such a GTX 960 or better.

Old workstations are capable of powering such GPU. The Dell Precision T series are valuable since they comes with a 875 W PSU in the case of the T5500.

A lot of peoples tweak this model  for gaming (thanks for the link), for example this one. Several videos show such Dell Precision workstations running games:

•  for example with a NVidia GTX 1050 Ti.
• with a GTX 1070.

I bought on Ebay a Dell Precision T5500 (12 Gb RAM, xeon 5520 CPU, no HD, no cable, 875W PSU) for 133€ (~100€ for the machine and ~30€ for the transport from UK to France).

3x4 Gb RAM

Dust included

PCI slots

When first connected to a TV as monitor (no HD at this stage), an error message was displayed (due to the size of the screen?):

Fortunately, initial tests show no hardware problems:

A 128 Gb SSD from a previous laptop computer was connected on SATA1:

Ubuntu 16.04 was installed from a usb-key. Installing Ubuntu took 15 minutes.

Some BIOS settings were modified to boot on the SSD. SATA-1 was checked and other settings were unchecked:

 RAID config was switched from "RAID-on" to "RAID auto detect/ AHCI":

Powering up looks like:

 

If needed, the boot-info is here.

CPU

This Precision T5500 model comes with a 0D883F motherboard accepting E55xx or E56xx CPU, this is not the best possible choice for a T5500, currently, it has one Xeon E5520 CPU.

GPU card:

The graphic card provided with the computer can't be used for deep learning, even not for 3D with geogebra !! It's a:

 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV635 [Radeon HD 3650/3750/4570/4580]

in 2016, Denys Katerenchuck recommended the GTX 960 4Gb as a good tradeoff between price and performances. In 2018 such card can be found on ebay for less than 150€. The Dell T5500 can power a GTX 1050 from its PCI slot. It has also a six pins power connector (left) to power an EVGA GTX 1060 for example.

There's  an interresting post regarding powering GPU and 6/8 pins cable. Again, according to Akshat Verma (see the answer about T5500) it seems possible to plug a GTX 1070 in a T5500.
Somes reported that they installed a GTX 960 and even a 1080.

BIOS

Dell splash screen visible on first boot

The bios version is a A02 from 2009.  Dell provides updates for A02 both for windows and linux (for redhat, not ubuntu). It seems that the BIOS has to be upgraded in order to install a recent GPU.

Questions about BIOS version and GTX960 can be found, The BIOS upgrade is a recurrent question.

BIOS version history

Latest version is A17

 

Last Update	BIOS	Fixes & Enhancements
07 Jun 2013	A16	1.Enhanced Broadcom onboard NIC 5761 support. 2.Enhanced CCTK support for Turbo Boost setting.
27 Feb 2013	A15	1.Enhanced PSA support. 2.Update PSA to A4527. 3.Enhanced Dual CPU memory configurations support.
04 Dec 2012	A14	1.Enhanced Windows 8 support.
07 Jul 2012	A13	1.Add High IO performance option in Setup. 2.Disable QPI L1 when C-States have been disabled. 3.Add the changes that ties ASPM to the C-State switch. 4.Add TSEG protection feature ENABLE_TSEG_SECURITY.
07 Jul 2012	A12	1.Update new RAID OROM v10.8.0.1303.
03 Nov 2011	A11	1.Enhanced TPM support.
03 Nov 2011	A10	The following changes have been made to BIOS A09 to A10: 1. Added support for newer Processors. 2. Added support for bridged AGP video cards. 3. Updated the copyright to 2003. 4. Fixed UDMA support for 48-bit LBA hard drives (> 137 GB). 5. Fixed problem where system would occasionally shut off when Ctrl-Alt-Del is pressed. 6. Free up unused portions of E000:0000 to E000:FFFF memory for memory managers to use.
07 Jul 2012	A09	1.Added new Hard Disk master password algorithm support. 2.Added support to install single graphic card in 2nd PCIEx16 slot. 3.Enhanced PCIE Slot interrupt handling. 4.Enhanced the memory map algorithm for 128GB configuration.
07 Jul 2012	A08	1.Enhanced method to check the failure memory 2.Updated ACPI SRAT table information
07 Jul 2012	A07	1. Enhanced security device support
07 Jul 2012	A06	1. Updated Intel Xeon?Processor 5600/3600 Series microcode to rev 10. 2. Updated Intel Xeon Processor 5500/5600 Platform Reference Code to revision P2.91. 3. Enhanced TPM remotely provision. 4. Enhance VT-d
07 Jul 2012	A05	1. Updated to the latest Intel (R) Xeon (R) Processor 5600 Series microcode
07 Jul 2012	A04	1. Supported E5620, E5630, E5640, X5650, X5660, X5667, X5670, X5677, X5680, W3680 CPUs. 2. Updated Intel Xeon?500/5600 Platform Reference Code to revision P2.7 3. Removed S1 support. 4. Removed "Optional HDD fan" support. 5. Reported riser's DIMMs information(asset tag, serail number) in SMBIOS. 6. Used the same fan setting before and after S3. 7. Enhanced the compatibility with certain PCI-Express Gen1 cards. 8. Enhanced NUMA under RHEL5.3. 9. Removed HDD Acoustic support.
07 Jul 2012	A03	1. Added error detection for bad monitors or cables when entering setup menu. 2. Enhanced PCIe bar allocation. 3. Enhanced algorithm for Graphics card with multiple OPROMs. 4. Updated BIOS fan descriptions. 5. Fixed BIOS cannot boot from SATA CD/DVD when setting USB controller to "No Boot" in BIOS setup menu.
07 Jul 2012	A02	1. Implemented fix for intermittant boot issue with 6.4 GT/s CPUs. 2. Added feature to display a message if the DIMM configuration is not optimal. 3. Updated Intel(R) Memory Reference Code. 4. Added microcode update revision 11 for Nehalem D0-step. 5. Fixed boot issue with RAID and ATI FireMV 2450. 6. Added TCM support. 7. Added updated IO programming. 8. Fixed possible hang condition when VT-d is enabled. 9. Moved "Optional HDD Fan" to the "Post Behavior" section in Setup. 10. Updated SMBIOS tables. 11. Added support for Windows 7. 12. Updated fan settings. 13. Corrected memory channel information in Setup. 14. Improved the allocation of system resources.

Testimony of a user showing he had a hard time when he tried to upgrade the firmware. The user reported that : "Oh I should state that when I got the machine it had bios version A5 on it. I was able to get to A9, but I had to flash each BIOS in order to get to that point."

Different feedbacks from the web, advice to upgrade the BIOS directly from the current version to the last version

Upgrading from windows yields:

From Ubuntu or freeDos:

There are different possible ways to upgrade the BIOS:

• Using a binary linux, .bin, file
• Running the .exe version from a bootable freedos usb-key and possibly from ubuntu too.
• a 2017 article about the use of freeDOS  (not related to T5500) 
• a post on ask ubuntu reports that the use of a windows- 7 repair disk is necessary

By the way, The T5500 refused to boot from a freedos usb stick up to now ...

Motherboard versions 

It seems  that the motherboard can comes in different flavours:

• D883F
• CRH6C
• W1G7K 

The most capable motherboard seems to be the CRH6C model supporting up to X5690 xeon CPU. One user reported the use of two X5660 on the D883F model with the latest BIOS (shoud be A16).

 Upgrading the GPU

An asus GTX 960 4Gb turbo was waited:

The card is long for a T5500, some place has to be made to plug it in the case:

The blue plastic locker was removed

The hard disc support was partially cut and bent.

As the T5500 was powered on, it just boot (from a BIOS A02!) and open a linux session (with a nouveau driver):

So no firmware update!

Some softwares were installed (nvidia ppa, driver 384-130...) and this yields:

 

the sensors yields the same info regarding the GPU temperature:

CUDA 9 + cuDNN 7 on ubuntu 16.04

Several tuto are available to install cuda 9 + cudnn:

• https://yangcha.github.io/CUDA90/
• from a script

After installation, the nvidia driver was updated to 396.26:

Checking install

Copy the cuda sample files somewhere to  the home directory and build them:

Launch deviceQuerry for example:

The GTX 960 card (PCIe 3) is plugged on a PCIe 2 16x slot. How the card and the computer communicate? May be bandWidthTest can bring some insight: