Overlapping Chromosomes Segmentation: SegFormer’s 10-Epoch Journey

 Finally, it seems to work:

Dataset available on Zenodo: https://bit.ly/4m1DXXX 

GoogleColab Notebook : https://bit.ly/3EVu39z 

Looking inside the "2164 dataset": How balanced is it?

This dataset consists in synthetic images of overlapping pairs of chromosomes+ground-truth labels (chromosome1, chromosome2, overlap). A simple morphological sieve was applied on the dataset to sort the different kind of chromosomal overlaps.

2164 full resolution pairs of synthetic images of two overlapping chromosomes

After having fixed the groundtruth images of the "13434" dataset, an older but full resolution dataset, has to be repaired too.

Repair of the "overlapping_chromosomes_examples.h5" dataset:

This dataset contained originally 2854 (grayscaled+groundtruth) pairs of 190x189 images, stored in a unique numpy array. Its shape was 2854x190x189x2.

The grayscaled images could suffer from two problems:

• Some grayscaled images components had black dots: those images were removed (with their corresponding groundtruth labels).

• The images dtype was int64, it is now np.uint8

The overlapping domain is also now more realistic compared with real overlapping chromosomes:

The labels of the groundtruth don't have no more spurious pixels:

Dataset format:

Once downloaded, the dataset shape is (2164, 190, 189, 2) available as:

• an hdf5 file
• a compressed (.npz) numpy array

Download the dataset with a jupyter notebook:

 

 

Cleaning the groundtruth images from the LowRes_13434 dataset

The ground truth images from the LowRes_13434_overlapping_pairs.h5 dataset contains spurious pixels.This dataset is fixed:

Initial dataset with spurious pixels (Top). Example of cleaned ground truth image (Down)

The cleaned dataset is now available on github as a compressed hdf5 file.

The images were cleaned as follow:

Bad predictions from OverlapSegmentationNet after 14 epochs - Solution by Saleh, Saad & Isa

The problems mentioned bellow seems to have solved by Saleh, Saad & Isa in Procedia Computer Science 159 (2019) 524–533

OverlapSegmentationNet is a UNet model implemented with Keras. The network was trained for 14 epochs (~8 hours on a GT740M GPU), this is not enough to make a good prediction but it is enough to play with the model to understand how a prediction from a single image looks like.

Dataset

The dataset used for training was the first proposed on kaggle. It consists in 13434 pairs of greyscale/ground truth images of size 88x88. Each greyscaled image has a pair of overlapping chromosomes. The overlaps were synthetically generated from images of single chromosomes:

Making a prediction from a single grey scaled image:

The aim is to predict the segmentation labels with a Keras model (OverlapSegmentationNet). The 88x88 greyscaled image must be converted into a 4D numpy array (1, 88, 88, 1).

A prediction is then 4D array of shape ( 1,88,88,4), since there's one component for each label (background, chromosome 1, chromosome2, overlap). A prediction looks like:

where each component is a floating-point image. With another chromosome, we have:

A prediction takes ~ 40 ms for one low resolution image.

Prediction made on an image which do not belong to the train dataset:

A prediction was made on an image belonging to a validation dataset:

To be used for prediction, the image was down sampled and recut to fit 88x88 (and its corresponding ground truth segmentation too):

Clearly, with so few epochs the prediction is bad:

The one-hot encoded prediction can be thresholded and back converted into a multilabel segmentation and compared to the ground truth segmentation:

Notebook is here:

More details are available in the notebook:

 

Resolution of pairs of overlapping chromosomes by semantic segmentation: results from cooperation.

Recently, Rohit Gosh and  Lily Hu  posted reports that a neural network called U-Net can be trained to resolve pairs of overlapping chromosomes.

The story of the dataset:

A first version of the dataset was proposed on kaggle:

• https://www.kaggle.com/jeanpat/overlapping-chromosomes
• https://www.kaggle.com/jeanpat/metaphase

By that time, François Chollet, the developper of Keras, openned the ai.on project, a place where to propose different problems to be takled with artificial intelligence approaches.

The "chromosomes problem" was retained and a community of interrested peoples started to think on the problem (There's also a google group)

First results

Rohit Ghosh tried to mix both categories of pixels belonging to non overlapping chromosomes (green), this results in 97% accuracy in the predicted mask !! However, this can lead to combinatoric issues when trying to solve the overlapp.

Rohit Ghosh's results considering overlapping pixels and mixing single chromsomes in a unique class of pixels.

Rohit Ghosh succeeded also in training a U-Net to predict the three labels:

Rohit Ghosh's results on training U-Net to predict the three labels.

According to Rohit Ghosh, when the three labels are predicted, the prediction can reach an accuracy above 80%.

Lily Hu's results

Lily Hu succeeded to train a modified version of U-net (It again !), here's what she got:

Examples of Lily's Hu results (https://blog.insightdatascience.com/separating-overlapping-chromosomes-with-deep-learning-based-image-segmentation-22f97afd3283)

Here the single chromosomes are are represented in red/green and the overlapping domain in blue. All the examples represents,partial overlapps (the cytogeneticist's nightmare). The second prediction is a little bit weird, but the remaining are  really good.

As mentionned in her post, Lily Hu had to pre-process the labels to remove spurious pixels:

https://blog.insightdatascience.com/separating-overlapping-chromosomes-with-deep-learning-based-image-segmentation-22f97afd3283

I am totally guilty, those pixels originate from the fact that masks were generated prior chromosomes rotation as it was explained:

Example of chromosome resolution (with removing of spurious pixels)

In the new version dataset, the labels are clean.

The traing curves are interresting:

https://blog.insightdatascience.com/separating-overlapping-chromosomes-with-deep-learning-based-image-segmentation-22f97afd3283

 The IOU index for overlapping pixels is above 90%!! Wow!

I suspect that with the new generation dataset (the clean one) which contain much more images at full or low resolution, the training could be improved.

And now what?

Rohit Ghosh an Lily Hu proposed an implementation of U-Net and showed that the approach makes sense.

Here let me allow to cite Lily Hu's conclusions in her post and to propose solutions:

"Future Efforts

I am excited by these results but also looking forward to what the open source community could do to further these efforts. Here are a few thoughts on this, based on my experience:

• The data set can be supplemented with images of single chromosomes and more than two overlapping chromosomes."

15 metaphases are available in in the raw images directory, where single chromosomes can be extracted. 

Regarding overlapping with more than two chromosomes, there's a combinatoric explosion issue.

• Data augmentation can also include transformations such as rotations, reflections, and stretching.

It may be not necessary, the new dataset contains more examples of overlapping. Additionnal metaphases could also be used, introducting thus variations due to the variability  in chromosomes morphology.

Sources

Datasets + notebooks generator

•  first versions
• second version

 Semantic segmentation with UNet on github

Rohit Ghosh 's results
Lily Hu' results

100 overlapping, full resolution, chromosomes isolated from 14 metaphases of human lymphocytes

Metaphasics chromosomes were hybrized with a PNA telomeric probe (CCCTAA-Cy3) and an additional oligo DNA probe (Cy5) for testing purposes.

100 overlapping chromosomes can be found in that dataset (jpp21). The dataset consists of 12 bits raw greyscaled images available on the DeepFISH repository. The images of overlapping chromosomes were isolated and converted into 8bits rgb images:

Sample of 100 overlapping chromosomes (full resolution)

The overlapping images are also available as combination of DAPI and Cy3 components:

Combination DAPI+Cy3 images (inversed greyscale)

• link to the color images
• link to the greyscaled images 
• link to the notebook

DeepFISH : a github repository where to find datasets, notebooks and raw images

For the opening of the topic about chromosomes segmentation on AI.ON, a github repository, DeepFISH (Sorry for the name) was created.

Today, this repo contains:

• datasets: hope to train some kind of convolution neural network to perform semantic segmentation to resolve overlapping chromosomes. The two datasets modelize the overlapping of two chromosomes.

 

• notebooks: for generating the dataset and for loading and displaying the datasets. The jupyter notebooks are written in python 3. Initially the notebook used to generate the datasets was written in python2.

 

• raw images: This is additional raw images of metaphasic chromosomes staines with DAPI and labelled with a telomeric probe (PNA-Cy3).

An example of modelization of overlapping chromosomes

example of generation of overlapping chromosomes.Left: greyscale image (DAPI+Telomeres). Right: red label maps to pixels belonging to the overlapping domain, blue and yellow maps to pixel belonging to single chromosomes.

This image is used for illustration on kagle.

Generating images of overlapping chromosomes

Generating examples of two overlapping chromosome from single chromosomes: 

Notebook:

A notebook showing the different steps to generate overlapping examples is available on kaggle. As well as the raw images.

The idea:

Images of single chromosomes hybridized with with telomeric Cy3-probe and DAPI counterstained were extracted from the image of a metaphase:

DAPI stained chromesomes (left), (CCCTAA)3-PNA-Cy3  probes (Red)

The monochrome DAPI and Cy3 images were added. Single chromosomes were isolated:

All images are 131x128

Given two single chromosomes, examples of overlapping chromosomes were generated combining rotations and translations:

Each greyscale image of a pair of overlapping chromosomes, is associated with a mask containing labels. The pixels of value equal to 3 maps to the overlapping domain of the two chromosomes:

Then a dataset containing 2853 examples of overlapping chromosomes was generated and saved as a h5f file available here:

 examples of overlapping chromosomes 

Once downloaded and uncompressed, the examples can be loaded as a hdf5 file as shown bellow:

Resolving overlapping chromosomes: an emblematic case

Metaphasic chromosomes preparations contain most of the time overlapping chromosomes as follow :

A cluster of two overlapping chromosomes

Here a cluster of two chromosomes was isolated and the two chromosomes were segmented using a contour based method implemented in an ipython notebook.

First a polygonal approximation of the contour is performed and negative corners are extracted as previously:

Polygonal approximation of the cluster contour (red line). Negative corners (yellow dots), positive corners (red dots).

Four points (the yellow ones) are then used to define a quadrilateral of maximal area:

The quadrilateral is converted to an image (overlapp domain:right image) and substracted from the original thresholded image yielding here four chromosomal elements (left image):

The four chromosomal elements are labelled:

Left:labelled chromosomal elements (the size is figured above the image). Right: labelled overlapping domain.

and isolated:

The idea is to used combinations of elements to build candidate chromosomes. Here the image is supposed to contain two chromosomes, so combinations of two elements in a set of four elements yield the following candidates:

For each candidate, several descriptors are computed: the number of connected components (cc), the eccentricity (ecc), the ratio chromosome area over its convex hull area (r) and the number of negative corner on the contour.

The two first highest ratio correspond to the two overlapping chromosomes which can be isolated:

Download the ipython notebook from google drive

Or view it