Looking inside the "2164 dataset": How balanced is it?

This dataset consists in synthetic images of overlapping pairs of chromosomes+ground-truth labels (chromosome1, chromosome2, overlap). A simple morphological sieve was applied on the dataset to sort the different kind of chromosomal overlaps.

COCO dataset from scratch : try and fail ...

Detectron2 provides several algorithms for instance segmentation, so it was tempting to submit the overlapping datasets to one of those. However, to use one of these algorithms, the dataset format seem to follow the MS-COCO format.

One available dataset consists in 2164 pairs of grayscaled+groundtruth images.To give a try, a minimalist dataset with one image and two instances could be converted to COCO format:

 

The two instances (right) are obtained from the groundtruth image showing the overlapping chromosomes. The instances are numpy arrays which can be saved as png images. To generate a COCO dataset associated to the gray scaled image (left), the following steps were followed:

• generate a python dictionary according to the COCO format specification found in the detectron2 documentation and convert the binary masks to their bounding boxes and compressed rle using pycocotools.
  
• Save the dictionary as a json file
• Load the json file with pycocotools (or detectron2) in order to visualize if possible the instances overlaying the gray scaled image.

 The whole process is available in a jupyter notebook on Kaggle.

Unfortunately, the dataset is not a legit COCO dataset as the dataset registration fails. Hope to get some help on Pytorch forum or from stackoverflow.

Modeling overlapping pairs of chromosomes: a dataset containing more than 200 000 pairs of images in png format.

Two datasets modelizing pair of overlapping chromosomes are available on github:

• original github repo
• forked repository

Images are available at full resolution but also at a lower resolution to reduce the memory imprint. To get a smaller memory imprint, the two componants (Cy3, DAPI) of the original image were combined into one grayscaled image.

The chromosomes were chosen to represent the different chromosomal morphologies. The training and the validation datasets were generated with different chromosomes belonging to the same metaphase (slide jp21,metaphase 3) from human normal lymphocytes couterstained with DAPI and hybridized with a telomeric Cy3-labelled PNA probe.

Low resolution images dataset: 

Training dataset

The training dataset was generated from six chromosomes:

The dataset consists in 100 085 pairs of greyscaled/groundtruth images of shape 80x82. 

Simulation of overlapping pairs of chromosomes. Sample of low resolution image dataset.

Groundtruth labels of the training dataset

Validation dataset:

The validation dataset was generated with other chromosomes from the same metaphase:

The validation dataset  contains 111 123 pairs (grey + groundtruth label) of images (8bits, png)

Full resolution dataset:

• The training dataset contains 97 976 pairs of 201x211 images.
• The validation dataset contains 50 543 pairs of grey / groundtruth images.

Generation of 82146 chromosomal overlappings from pairs of chromosomes

The following jupyter notebook was published on github . The aim is to generate a large dataset of overlapping chromosomes (grey scaled image + ground truth label image) to train a neural network to perform semantic segmentation on such images.
To gain a large number of images the resolution was decreased by 16. A first try proposed in the ai.on project seems to do a very good job. The results was obtained from a dataset of 13434 pair of images from a python implementation of Unet.