Convolutional neural networks for the automatic diagnosis of melanoma: an extensive experimental study

Authors

Eduardo Pérez, Oscar Reyes and Sebastián Ventura

Source code is available to download at Github.

Abstract

Melanoma is the type of skin cancer with the highest levels of mortality, and it is more dangerous because it can spread to other parts of the body if not caught and treated early. Melanoma diagnosis is a complex task, even for expert dermatologists, mainly due to the great variety of morphologies in moles of patients. Accordingly, the automatic diagnosis of melanoma is a task that poses the challenge of developing efficient computational methods that ease the diagnostic and, therefore, aid dermatologists in decision-making. In this work, an extensive analysis was conducted, aiming at assessing and illustrating the effectiveness of convolutional neural networks in coping with this complex task. To achieve this objective, twelve well-known convolutional network models were evaluated on eleven public image datasets. The experimental study comprised five phases, where first it was analyzed the sensitivity of the models regarding the optimization algorithm used for their training, and then it was analyzed the impact in performance when using different techniques such as cost-sensitive learning, data augmentation and transfer learning. The conducted study confirmed the usefulness, effectiveness and robustness of different convolutional architectures in solving melanoma diagnosis problem. Also, important guidelines to researchers working  on this area were provided, easing the selection of both the proper convolutional model and technique according the characteristics of data.

Index

1. Image datasets for melanoma diagnosis
2. Optimization algorithms for training CNNs
3. Weight balancing
4. Transfer learning
5. Balance data using data-augmentation
6. Combining transfer learning and data augmentation
7. Complexity of the Convolutional Neural Network structures used in this work

Image datasets for melanoma diagnosis

Due to the incidence of melanoma is continuing to increase worldwide, in the last few years several private and public datasets have been published, thus allowing a better study of this illness and, therefore, the design of better approaches for its automatic diagnosis. The most popular private data collections of dermoscopic images are the Interactive Atlas of Dermoscopy (Argenziano et al., 2004), Dermofit Image Library (Ballerini et al., 2013), and the dataset presented by Esteva et al. (2017) which conducted a comparison with 21 dermatologists using 129,450 clinical images; to the best of our knowledge this last dataset is the largest one reported in the literature.

Regarding public datasets for studying melanoma, the biggest collection of datasets can be found in ISIC repository, comprising a total of 23,906 images that were labeled by expert dermatologists. In this repository we can found the HAM10000, MSK and UDA datasets which appeared in (Tschandl et al., 2018), (Codella et al., 2018), and (Gutman et al., 2016), respectively. Furthermore, this repository provides the different datasets presented in the annual ISIC challenges (ISIC-2016, ISIC-2017, etc), which are also commonly used as benchmarks by the researchers. Mendonca et al. (2013), on the other hand, created the PH2 dataset that comprises 200 highquality dermoscopic images. Finally, Ericsson (2015) presented the MED-NODE dataset that collects 170 non-dermoscopic images shot with common digital cameras; it is noteworthy that this source of data is very important nowadays since the use of technological devices (e.g. smartphones and tablets) is constantly growing.

Dataset	Source	# Img.	ImbR	IntraC	InterC	DistR	Silhouette
HAM10000	(Tschandl et al., 2018	7,818	6.024	8,705	9,770	0.891	0.213
ISBI2016	(Gutman et al., 2016)	1,273	4.092	10,553	10,992	0.960	0.101
ISBI2017	(Codella et al., 2018)	2,745	4.259	9,280	9,674	0.959	0.089
MED-NODE	(Giotis et al., 2015)	170	1.429	9,029	9,513	0.949	0.068
MSK-1	(Codella et al., 2018)	1,088	2.615	11,753	14,068	0.835	0.173
MSK-2	(Codella et al., 2018)	1,522	3.299	9,288	9,418	0.986	0.062
MSK-3	(Codella et al., 2018)	225	10.842	8,075	8,074	1.000	0.112
MSK-4	(Codella et al., 2018)	943	3.366	6,930	7,162	0.968	0.065
PH2	(Mendonca et al., 2013)	200	4.000	12,688	14,928	0.850	0.210
UDA-1	(Gutman et al., 2016)	557	2.503	11,730	12,243	0.958	0.083
UDA-2	(Gutman et al., 2016)	60	1.609	11,297	11,601	0.974	0.020
Table 1: Summary of the benchmark datasets. Img: total number of images; ImbR: imbalance ratio between the benign and malignant classes; IntraC: average distance between samples of the same class; InterC: average distance between samples of different classes; DistR: ratio between IntraC and InterC; Silhouette: silhouette score.

 

Optimization algorithms for training CNNs

Table 2 shows the results obtained after training the twelve CNN models with the three selected optimization algorithms, SGD, ADAM, and RMSprop, using the learning rates recommended by their authors.

Table 2: Average MCC values on test sets by using the three optimization algorithms.

Table 3 shows the results obtained after training the twelve CNN models with the three selected optimization algorithms, SGD, ADAM, and RMSprop; Adam and RMSprop used the same learning rate as SGD.

Table 3: Average MCC values on test sets by using the three optimization algorithms.

 

Weight balancing

Table 4 summarizes the results obtained after training the twelve CNN models but considering the weight balancing approach.

Table 4: Average MCC values on test sets by using weight balancing.

 

Transfer learning

Table 5 shows the results of each model after applying transfer learning.

Table 5: Average MCC values on test sets by using transfer learning.

 

Data augmentation

Table 6 summarizes the results attained by applying the data augmentation (just in train) approach in each model.

Table 6: Average MCC values on test sets by using data augmentation in train.

Table 7 summarizes the results attained by applying the data augmentation (in train and test) approach in each model.

Table 7: Average MCC values on test sets by using data augmentation in train and test.

 

Combining transfer learning and data augmentation

Table 8 shows the results of combining transfer learning and data augmentation.

Table 8: Average MCC values on test sets by using transfer learning and data augmentation in train and test.

Complexity of the Convolutional Neural Network structures used in this work

The following Tables show some useful metrics to measure the complexity of the CNN models applied in this work. The run time and the amount of utilized GPU memory are reported considering all datasets. The models were analysed considering the settings from Phase 1, meaning that data augmentation was not performed. Regarding run time, we considered two scenarios: (I) how long it takes a CNN model to complete 150 epochs and (II) how long it takes a CNN model to complete an evaluation cycle (150 epochs × 10 folds × 3 repetitions); the run time is expressed in hours and the amount of GPU memory in megabytes.

Table 9: Run time employed by the CNN models during training for 150 epochs. The run time is expressed in hours.

Table 10: Run time employed by the CNN models to complete an evaluation cycle (150 epochs × 10 folds × 3 repetitions). The run time is expressed in hours.

CNN	GPU memory (megabytes)	Parameters (millions)
MobileNet	671	4
NASNetMobile	703	5
ResNet50	873	25
DenseNet121	933	8
DenseNet169	1015	14
DenseNet201	1043	20
InceptionV3	1085	23
InceptionV4	1241	41
Xception	1271	22
InceptionResNetV2	1437	55
VGG16	2051	138
VGG19	2099	143
Table 11: GPU memory used by the Convolutional Neural Network models. The models are sorted by GPU memory.