Clinical validation study of artificial intelligence algorithms for early noninvasive detection of in vivo cutaneous melanoma
Conclusions
The device has demonstrated an excellent performance in terms of malignancy prediction, which turns it into a valuable tool to prioritize patients according to their risk of presenting malignancy.
The AUC metric for the malignancy prediction was 0.8983, which is comparable to that of expert healthcare professionals (HCP) and speaks to the potential of using the device to improve clinical workflows.
Regarding skin lesion recognition in general terms, the Top-5 accuracy was 84.22%, which supports the device's intended use as a clinical decision-support tool. Specifically in melanoma, the AUC metric was 84.82% which is considerably high and means the consecution of the goals set out in the hypotheses of the study. On the downside, the Top-1 accuracy was 55.01% in the multiple ICD classification task, but the Top-3 accuracy increased to 75.69%. However, it's important to keep in mind that the Top-1 accuracy metric was not a relevant metric to this study, nor the performance of the device, because the device is designed to always output at least the top five predicted classes. This is aligned with its intended purpose as a clinical decision-support tool.
Given the results of the image dataset collected in this study, it is clear that images of better quality would have improved the functioning of the device and provided more valuable insights. Also, as the raw images were usually taken far away from the skin lesion, the cropping resulted in images of suboptimal resolution, which would have also been corrected by higher quality images - or by actually taking the image closer to the lesion.
Summary
- Code: LEGIT_MC_EVCDAO_2019
- Status: Finished
- Start date: November 22nd, 2019
- Finish date: April 10th, 2024
- Acceptance criteria:
- AUC greater than 0.8
- sensitivity of at least 80% or higher
- specificity of at least 70% or higher
Background and rationale
The incidence of cutaneous melanoma (CM), the deadliest form of skin cancer, has been rising significantly, with a sharp increase in both new cases and mortality rates over the past decades. Melanoma metastasizes quickly to distant organs and becomes resistant to conventional therapies, making treatment difficult. However, when diagnosed early, it can be effectively treated through simple surgical excision. The main challenge in early detection lies in differentiating melanoma from benign pigmented skin lesions, as they often share visual similarities, especially during an initial visual examination. Limited access to dermatologists and a lack of public awareness further complicate timely diagnoses, often leading to delayed detection when tumours are already advanced.
Given these challenges, early melanoma diagnosis and prevention have become critical. Recent technological advancements in image recognition and artificial intelligence (AI) have shown promise in improving diagnostic accuracy. Studies have demonstrated that AI algorithms can classify skin lesion images, including melanomas, with a competency level comparable to that of dermatologists. These AI-driven computer-aided diagnosis (CAD) systems present a valuable opportunity for improving early detection and survival rates by assisting healthcare professionals and potentially empowering non-experts to detect melanomas earlier.
This study aims to clinically validate an AI-based system for diagnosing cutaneous melanoma, leveraging artificial vision and machine learning to differentiate between melanomas and benign skin lesions. By developing and validating this technology, the study seeks to improve diagnostic precision and contribute to earlier interventions, potentially enhancing patient outcomes and survival rates.
Product Identification
| Information | |
|---|---|
| Device name | Legit.Health Plus (hereinafter, the device) |
| Model and type | NA |
| Version | 1.0.0.0 |
| Basic UDI-DI | 8437025550LegitCADx6X |
| Certificate number (if available) | MDR 792790 |
| EMDN code(s) | Z12040192 (General medicine diagnosis and monitoring instruments - Medical device software) |
| GMDN code | 65975 |
| Class | Class IIb |
| Classification rule | Rule 11 |
| Novel product | FALSE |
| Novel related clinical procedure | FALSE |
| SRN | ES-MF-000025345 |
Limitations
The data gleaned from our study underscores the imperative need for targeted training programs aimed at enhancing the skills of HCPs in capturing high-quality clinical images. Proper training is paramount to ensure that the images taken in real-world clinical settings are of sufficient quality to yield accurate and reliable diagnostic outcomes. This aligns closely with the findings from our research, indicating that when HCPs are adept at taking good images, the real-world performance of diagnostic tools and assessments can be significantly improved, closely mirroring the positive results obtained in controlled research settings. Thus, investing in comprehensive training for HCPs on effective image-taking techniques stands out as a critical strategy for optimizing patient care and enhancing the overall efficiency of the healthcare system.
Additionally, we believe another factor that limits the results is that, in some cases, malignant cases can not be easily analyzed simply by observing the image. Indeed, some cases require a biopsy to ascertain a diagnosis, regardless of the experience of the observer. This is not a problem for the performance or the safety of the device because whenever there is a suspicion of melanoma, clinicians universally adhere to the protocol of conducting a biopsy to confirm a diagnostic suspicion. This established clinical practice is rooted in the fundamental understanding that the removal of a melanoma is a minor procedure compared to the significant risks associated with the disease. Consequently, practitioners will never rely solely on the device for information when it comes to identifying melanoma, ensuring a comprehensive and cautious approach to diagnosis and treatment.