Adapted from "Methods and definitions to rate the quality of outcome measures". Schmitt, J., Langan, S., Deckert, S., Svensson, A., von Kobyletzki, L., Thomas, K., & Spuls, P. (2013). Assessment of clinical signs of atopic dermatitis: A systematic review and recommendation. Journal of Allergy and Clinical Immunology, 132(6), 1337--1347. doi:10.1016/j.jaci.2013.07.008.

How can you create a new scoring system?​

Developing a new dermatological scoring system is a complex but rewarding endeavour. By following these key steps and incorporating cutting-edge technology, scientists can significantly contribute to the field of dermatology, enhancing diagnosis and treatment outcomes.

Here's a list of steps you can follow to create a scoring system:

1. Identify a need​

The first step is to identify a need. You may look for instances where determining the severity of a condition has a high degree of subjectivity. You may also seek to automate scoring systems that are very tedious to fill in. Most importantly, it is useful to look for situations where patients are suffering due to a lack of access to a specialist who is capable of measuring the severity.

All these instances create an opportunity to innovate. This is perfectly conveyed by Professor Ramon Grimalt, who explains the motivation that led him and his collaborators to create a new scoring system for dermatitis:

Through this project, it has been possible to create, for the first time in history, an artificial intelligence system that is in charge of this work. We are very satisfied with the results that will allow clinical trials and more precise diagnoses.

Even before starting the development process of a new scoring system, the researchers at Legit.Health make sure they are addressing an actual need. Staying up-to-date with the state of the art in dermatology and extensively reviewing the pertinent literature is a key step before any research project.

2. Selecting the dataset​

As is often the case in medical science, one of the biggest issues our researchers have to face is selecting an optimal dataset. To do this correctly, there are some crucial things to consider:

Collaborate with experts​

Collaboration with dermatologists and other experts is vital. Their insights help tailor the system to real-world needs. Simultaneously, gather a comprehensive dataset encompassing a wide range of cases and variations within the disease spectrum. This dataset will form the foundation of your scoring system.

Establish annotation and assessment protocols​

Develop clear guidelines for annotating and assessing lesions or other dermatological features. This could involve detailed criteria for lesion count, size, color, or other relevant factors. Ensuring consistency in these initial stages is crucial for the system's reliability.

Regarding the quantity of images, as a rule of thumb, the more parameters we want to include in the new scoring system, the more images we need to have in the dataset. But of course, the more images the better.

One trick that may help you to better optimise the usage of the dataset is to start using the images gradually, looking for the moment when the results stabilise, thus deciding the dataset size as you gather it, so that you don't fall short or use too many images.

3. Building a model​

This step is somewhat difficult to cover in this article, but right below you will find a scientific publication where we detail how we trained a model for hidradenitis suppurativa.

Select the most impactful clinical signs​

All scoring systems look at clinical signs. However, not all of them contribute equally to the affectation of the patient. That is why a key step in building a model is selecting the most relevant clinical signs, and looking for the optimal combination.

Often, for each condition, you can find several pre-existing scoring systems, each one with some strengths and weaknesses. It is by analysing them that we can find out which parameters are the most important ones for the condition

For example, in the case of acne, the literature suggested that aside from the count of lesions (a parameter that showed in every other scoring system), the density of lesions was also noted as a very prominent sign of a severe condition.

The selected parameters will then be measured and identified in a selection of clinical pictures, by specialized doctors who assist in this process. These values will be compared with those of the gold standard scoring system for that same array of images.

This allows researchers to correlate the values of the previously existing method with the newly defined variables.

4. Optimising the model​

Once the researchers have all the parameters defined, it is time to create an equation that combines them all to best represent the severity of the different clinical images used in the study.

This is known as an optimisation problem, a mathematical term that describes the process of finding the best possible solution to a specific problem: in this case, how to represent the severity of a condition.

Without delving too deep into the mathematical intricacies of the process, different combinations of operations are tested on the parameters: adding, subtracting or multiplying them to obtain a single combined value that represents the severity of the condition.

Then, the results of each of these proposed equations for every single image in the set are compared with the results given by the selected gold standard, looking for the best possible correlation with the previously existing problems.

5. Prove it works​

Validation is key. Test the system against both interobserver and intraobserver variability. Use statistical measures like the F1 score, mean absolute error (MAE), and Krippendorff alpha to assess the system's reliability and accuracy. This step might involve multiple iterations to fine-tune the system.

This is done with a subset of clinical pictures that have not been used to optimise or train the model in the previous steps. After all, your equation has been created to perform perfectly when tested against those images, so to demonstrate that it can work with any picture of the condition, we have to test it with some new ones.

Simply put, our researchers work closely with specialised doctors to label the images using both the gold standard method and the newly developed model and will compare the results to see if the new model performs better than the original one and still finds a correlation with it.

6. Put it into practice in a clinical environment​

Finally, the model is brought to the forefront of medical experimentation and compared with the results of dermatologists observing the same affection in clinical practice.

Continuous monitoring and feedback from users will provide valuable insights for ongoing improvement. The system should evolve with advancements in dermatological knowledge and technology.

This step is key to ensure that the method can keep up with the reality of the day-to-day practice of medicine, as a good and reliable scoring system is not only more objective and less error-prone than the clinical eye but also needs to be fast and not overload doctors with more work.

That's why at Legit.Health we link our scoring systems to computer vision algorithms, to put the speed and precision of artificial intelligence in the hands of doctors and help them help their patients.

Get access now​

This free 23-day trial of Legit.Health gives clinics and hospitals a hands-on look at how to drive increased adherence and improve patient outcomes, as well as improving efficiency and overall quality of life.