Teaching a Computer to See Like a Dermatologist


A picture is worth a thousand words in most places, but not at the dermatologist’s office.

“Usually when I’m in clinic, I’m looking at photos and comparing photos,” said Leslie Castelo-Soccio, MD, PhD, an attending dermatologist at The Children’s Hospital of Philadelphia and assistant professor at the Perelman School of Medicine at the University of Pennsylvania. “There hasn’t been a really good way for us to say other than just visually, are things improving or getting worse? We want to have more ways to put an easy score on each of the images.”

Dr. Castelo-Soccio and computer scientist Elena Bernardis, PhD, are now developing computational imaging tools to examine photos of the condition alopecia areata and analyze changes automatically. Their system will eventually identify new quantifiers to convert photographs into more comprehensive and useful numerical scores. They received a new grant from the National Alopecia Areata Foundation to pursue this work.

Alopecia areata is an autoimmune condition that causes the body to attack its own hair, causing varied patterns of hair loss. It can occur at any age, from infancy through adulthood. In addition to causing psychological distress among patients upset by the change to their appearance, the condition also comes with the risk of other co-occurring autoimmune conditions.

Dermatologists have some numerical scoring systems to measure and track alopecia improvement or worsening, but these methods have significant shortcomings. They are time-consuming for busy clinicians because generating a score requires a trained clinician to manually assess a photograph in detail. And, because the scores only measure the percentage area of hair loss and hair growth, they fail to take into account many characteristics that clinicians perceive as clinically relevant, such as hair pigmentation, type, and texture.

“The ultimate goal would be that, when testing new therapeutics, there would be a standard tool that we could use for research studies to go beyond just calculating percentages of density,” Dr. Castelo-Soccio said. “We want a tool that is faster and easier to use, that perceives more refined detail than existing scoring methods, and that generates consistent results among different users.”

To build that tool, Dr. Castelo-Soccio is equipping Dr. Bernardis with the clinical knowledge she needs to teach a computer to think like a dermatologist. As a first step, they have developed a computer program that can accurately score images using the existing method of quantifying an area of hair loss, known as the SALT score.

“Elena is pretty unique in that I don’t know any other dermatology programs that have a computer scientist that is part of the section that can do this kind of work,” Dr. Castelo-Soccio said. “Most clinicians, even researchers like myself, don’t have this training and ability. But I can tell her what I’m looking for, and she can tell the computer to do that, which is pretty amazing.”

Computers do not see and interpret images in the same way humans do automatically, let alone in the specific ways an experienced clinician does. By uploading a set of standardized images, Dr. Bernardis is training the computer algorithm to understand differences between hair and scalp when it encounters new images. Such uses of computer vision algorithms are extremely rare in dermatology. Dr. Bernardis is applying methods that have originally been developed for complex texture analysis and cell counting in microscopic image analysis. These methods require the computer to automatically create pixel groupings for image regions that have very faint and fuzzy boundaries.

“A person can actually interpret a lot of things from the images, but if you zoom in and look at a pixel level, they are just a blobs of colors,” said Dr. Bernardis, who is a research associate in the Dermatology Section at CHOP. “Trying to get information out in a coherent way from the pixels, trying to figure out how to group them together, gets extremely challenging without telling the algorithm that you’re looking for hair. It doesn’t actually know what I’m looking for.”

In later stages of the project, she will train the algorithm to detect other subtle differences in hair appearance that the SALT score does not consider. For instance, progression of disease or remission might be detectable through changes in hair thickness or color. Eventually, she hopes to build in computer vision capabilities so the program will see beyond what the clinician sees — detecting differences in hair appearance that may be useful predictors of disease progression or recovery, but that dermatologists do not consciously note in their observations.

Computer vision methods have reached dermatology later than other specialties, in part because human vision and analysis of photographs has been a reliable standard of the discipline for decades. At CHOP, Dr. Bernardis is collaborating on similar research on acne with Albert Yan, MD, chief of dermatology.

For alopecia areata, these methods could help shape a new phase of research. The timing is ideal because new genetically targeted medications for the condition are now in clinical trials for adults and could reach pediatrics soon.

“I think there are some subtle changes in alopecia areata, other than pure areas of hair growth, that may be important to understand which patients might respond better to those medicines, or to identify any subpopulations of patients who respond more quickly or less quickly,” Dr. Castelo-Soccio said. “I think that by measuring those changes, the tool we are developing will create more quantitative research.”

Share This