The Skin Appearance Laboratory

Current Research

Getting immersed in a new area of research has consistently been a time of opportunity. When starting out in biological vision, computer vision, robotics, or artificial intelligence, there was always the pleasure of being exposed to a diversity of new ideas not yet coalesced into understanding. Future benefits were certain to be obtained from the integration of novel frameworks with past experience. In the 13-plus years I’ve been at the Dermatology Department at the University of Rochester, being proximal to the workings of a clinical department has been an important advantage, advantageous in the exposure both to medical research and to the practice of medicine. In preparation for this inquiry into diagnostic imaging, I took part in the core training program for the dermatology residents during my first three years in the Dermatology Department – attending lectures, Kodachromes, journal club and grand rounds. This preparation turned out to be essential to acquire not only an appreciation of the minutiae characteristic of healthy and diseased skin as well as a familiarity with the extensive lexicon necessary to accurately convey that detail, but also to develop an understanding of the diagnostic process and to use that understanding to design new devices and procedures. I have subsequently applied this knowledge at every available opportunity by imaging patients in the clinic using the new technologies.

There are currently many active projects in The Skin Appearance Laboratory. Most address different qualitative and quantitative aspects of the appearance of skin, methods for the acquisition of this data, or opportunities for its application. This section contains short descriptions of recent projects in the lab. Many more projects await resources, students and time. At the end of the list are a few items that just happened to come along.

Additional material is available by following the thumbnail image links.

Skin Imaging

The space of potential skin representations is complex and complete coverage will require the integration of data from many sources. With each offering a unique contribution, the available measures and metrics – radiometric measures, image quality metrics, clinical judgments, and subjective appearance ratings – can be combined in diverse ways to satisfy diverse ends. It is particularly important to relate these measures to a compilation of the anatomical details that are varyingly distributed throughout the surface of the body. In combination with an understanding of the changes in the concentration of chromophores and scatterers brought about by physiological processes that control both the growth and defense of the skin as well as those that support disease, these measures can lead to increasingly exact models of the interaction between light and tissue.

The quality, and thereby the utility, of these measurements often depends on knowing how images of the skin can be most favorably captured. Acquiring this specialized ability requires the study and development of novel imaging instruments. Proper interpretation of the evidence thus obtained also requires an understanding of the cognitive capabilities upon which subjective judgments of clinical and aesthetic appearance are made. Once acquired and appropriately interpreted, these results can be put into practice through applications ranging from clinical diagnosis to the generation of appealing photorealistic simulations.

All that remains then is to relate clinical performance or the appearance of a generative skin model to the analytic underpinnings that bind skin, light and vision and thereby close the loop, validating the process. Closing the loop between synthesis and analysis, deriving an optimal parameterization of the controlling variables, merging the triad of constraints into a consistent representation – the definition of these relations will all benefit from a formal expression of the diagnostic imaging process. Using images for clinical diagnosis will require the validation of a complete and detailed diagnostic imaging chain, incorporating representations of the illumination, anatomy, chromophores, scattering media, BRDF, capture, image format, compression, display, viewing environment, observer sensitivity, target salience, and diagnostic decision making. In addition, an analytical model of the visual system will be developed to measure the requirements for each of the components in the diagnostic imaging chain. With this model in hand, it remains only to relate these findings to the capabilities of the diagnostic observer in the clinic. In large part, by replacing the last component in the clinical model with a characterization of aesthetic judgment, much of the analytic structure of the diagnostic imaging chain can be incorporated into an aesthetic imaging chain that will support subjective measures of cultural preference.

In the course of this and previous research, I have worked on the characterization of transfer functions for various imaging chains including models of both spatial processing in the human visual system (HVS) and the transfer characteristics of different imaging devices. These studies have included individual transfer function components such as the OTF, MTF, and laser scanner sensor array models. Various sections of the diagnostic process are currently under study in the lab and are synopsized below.

DICOM WG-19 Dermatologic Standards

Currently, the principal activity in The Skin Appearance Laboratory is centered around the development of standards for the use of digital imaging in dermatology. Digital Imaging and Communications in Medicine (DICOM) has evolved to become a standard that is the essential and unequivocal arbiter of all things associated with the communication of medical images and any related information (e.g., images, waveforms, derived measurements and assessments, image presentation controls, and workflow management).

DICOM standards are the result of a consensus built principally among three constituencies: the equipment vendors, the professional societies, and the standards development organization (SDO), in the present case, the National Electronic Manufacturer’s Association (NEMA) through the Medical Imaging and Technology Alliance (MITA). The vendors provide the specifications of the equipment and historically have carried out a large portion of the validation studies that support the certification of the proposed imaging functions. The professional medical societies provide a description of best medical practice, the requirements of care to which the vendors strive to adhere. Lastly, the SDO maintains the integrity of the process by ensuring the consistency and adequacy of the standard’s organizational structure and the transparency of the evaluation and decision process.

Central to this cooperative standards development activity are the working groups. They are created to focus development activities on specific areas of interest in response to the demands of the changing medical imaging landscape. At present, there are approximately 29 active working groups. Each is staffed by volunteers from the vendor, clinical, academic and user communities who have commercial and scientific interest in the topics addressed by the working groups as well as expertise in the respective problem areas. Using this organizational structure, the DICOM standards process has proven exceptionally flexible in accommodating individual needs while generally serving the collective best interest of all parties.

The original DICOM success story started with the development of specifications for the communication of the various parameters required to capture, store and display radiological images. NEMA and the American College of Radiology (ACR) worked together to assure interoperability among devices from different manufacturers. The resulting standard secured the extended success enjoyed by radiological imaging over the last quarter century. It should be noted that although the existing organizational structure among vendors, professional societies and the SDO has contributed greatly to this extended growth of digital radiology, variations in the existing organizational model will likely be required to bring the same success to other specialties that possess different needs and resources.

The growth of digital camera sales during the 1990s resulted in an expansion of clinical imaging applications in specialties beyond radiology. Capital-intensive devices that used ionizing radiation were joined in the medical imaging armamentarium by devices that were inexpensive and required only visible light to obtain diagnostic information. In many specialties, and in particular in dermatology, consumer-grade capture and display devices were brought into service. This shift away from high-end, specialized diagnostic equipment to low-cost mass-market devices has had important consequences for the use of imaging in diagnostic decision making as well as for the quality and accessibility of these consumer-grade images in enterprise-wide medical systems.

Of foremost concern is the use of devices whose design is controlled by a market in which healthcare is an insignificant factor. These consumer-grade cameras and displays are optimized to reduce manufacturing cost and energy consumption and to enhance the perceived image aesthetics – not exactly influences one might want in devices used to obtain and present radiometric verisimilitude. For example, consider that the aggregations of melanin (small black dots) that may be diagnostic for some life-threatening diseases can fall within the bandpass of the very visible raw noise spectra of the CMOS sensors. In-camera image processing that very often cannot be turned off may result in the attenuation or elimination of this diagnostic information. Also of concern, the absence of display controls to compensate for chromatic and brightness variations across a monitor may result in difficulties in detecting diagnostic features and in conducting the accurate comparison of lesions across multiple images. Compounding the consequences of these concerns is the absence of a business model to provide the very validation research that is needed to establish the conditions under which these devices would be safe to employ. Neither the cost of the consumer equipment employed in dermatologic diagnosis nor the remuneration of the services they facilitate is sufficient to underwrite a level of support that is anywhere close to that necessary to produce the required validation research.

It is not the intent here to argue against the use of these inexpensive imaging devices in clinical applications. On the contrary, taking advantage of such potential cost savings is exactly what is necessary to slow and, ultimately to reverse, the unsustainable growth in healthcare expenditures. The traditional medical device development model was designed around very expensive equipment and an even more expensive certification process overseen by the FDA that ensures a level of safety and performance that the public has come to expect in the medical arena. New and more flexible approaches in funding and regulation are being implemented – and more will be required.

What is needed to support dermatologic imaging validation research is a new business model, one that does not depend on the capture and display device manufacturers. A model is needed instead that depends on the vendors of healthcare information technology, the people who rely on the image information the consumer-grade devices produce. In a world where information liquidity is increasingly important, healthcare IT vendors have a growing interest in providing enterprise-wide solutions. Seamlessly handling the imaging needs of all the specialties is becoming a requirement for PACS suppliers. Imaging diversity will also will be a concern for the newer players, the PHR and EHR providers as individuals, both patients and physicians, find advantage in populating electronic medical records with image data they themselves collect. It is in the interest of those vendors who serve this new and growing healthcare constituency to contribute their part to the DICOM consensus process and to support the validation of the many new imaging applications upon which their success, and that of their products, will increasingly depend. This support will be especially important as models of service expand from the radiological reading rooms to more universally accessible Web-based applications. It is hoped that these organizations will contribute to this effort either under the aegis of the DICOM working group structure, or through direct support of individual projects.

It is my goal to enhance the capabilities of The Skin Appearance Laboratory to a point where it will be possible to conduct research in all aspects of the diagnostic imaging chain. Such a comprehensive approach is required to bring all the constituent elements together, both to detect problems as information passes through the diagnostic process as well as to recognize and to take advantage of any positive interactions not observed with the analysis of any isolated component. I have included a more extensive rationale for the requirements for an imaging standard in a whitepaper describing a proposal for the resurrection of DICOM Working Group 19, Dermatologic Imaging Standards. In addition to the modeling research for the diagnostic imaging chain, the laboratory will be outfitted for camera and display characterization and calibration. The exam room will be adapted for the development of diagnostic procedures and equipment, as well as for the gathering of data for metrics of diagnostic performance. In the current university medical center setting, there is access to a varied population of general and clinical subjects, and to an unending supply of very capable BME, CS, ECE and Optics students just across the street.

Teledermatology

For more than three years ending in 2008, I was Program Director for VISN 2 Telemedicine at the Department of Veterans Affairs. During my time at the VA, there was exactly one fulltime staff dermatologist to cover the region (all of NY State excluding NYC and L.I.). The customary wait for a face-to-face dermatology appointment was often more than three times the VA’s stated 30 day target for appropriate care. There was a clear need to design a delivery system that would provide better specialty service to the veteran patient population. To accomplish this, a new model for teledermatology was proposed.

Telemedicine is in a state of transition. The professed motivation for telemedicine, and teledermatology in particular, has shifted. What was initially presented as a solution to correct large disparities in the geographical distribution of healthcare services is now being increasingly promoted as a method to amplify the effectiveness of specialty clinicians. In an environment where rising healthcare costs are an increasing concern for the country, the promise of converting clinical effectiveness and efficiency into savings is having an appreciable impact. Saving money apparently has the ability to bring down longstanding regulatory and jurisdictional barriers that the provision of a more equitable distribution of healthcare alone has not.

Providers whose services are in high demand, such as dermatologists, have the opportunity to improve access to healthcare by moving from a model where care is limited by the number of patients that can be examined in face-to-face encounters with board-certified specialists to one where the midlevel staff is collectively empowered to see many more patients. This empowerment is facilitated through the use of digital teledermatology. Documentation of the care provided by midlevel staff is enabled by the capture of diagnostic-quality images that can be sent on to the dermatologist for review and consultation. In concert with the patient history and exam notes, the consistent availability of accurate images allows the specialist to assess each case remotely. From a review of the diagnostic-quality images, it is possible to determine that the accumulated experience from prior training in dermatology clinics and the guidance provided by protocols defining the scope and procedures to be employed by midlevel staff in the provision of dermatologic care has appropriately been applied in each patient encounter.

Savings are accrued because of the relatively lower cost of the services provided by midlevel caregivers and because far less time is required for the review of care provided by a distributed group of trained staff than for the dermatologist to deliver that care directly. With experienced staff, detailed care protocols and appropriate review, a range of conditions determined to fall within the purview of midlevel staff can be treated at multiple sites improving both the wait times in the appointment queues and proximity of care for the patient population. In addition, the cases that require direct specialty access will percolate more quickly through the system. Those patients will be able to get a timely appointment directly with a dermatologist because more patients are being seen by non-specialist staff and therefore the queues to see the specialist are also shorter.

The creation of multiple Skin Evaluation Clinics staffed by nurse practitioners, physician assistants, or non-specialty physicians forms the core of the proposed Distributed Specialty Care Model of dermatologic service. These clinics are the principal mechanism through which scarce dermatologic specialty skills can be leveraged to greater advantage. Another fundamental component on the front end of this model is provided by the services of the primary care physicians (PCPs). These clinicians act the gatekeepers for dermatologic service. Working with the Primary Care Council at the VA in VISN 2, an agreement was reached on the criteria for cases to be handled by the Dermatology Service, as well as a list of those conditions not appropriate for the dermatologic care in the VA. An additional adaptation to the triage of patients was the agreement of a list of 10 common dermatologic diseases (with currently two other potential candidates under consideration) each with well-accepted treatments that would be addressed by the PCPs for an initial, limited interval of 4 to 8 weeks. If these cases weren’t resolved in the specified time, they would be referred to the closest Skin Evaluation Clinic. Patients presenting with known diseases not on the list or with unidentified conditions would also be referred by the PCP to a Skin Evaluation Clinic, or sent directly to the Emergency Department, depending on a set of specified criteria. Incorporation of this front-end service by the PCPs into their practice workflow makes it possible for a large proportion of patients to receive treatment straightaway, with the remainder experiencing far shorter wait times for an appointment with the specialist.

Behind all the organizational innovations are the images. Experience has shown that the backbone of the supervised distribution of dermatologic care is predicated on maintaining a high level of quality in image acquisition. Teledermatology systems that don’t focus on image quality fail. This is why photographic training and image quality assessment were integrated into the teledermatology system at the VA from the start. It was found that with the appropriate resource material, caregivers possessing a wide range of photographic experience could be trained to capture diagnostic quality images.

Notes on the Distributed Care Model and its experience with its use at the VA from 2004 to 2008 have been collected and will be turned into several technical reports. Further description of this effort as well as the teledermatology imaging training manuals are available by selecting the link through the thumbnail image above.

Melanosome Transfer

A very different skin imaging project began with a request from a colleague to convert a series of high resolution microscope stills into a movie. Ordinarily, making a movie out of a series of still images captured in a regular temporal sequence would be straightforward. However, using differential interference contrast (DIC) imaging and a 100X microscope objective, there was a lot of detail to apprehend within the 100 by 130 micrometer field of view. Taken as a whole, the cellular activity in the field of view of these movies was very dense and individual functional activities were difficult to discern. Clearly, the ability to crop an image and to zoom in on a portion of the cell culture would be desirable, but at 10 pixels per micrometer it would require some finesse to increase magnification while enhancing the actual and eliminating the artifactual. Using Matlab, I created a GUI to automate the processing of the still microscope images. In addition to cropping the field of view and sharpening detail, there was a need to lock onto features of interest to cancel the motion induced by the nutrient flow in the culture chamber, to neutralize transillumination variations due to thickness differences in the in vitro cell mat, and to define a moving aperture that made it possible to dynamically focus on different activities for study.

Truth be told, things took an unexpected turn when I decided to make homage to the scene in Star Wars where Luke was attacking the Death Star. In this movie, a melanocyte was to be Luke’s target. Swooping down on the soma and coursing out low along a dendrite, I kept on pushing the enhancement of visible detail at higher and higher magnification. With a putative 100 nanometer pixel resolution available from the microscope magnification, a sampling well beyond the optical resolution limits, there was no shortage of optical detritus to filter out of the images; however, the Nomarski imaging provided interesting hints of unexpected activity in the movies. Eyes with much more training reading histology than mine were able to interpret the moving lines formed from step increases in the index of refraction as filopodia and the entrained black spheres as enclosed melanosomes. The question was what were they doing there?

The prevailing melanin transfer model in the literature was that melanosomes were trapped in an actin logjam at the tips of melanocyte dendrites and were taken up by keratinocytes by way of phagocytosis. However, once the relation between melanosomes and filopodia was initially observed in the high resolution movies, melanosomes were then found moving through filopodia emanating from all portions of the melanocyte cell surface, some extending to keratinocytes in coculture. Clearly, this was a common melanosome activity. Labeling studies taken together with the movies made a strong case for a new model for the transport of these organelles into keratinocytes via filopodia (Scott et al., 2002). Recently, definitive evidence in support of this model of melanosome transfer was published (Singh et al., 2008).

A great deal can be learned from observing the structural and dynamic interactions between melanocytes and keratinocytes and, in particular, from the distribution and transfer of melanosomes in, and between these cells. Much of the data required to observe these interactions can be obtained from the application of image processing techniques to DIC imaging of cell cultures. This approach has only just begun to be explored. Recent findings have shown important differences between the physiological mechanisms at work in 3D malignant melanocyte spheroids as compared to the activity in 2D cell mats. The dynamics of the more natural spherical melanocyte accretions need to be imaged. High resolution DIC imaging of dynamic cellular activity can provide an important complementary view to that provided by the biochemistry. The events captured using this technique have implications for the structure and stability of the epidermis in both health and disease, for the protection of cells from UV radiation, and for the degree of pigmentation applied to the skin. Beyond functionality specific to the skin, recent interest has grown in the use of filopodia to transport more than melanosomes. They are proving to be conduits for viruses as well.

In Vivo Confocal Imaging

An application of image processing to the understanding of skin physiology similar to that of the DIC melanosome movies can be found in the production and processing of confocal images. Because of the restricted nature of light collection in confocal microscopy, the quality of the resulting images is often impacted by absorption and scatter, increasingly so as the target tissue is located deeper within the skin or is located beneath a particularly opaque structure such as a hair shaft.

For a period of time, I had access to a VivaScope 1000, a first generation in vivo reflectance confocal microscope from Lucid, Inc. Using this device, I was able to obtain pilot data on patients in the clinic and volunteers in the lab. To enhance the quality and utility of the obtained images, I developed a GUI that offered stabilization, registration, noise reduction, and contrast normalization functions. The data suggested that additional improvements from the application of deconvolution algorithms, cell boundary segmentation, or the creation of three-dimensional models from the image stacks would be productive. Unfortunately, a rotating galvanometer mirror seized up and this avenue of inquiry went on the shelf. Nonetheless, it remains clear that confocal skin imaging is a modality that possesses an appreciable amount of promise, especially so because of the ability to non-invasively image internal structures of the skin with non-ionizing radiation. There is much room still within vivo reflectance confocal imaging for advances in image processing, acquisition procedures, and hardware innovation – all leading to better science and improved diagnostics.

Body Mapping

There are several compelling reasons to capture a complete record of the condition of a patient’s skin. Foremost among these is the existence of a number of diseases that produce a multitude of lesions on the skin. It is desirable to be able to repeatedly locate and identify each lesion so as to be able to track any changes in its size, shape or coloration. Body mapping may be called upon to index large numbers (up to hundreds) of these discrete lesions (e.g., for dysplastic nevus syndrome). At times, it also may be desirable to accurately measure the extent of rashes that cover large regions of the skin or to document the condition of the skin, both pre- and post-surgery, for any number of procedures.

In some cases, it would be useful to register images of past states of discrete lesions or rashes with current views. Such comparisons could be used to appreciate and quantify any changes and thereby monitor the progression of disease, of treatments, and of the subsequent healing process. In some cases detection of alterations in a portion of a rash or in an individual lesion may provide sufficient cause to apply therapeutic intervention. Another growing application of body mapping is the use of imaging to objectively quantify the efficacy of treatments in clinical drug trials. Properly configured, digital skin images offer the ability to increase accuracy, precision, and complexity of study measures over traditional procedures while also reducing cost.

The principal problem with body mapping lies in designing an imaging protocol that fits into the clinical workflow. Capturing a survey of lesions over the entirety of the skin in a manner both efficient and effective has proven elusive. The traditional method of producing a boxful of Kodachromes and then trying to reconstruct their spatial distribution and compare them to another boxful of Kodachromes acquired at a later time, clearly does not work. It does not work because of the required time. It does not work because of the difficulty in comparing two Kodachrome slides.

Moving forward into the digital age does not, in and of itself, provide a solution. While tools exist that facilitate the comparison of digital images, imaging an articulated, deformable, three-dimensional patient who often has a restricted capacity to assume an optimal pose, in close quarters, with variable lighting, and always with the excessive time constraints of a clinical workflow is never going to be easy. It should be acknowledged from the outset that imaging in a production environment such as a dermatology clinic is also never going to be perfect. Under these circumstances, it is usually the case that one can only attempt to make the process adequate by continually striving to make it better.

With these caveats in mind, a new system has been developed in the lab that takes advantage of digital imaging and integrates it into the clinical workflow by providing GUIs and other software utilities so that the acquisition task is amenable to the skill set normally available in the clinical setting. Pilot studies have been run and clinical studies await a bit more design, and time.

World Enough, and Time

When in graduate school, I made a list of different areas of research that I felt were important to the understanding of human vision, possessed exceptional opportunities for future research, or both: retinal physiology, cortical physiology, spatial psychophysics, image quality metrics, perceptual scaling, visual cognition, computer vision, robotics, medical imaging, device characterization, film characterization, photorealistic body simulation, and photographic portraiture. Over time I have had the opportunity to explore most of them. The two remaining unexplored areas on the list, the photorealistic synthesis of the human form and the analysis of photographic portraiture, combine both aesthetic and technical aspects of the representation of human beings with a complexity that requires the integration of much of the understanding previously obtained from the other vision specialties on the list.

Photorealistic Skin Simulation

The creation of a realistic simulation of a given scene is often overly constrained by the spatial and temporal resources the process has been allocated and, as a result, the quality of its appearance suffers. For the simulation of human beings, however, this insufficiency has proven to be not just a matter of the coarseness of the available spatiotemporal resources. Realistic appearance has been found to depend on the inclusion or exclusion of specific properties that modify a whole host of substantive details. Does a single hair shaft emerge from each follicle? (Some follicles produce more than one hair shaft, and some none at all.) Why do bags and dark circles appear under some people’s eyes? (They result from a combination of genetic traits that affect anatomical structures such as a particularly thin dermis in the periocular region leading to greater transparency that renders backscatter from the underlying vascular plexus and ocular muscles more visible, a reduced containment of orbital fat that produces tissue bulging and increases in associated shadowing in the overlying skin, and changes in the vascular pressure distribution superior to the heart that results in increased pooling of blood and consequent local darkening.) What is the demographic for gray hair in middle-aged men? (50% of men are 50% gray by the time they are 50.) Why do scars often appear lighter than the surrounding undamaged tissue? (A combination of vascular regression during scar maturation due to the decreased metabolic load of the restructured tissue and a more horizontally parallel organization in the deposition of scar collagen results in an increased, paler (less hemoglobin-influenced) backscatter distribution while the melanin contribution is comparable to that of the proximal noninvolved skin; in comparison, the appearance of the lip vermillion region is just the opposite in that it has a deficit in the other major skin chromophore, melanin, but it has a vascular component at least equal to that of the surrounding skin tissue.) Are beard and scalp hair the same? (Across racial groups, a beard hair shaft has a larger cross-section, is less round and has more cuticle layers that are less uniformly distributed.) How does the color of the tongue vary over its surface? (The tongue possesses a complex chromatic reflectance distribution resulting from anatomical variations across its base, tip, rim, and central regions that is compounded by additional patterns due to aggregations of desquamations, nutrient stains, and bacteria that each display their own regional variations across the dorsal surface and that usually require hyperspectral imagery to be properly segmented.) Details such as these, and hundreds more, are some of what is necessary to make human simulations appear real to human eyes.

Over the past dozen years, I have collected a wide range of the physical minutiae that contribute to variations skin appearance, details that often act as the signposts of health and illness. These are properties that a dermatologist learns to appreciate and unconsciously internalizes as the result of examining thousands of patients during residency. This compendium of dermal detail is derived from isolated observations loosely scattered across a 30 GB (and growing) database of clinical, biological and engineering literature related to the skin. Taken together, these details constitute a foundation for dermatologic inquiries both clinical and aesthetic. They provide a core of an individualized description of dermal appearance, the geography of a person’s skin. One example where the dependence of the complexion on such detail can be demonstrated occurs in a situation where our perceptions and preconceptions commonly betray us: the appearance of our veins. This instance of perceptual failure is described in a (newly updated and expanded) presentation I gave to the department shortly after I arrived in dermatology (for additional details view the speaker’s notes).

This atlas of cutaneous geography, a gazetteer of anatomical features, is complementary to and supportive of generative light transport models and reflectance measurement techniques. Examination of existing anatomical descriptions in medical textbooks, scientific papers, and in the clinical patient record provides a candidate taxonomy of requisite skin detail. What remains is for these groupings to be stocked with collections of features, measured values, and demographics: age, sex, skin color, texture/dermatoglyphics and anatomical location, especially for the eyes, ears, nails, nipples, lips, nose, genitalia, and scalp, beard and body hair. This classification effort should strive to produce a product that is more than just a CGI version of Gray’s Anatomy. In the process of extracting the parameterizations of skin components, the correlations that exist among those values will also need to be retained so as not to lose the essence of a whole that is certainly more than a constituent sum.

Many skin image databases are accessible on the Internet that house collections of diseases, some comprehensive and others very specialized. Fewer repositories of skin images exist that catalog the variations in healthy skin across age, sex, skin color and anatomical location – and these collections are especially rare for specific racial groups. Almost no resources are available that rigorously control the quality of the images to the degree necessary to subserve clinical diagnosis or even to provide a relative consistency of appearance across images in the database. There is an opportunity, indeed a requirement for future progress in the development of essential utilities, for a collection of stills and videos that capture the gradations of skin and appendages at a level of quality that will allow them to be used as exemplars in critical diagnostic applications such as for clinical specialty training, in evidenced-based medicine and for clinical decision support systems. Uses such as these will require the investigation of new methods for the imaging of skin detail, both static and dynamic.

How is this detail to be acquired? Recently, driven in large part by the application of CGI and 3D in feature films, there has been a surge in the development of 3D acquisition hardware. What were once research tools (e.g., laser scanners and photogrammetry packages) now can be purchased off the shelf. As the requirement for quality in the acquired detail is increased, there will be a need for new capture devices, both video and still imaging, that look to increasing the dimensionality rather than simply increasing resolution. Among the candidates for sources of photorealistic input are clinical devices such as digital dermatoscopes, instruments that already are being designed consistent with diagnostic imaging requirements. There will also be a need to capture the spectra of the skin chromophores such as hemoglobin and melanin with hyperspectral imagery to determine how they are altered in vivo by tissue depth and redistributed by the influence of wounds, scars, bruises, erythema and the effects of aging as well as by the subtle differences among the benign, and not so benign, rashes and pigmented lesions. These measurements can act as documentation of the more subtle, but nonetheless important, skin interactions that are not appreciable from the more generic trichromatic digital representation.

An interesting example of the trend in promising new technologies that comes from the CGI domain is the motion capture process, Contour. Developed by Steve Perlman, this technique uses a phosphorescent mixture that supplies texture by being sponge-applied to the skin and that allows capture of highly detailed 3D surface shapes from video recordings. This process received many accolades for its use in The Curious Case of Benjamin Button including Academy Awards in 2009 for makeup and visual effects. However, questions remain. Although the phosphorescent component might be exceptionally pliant and be invisible to the eye under normal illumination, wouldn’t its flesh-colored vehicle fill-in skin crevasses and mute much of the BRDF variation that makes skin appear vital? How is it possible to avoid creating replications of a Gumby-like appearance – skin with attenuated dermatoglyphics and an unnatural planarity? Is Contour another technological advance that still contains fatal flaws or is it a contribution that will hasten the time when simulations will be expected to leapfrog the Uncanny Valley with regularity?

A good simulation does not just require the presence of a continuous surface devoid of artifacts. It necessitates the inclusion of specific, region-appropriate detail to realistically model surface reflectance. In addition, devices and algorithms will be asked to provide a more complex dynamic component that reflects the articulation of the body and the resulting pressures brought to bear on soft tissue. Improved representations will flow from capture the detailed changes in both the specularity due to surface distortion of the dermatoglyphics and in the backscatter pattern due to better assessment of the internal vascular redistribution. With Contour, it is unclear how these skin properties can be captured simultaneously under the makeup, a composite that is composed of the emissive phosphor and a skin-color base. It should be noted, however, that the technique does come with an impressive claim that not only is the makeup invisible, it does not does not disrupt normal skin reflectance nor does it alter subsurface light transport under normal lighting conditions. Time will tell.

Another avenue of opportunity that needs to be more fully developed involves the use of video to stitch together a range of views of a stationary individual that can provide information not present even in the most accurate capture by a single still image due to occlusion and resolution limitations. It is of particular interest to determine the set of views required to capture the information necessary to create a 3D model with a defined level of detail. Additional information can be learned through imaging the individual articulation patterns. Much of what contributes to the motion patterns characteristic of an individual are based on a nonuniform distribution of preferences selected from the range of potential movements. Even then, motion capture must do more than provide a representation of gross movements, such as ambulation, correctly to make simulations appear vital. Video sequences will also need to acquire the essence of small movements and deformations that that can be used to help simulations retain vibrancy or to aid in the discovery of subtle clinical clues. The question is how are these slight motions and deformations to be obtained with sufficient accuracy so as to achieve the desired end result in image quality. New devices are being developed here in the lab for clinical purposes that can capture high resolution still images and video of skin on a near microscopic scale. With little or no modification they can be applied to aesthetic tasks as well.

Photorealistic simulations have the potential to be useful in many domains. . It is certainly true that photorealistic simulations already have had, and will continue to have, a major impact on the cinema. They also can be used in social interactions to create avatars that subserve functions such as a trusted companion in populations with elderly-dominated demographics (e.g., Japan and Italy). Humanoid caricatures may entertain, but the degree of photorealism necessary to evoke a significant empathetic connection remains to be determined. A personalized simulacrum could also be used to display the effectiveness of cosmetics. Any compound whose spectral characteristics are known could be assessed without the cost of manufacture or, once developed, individualized prior to purchase. Again, the degree of verisimilitude required to be an acceptable conveyor of an individual’s personal presence, especially to one’s self, is yet an open question

The interaction between medical and entertainment applications can be bidirectional. Highly detailed simulations are not only useful for aesthetic applications. They can also be used in clinical research. Clinically accurate simulations could serve as an informative medical interface for resident training or to enrich patient education. Enhanced 3D simulations that incorporate appropriate skin texture and BRDFs may well provide a better training venue for the detection and recognition of lesions than is possible with a 2D image, especially so for uncommon or contagious conditions. If a skin condition can convincingly be synthesized in a realistic manner, that information can be used to fortify the detection process in assisted diagnosis. It follows that the simulation parameters contain the subjective essence of what is being sought, since all of the lesion’s visible features have been capably rendered.

This enhanced attention to anatomical detail as well as the knowledge of the underlying optical processes can be applied in combination to produce no-excuse simulations. These simulations can be analyzed to formulate procedures for the capture of diagnostic images that do not contain perceptible flaws. In both cases, it is clear that it will be important to supply sufficient quantitative resources so that the qualitative results will be as accurate as aesthetically desired or clinically required. It is a good, if rarely applied strategy, to first invest as many resources as is practicable to allocate when creating a simulation and only then to use that experience to determine what resources can be pared back to what is actually necessary. Sadly, and all too often, design cutbacks are set a priori and the adequacy of reduced performance is simply asserted.

Finally, it also would be interesting to investigate what the limitations are, if any, to the appearance of human representations due to the 2D projection process referred to above. What is the impact of the loss or the corruption of 3D cues such as stereo, parallax, flow, or perspective? What is the best performance that can be expected using planar structures to represent the human form in aesthetic, and in clinical, applications and how does this vary across the observer population? As has recently been demonstrated, not getting this right goes beyond incurring aesthetic failures into risking serious clinical consequences.

Photographic Portraiture

Several years ago at the then recently renovated AFI Theater in Silver Spring, MD, I had the opportunity to view a newly restored 35mm print of Fred Zinnemann’s classic western, High Noon. The pristine, black and white (such a misnomer, particularly in this instance) film and the highly reflective and innovative screen technology combined to make the close-ups of the actors such as Katy Jurado qualitatively better than any I’d ever seen. The exceptional detail and tonality afforded by that moving facial image meters in height put it in a class of experiences occasionally produced only by the best of the early photographers’ (e.g., Stieglitz, Steichen, Strand and Weston) efforts as they attempted to capture the potential of their new art form. There is something clearly transformative when imaging is done right.

What is it in these images, these photographic processes, that allows the humanity of the human form to be captured so effectively? One early task in this effort will be to examine and catalog the tonality and noise structure of classic movie and still images using the theoretical constructs of modern image processing. Through a review of the literature describing traditional tonal measures such as the Zone System by Adams and Archer and by translating those descriptors to the more recent theoretical perspectives such as those associated with the analytic and quantitative tonal representations derived from current HDR image research, it is expected that a better relation will be established between the underlying physical mechanisms and the resulting aesthetic experience.

What are the subjective influences that characterize different varieties of classic movie film stock and determine their aesthetic appeal? Also, what is the relation between the current use of digital intermediates and earlier analog film processing? What were the techniques and tricks that were applied during development or printing by early artists that now have been taken over by Photoshop or the processing in digital intermediates? Original stock will be examined and analyzed so as to capture the characteristics of the fine detail, fluid tonality and hallmark noise structure associated with those classic materials. This analysis will hopefully shed light on which of these contributors to appearance have particular application to the representation of the human form.

Different display transformations will be investigated in an attempt to reproduce in the current hard and soft copy modalities portraits that recapture the exceptional flavor, quality and interest so manifest in early prints and film. Are there essential differences between the digital and the chemical? Is something inevitably lost in the transition to digital representations? The answer will require an empirical investigation; however, I believe it unlikely. Representational properties should be lost only if the transformations are flawed and are unnecessarily inadequate in design or inappropriate in application to retain the required information.

The research will explore additional opportunities to define the influence of technology on photographic portraiture. Does early film appear the same when displayed by modern projectors as when shown on the early devices? How does the collimation or convergence of the transillumination of the film alter the tonality of the projected image? Does the back focal length and aperture diameter alter the perceived characteristics of film? How do viewing distance and capture focal length alter the perception of the projected image? How does the variation of these properties distributed throughout a given movie and given the vantage provided by a particular location within a theater or by the lighting and position of a print or a monitor in a gallery alter the manner in the presentation is perceived? There is much to be discovered.

Also of interest are the respective characteristics of B&W images as compared to those that incorporate color. Perceptual segmentation based on color certainly alters the subjective appearance of an image in a manner very different than that achieved by viewing images composed only of shades of gray. The proposed new vector difference metric being developed to assess the parameterization of the Diagnostic Imaging Chain Model may be useful in predicting the effects of color and grayscale on the appearance of images. There is a substantial physiological and psychophysical basis for the belief that the two classes of images are processed, and perceived, differently. This distinction might lead to new visualization techniques, of use both artistically and clinically. On the other hand, exceptionally evocative images are those that convey more than the physical. How do the interactions between pose and lighting alter the appearance of a subject? There is much that computational photography has contributed to technical functionality, perhaps it has much to contribute to aesthetic appreciation as well.

Odds & Ends

IP and Dermatologic Imaging

The benefit of asserting intellectual property rights in service to the delivery of dermatologic care is yet uncertain. The relation between IP rights and the practice of dermatology is a complex balance between splintering an already small market and providing financial incentives for the development of needed technologies. Progress in dermatologic innovation is inhibited because too many people are chasing too little money.

No single diagnostic device out of the diversity of protected technologies has been able to make sufficient inroads into the provider population for a consensus of what constitutes best practice to coalesce and for standards to emerge. There may perhaps be a greater opportunity for the application of technology to dermatologic diagnosis if IP rights were integrated with a more comprehensive system of care management and IT workflow. With the efficiencies of an all-inclusive healthcare data system, the potential to accrue additional savings could help underwrite the cost of IP development and promote acceptance of the new technologies among practitioners.

Even with the backing of enterprise systems providers, a workable business model still needs to be found. Why should investment be attracted if the potential for even a modest return is uncertain? One reason to take interest is that some of these new dermatologic technologies have the potential to project specialty support and supervision to remote patients and can thereby enable a much wider spectrum of caregivers who could provide significant levels of additional service. Supporting the integration of these comparatively low-cost devices has the potential to open up alternative business models that can leverage better management of scarce clinical specialty skills to reduce the cost of care and enhance its availability, especially in geographically isolated regions.

Unfortunately, there are other complications. The relation between IP rights and standards development is an evolving one. There is a trust inherent in the latter that gets in the way of exercising all the rights that come with the former. Some selectivity in investment is perhaps warranted until the advantages of the technology to the healthcare system are recognized and this leads to the elimination of statutory impediments. On the plus side, transformational influences should only increase as the Health Care Reform Act gets implemented.

Also of concern is the reality that dermatology does not require the large capital investment that subsidizes research in other specialties, reimbursement rates are low and this leaves little room to cover additional expenditures. In this environment, cost-sensitive specialties will be reluctant to converge around standards that contain proprietary components. This assessment will likely remain valid as long as dermatology retains a technology-lite, 20^th century model of care that depends predominantly on direct clinical observation by costly specialists. On the positive side, the need for HIT vendors of enterprise systems to incorporate dermatologic imaging may be a motivator for support. The current impetus to adopt a national system of electronic medical records that encompasses all specialties may be the significant factor that brings this effort together.

For those IP and patent wonks among you, here are some additional thoughts (in progress) on obviousness.

Brian C. Madden