282 A
A N A LY T I C A L C H E M I S T R Y / A P R I L 1 , 2 0 0 0
nalytical imaging systems, especially surface imaging systems, often produce multispectral images that divide sample information into a number of individual images (1). Visual perception of the sample information, such as exact spatial location of different elements relative to each other and the formation of occlusions, becomes complicated because of the large number of separate images, each showing a different aspect of the sample under investigation. Image fusion is a process whereby images obtained from various sensors, or at different moments in time, or under different conditions, are combined together to provide a more complete picture of the object under investigation. Using color introduces a new dimension of information that can be used to simplify image analysis and object identification. The goal is to increase the amount of information (the number of channels) that can be merged into one image to enhance the perception and interpretation of multispectral analytical images. Different color models for image fusion have been tested with sample images from different analytical sources, such as secondary-ion MS (SIMS), scanning electron microscopy (SEM), and electron probe microanalysis (EPMA), and the strengths and weaknesses of the various color fusion methods have been evaluated. Image fusion can be defined as the process by which several images, or some of their features, are combined to form a single image. The fused image should preserve, as closely as possible, all relevant information contained in the input images. The fusion process should not introduce any artifacts or inconsistencies that can distract or mislead a human observer or any subsequent image-processing steps. In the fused image, irrelevant features and noise should be suppressed to the maximum extent possible. The overall aim of the fusion process is to provide an observer with better visual perception of information rather than information separated into numerous images. To achieve this, the way the human eye perceives information also must be considered. A major effort has been made to evaluate different gray-scale image fusion meth-
ods in scientific imaging, especially in the area of medical imaging. The techniques start with simple approaches such as adjacent display, which shows each modality (spectral image) in a separate window with a linked cursor that indicates corresponding locations in the image slices (2). Another possible method uses a checkerboard display in which pixels from two or more images are alternated in cells like black and white fields on a checkerboard (3). All kinds of arithmetic integration, in which pixels are submitted to mathematical operations such as averaging, subtraction, or more sophisticated pyramid-based approaches, are also in common use (4–6). All these gray-scale fusion methods are at a disadvantage because the human visual system can distinguish hundreds of thousands of different color shades and intensities, but only about 100 shades of gray. Therefore, an alternative approach that uses color as an additional carrier of information to increase the density of information and improve perception and distinction of different phases in a single image must be considered. Consequently, fusion based on color models has been introduced to the imaging sciences, primarily in medical imaging. Freiherr tried integrating multiple positron emission tomography tracer images by assigning them to the primary colors red, green, and blue (RGB) (7), and Kamman and colleagues used RGB assignment for integrating multiparameter magnetic resonance images (8). However, there is an important difference between medical images and analytical images. Medical images usually consist of structures that can be easily identified by a human observer (e.g., skull, brain, or blood vessels); analytical images often contain information that follows no predictable perception pattern (e.g., the distribution of mass in a SIMS image). As a consequence, the information density is usually lower in analytical images than medical images. This fact strengthens the need for integrating color as an additional carrier of information in analytical image fusion.
So what is color modeling all about? To approach the idea of color fusion, it is necessary to understand the basics of color
A P R I L 1 , 2 0 0 0 / A N A LY T I C A L C H E M I S T R Y
283 A
24-Bit image
0
0
0
255 254
168
98
0
255 255
255
255
255
Red layer
FIGURE 1. A 24-bit RGB image. Each layer contains 8-bit color information for the primary colors red, green, and blue. All three layers added together yield a certain color.
theory and the way color models are built. Color is the perceptual result of light in the visible region of the spectrum, with wavelengths of 400–700 nm, falling on the retina. Physical power (or radiance) is expressed in a spectral power distribution (SPD), often in 31 components, each representing a 10-nm band. SPDs exist in the physical world, but color exists only in the eye and the brain—it is a perceptual phenomenon. Or, as Sir Isaac Newton stated, “Indeed rays, properly expressed, are not colored.” The human retina has three types of color photoreceptor cone cells, which respond to incident radiation with different spectral response curves. Because there are exactly three types of color photoreceptors, three numerical components can be used to describe a color, provided that appropriate spectral weighting functions are used. This is the concern of the science of colorimetry. In 1931, the Commission Internationale de L’Éclairage (CIE) adopted standard curves for a hypothetical “standard observer”. These curves specify how an SPD can be transformed into a set of three numbers that are the mathematical coordinates of color space. Their function is analogous to coordinates on a map. Cartographers have different map projections for different functions: Some map projections preserve areas, others show latitudes and longitudes as straight lines. No single map projection fills all the needs of map users. Similarly, no single color system fills all of the needs of color users; each color system has advantages and disadvantages because it is usually optimized for a certain purpose. The systems in use today for theoretical color specification include CIE XYZ, CIE Luv, and CIE Lab (9), which have been carefully designed to approach perceptual linearity to a human observer. (Human vision is nonlinear regarding perception of brightness. CIE Luv and Lab aim to compensate for this. A source with a luminance [brightness or darkness] only 18% of a reference luminance appears about half as bright as the reference. The perceptual response to luminance is
284 A
A N A LY T I C A L C H E M I S T R Y / A P R I L 1 , 2 0 0 0
(a) Blue (0, 0, 255)
Cyan
Magenta
al e
0
sc
255 0
ay
Green layer
Gr
Blue layer
called lightness.) However, these systems are not useful for image representation because of their cumbersome transformation to common imaging color systems such as RGB. A digitized color image is represented as an array of pixels, in which each pixel contains three numerical components that define a color. In theory, the three numerical values for image coding could be provided by a color specification system such as CIE. But a practical image coding system needs to be computationally efficient, generally needs to cover only a reasonably wide range of colors (but not all of the colors), cannot afford unlimited precision, and need not be intimately related to the CIE system. So, image coding uses different systems than color specification. The systems useful for image coding are RGB and derivatives thereof such as YCbCr (which is used for digital TV transmissions in which the color information is split into the luminance component Y and two chrominance components Cb and Cr) and hue–saturation–value (HSV) (10). What all these models have in common is that color information is encoded in a
Green (0, 255, 0) Black
Red (255, 0, 0)
Yellow
(b)
FIGURE 2. (a) The RGB cube and (b) additive color mixing. (a) Every color is defined by a certain point within the cube, whereas the gray shades lie along the diagonal black and white axes. (b) On the left is a mixture of red, green, and blue, and the inverse is on the right. By mixing different intensities of these base colors, almost all other colors can be produced in the RGB color model.
Blue (240°) Magenta
Cyan
three-dimensional parameter space, but they differ significantly in the way color is mapped to the corresponding color space. Any color that can be specified using a model will correspond to a single point within the subspace it defines. What follows is a more detailed description of RGB and HSV, the two most useful models for color fusion. The other color models mentioned either are similar to these models or are not suitable for the given purpose.
human eye perceives it; that is, by the hue (H), saturation (S), S H Green (120°) Red (0°) and value (i.e., intensity [V]). Yellow Albert H. Munsell, a pioneer in color sciences and inventor of the V perception-based Munsell color system (similar to HSV), defined these qualities: “Hue is the quality by which we distinguish one color family from another, as red from yellow or green from blue or purple. Black Chroma [the S in HSV] is that FIGURE 3. The HSV cone. quality of the color by which we distinguish a strong color from a H is measured in a circle starting at red. S is the distance from the axis. Colors on the surface of the cone are fully saturated (pure colweak one; the degree of deparors), and the gray-scale spectrum is located on the middle axis. For ture of a color sensation from these colors, H is undefined. RGB that of a white or gray. Value is In the RGB model, an image the quality by which we distinconsists of three independent image planes, where each guish a light color from a dark one” (11). layer contains the values of one of the primary colors red, The HSV color space is cylindrical but is usually repregreen, and blue (Figure 1). A particular color is specified sented as a cone, as shown in Figure 3. H is the angle by indicating the amount of each of the primary compo(0–360°) around the vertical axis (the intensity or value nents present. Figure 2a shows the geometry of the RGB axis). It increases in value when moving counterclockwise color model for specifying colors using a Cartesian coordi- in the cone (looking down the vertical axis from the top nate system. This is an additive model—the colors present of the pyramid), beginning with red at 0°. S is a ratio that in the light add to form new colors. This model is approranges from 0 to 1 and defines the distance from the vertipriate for mixing colored light. Figure 2b shows the addical axis, describing the amount of white light added to the tive mixing of red, green, and blue primaries to form the pure H (i.e., at the center where saturation is 0, there is three secondary colors yellow (red + green), cyan (blue + only white light and no color). The V component is the green), and magenta (red + blue), as well as white (red + vertical axis pointing up in the figure, carrying the inforgreen + blue). mation about the intensity (i.e., the brightness of the Numerically, any color would be expressed as the color, ranging from a minimum brightness of 0 to a maxiamount of red, green, and blue present, for example, red mum brightness of 1). would be (255, 0, 0), green would be (0, 255, 0), and Smith defines it vividly: “The HSV model mimics the yellow would be (255, 255, 0). The contribution of each way an artist mixes paint on his palette. He chooses a pure of the primaries ranges from 0 to 255 for mathematical hue, or pigment, and lightens it to a tint of that hue by reasons: Computers usually address colors in 24-bit space, adding white, or darkens it to a shade of that hue by with each primary having 8-bit space (28 = 256). This adding black, or in general obtains a tone of that hue by principle makes RGB a useful color model for computer adding some mixture of white and black or gray” (12). monitors, but when it comes to human color perception, This intuitive approach to human color perception makes the model proves to be unintuitive in use, that is, it does HSV an excellent model when color selection and mixing not resemble the way we select colors. Smith, the inventor are the main tasks. However, its use with computers is not of the HSV model, says, “Try this mixing technique by as straightforward as the RGB system. HSV and related mentally varying RGB to obtain pink or brown. It’s not color models, such as hue–saturation–lightness, hue–value– unusual to have difficulties” (12). Nevertheless, RGB is chroma, and chroma–colorfulness–intensity, (10) are all broadly used in many fields and supported by almost any nonlinear transforms of the RGB model, making calculagraphics application, which is a big advantage over many tions more complicated. less commonly used color models. In addition, all other color models can be mapped onto an RGB representation Transforming analytical information into colors by a formula. The next question is how to convey the analytical image information into these color spaces to increase the inforHSV mation density. Multispectral analytical images are usually In the HSV color model, color is represented as the conveyed as a number of 8- or 16-bit gray-scale intensity
A P R I L 1 , 2 0 0 0 / A N A LY T I C A L C H E M I S T R Y
285 A
distributions, each showing different aspects of a sample (different masses, secondary electron information, backscattered electron information, etc.). The original idea was to use these intensity distributions and assign them to the different color spaces in various ways, such as by filling the three-parameter space of the color models with the intensity information obtained from the analytical source image. When more than one source image is used for color-space parameter assignment, the resulting image has a higher information density than the original images because it combines the information of all the source images. In other words, the resulting image is a fusion product of the source images. This is in contrast to the false coloring of analytical images, which can improve the optical distinction but does not introduce any new information to the image. The characteristics of the images also have to be considered when choosing an appropriate fusion method because each image type has its specific requirements. Image arrays are roughly classified as two or more images with large
overlapping or nonoverlapping areas; one image with large areas and other images showing occlusions or border structures; and no images with large areas and having only occlusions or border structures. With the RGB color system, the obvious way to transform the source images to color space is by assigning each image to one of the primary colors red, green, or blue (Figures 4a–d). The result is color images that display one of the primary colors in areas where only one of the source images has noticeable intensities, and mixture colors in areas where two or three source images have intensities. However, these mixture colors are also the biggest disadvantage of the RGB approach; the way the mixture colors are produced is not user-oriented and does not conform to the way the human eye interprets mixture colors (i.e., shading and intensity) but instead is machine-oriented and more abstract and mathematical. Interpretation of such mixture colors becomes difficult because there is no direct correlation between the perception of a certain RGB color and understanding the composition of the
FIGURE 4. The most common fusion methods. Lateral intensity distribution of (a) 12C, (b) 43Al (aluminum oxide), and (c) 68Cr (chromium oxide) in an aluminum-alloyed hipped steel sample (16) measured by SIMS. The brightness of each pixel represents the information recorded by the mass spectrometer in a dynamic range of 65,535 gray-scale steps. (d) RGB fusion in which each of the source images was assigned to the color channels red, green, and blue. Distinguishing the three elements is now easier; however, the mixture colors orange and violet, produced by areas where at least two elements show enrichment, are difficult to interpret. The blue indicates that this part of the sample consists predominantly of aluminum. The green shows areas of chromium enrichment. However, the distribution of carbon, which should be shown in shades of red, cannot be clearly identified because all areas are overlaid either by aluminum, chromium, or both, thereby producing mixture colors that cannot be easily transformed to intensity information by a human observer. (e) The result of HSV fusion with constant H, formed using (a) as the base image for the distribution of carbon (because of its homogeneous distribution throughout the entire image); the carbon distribution was assigned to the V parameter. (b) and (c), indicating the distributions of aluminum and chromium, respectively, were assigned (fixed) the H values 0 and 120, respectively, and the S parameter was coded by their intensity distributions (aluminum and chromium, respectively). A classification algorithm is used to decide, for each pixel, whether aluminum or chromium contributes its H and S value, that is, which of the two masses predominates at a given location in the image. The result is a gray-scale image of the carbon distribution, indicated by the light gray areas, overlaid by two colors showing the distribution of aluminum (red) and chromium (blue) within the carbon matrix of the steel sample. The saturation of each color is proportional to the measured intensity of the respective element. (f) The result of HSV fusion with constant saturation, formed by assigning (a) to the V parameter, and (b) and (c) assigned a separate H range of 90° each, according to their intensity values, and the saturation parameter set to 1. A classification algorithm is again used to determine, for each pixel, whether image (b) or (c) contributes its H value. The result is a color image that shows the carbon distribution as the light blue background with the aluminum (green) and chromium (dark blue) superimposed. Both HSV fusions very clearly show the lateral distribution of the three different elements and retain the intensity distributions of the source images as well as possible. In the RGB fusion, the contrast of the aluminum and chromium is not as high.
286 A
A N A LY T I C A L C H E M I S T R Y / A P R I L 1 , 2 0 0 0
underlying source images. For example, no one would intuitively understand that dark brown represents an intensity of 124 in image 1, 46 in image 2, and 23 in image 3. In addition, RGB fusion has little flexibility and limits the number of images that can be fused to three (two images can be fused with the RGB model when a third black “dummy” layer is used as one of the channels). Nevertheless, RGB fusion is a classic approach that is easy, universally applicable, and normally produces acceptable-to-good results. It is especially useful for image arrays with two or more images that have large nonoverlapping areas and images with no large areas but only occlusions or border structures. These arrays do not require a base image as does HSV fusion, and perception of small structures is usually better with RGB than HSV. There are various possibilities with HSV for assigning source image values to the components. One possibility for HSV fusion is to have two source images separately encode the saturation and the value parameters and keep the hue fixed. For example, the intensities of source image 1 are used for V, the intensities of source image 2 are used for S, and H values are all set to a fixed degree. In Figure 5a, H would be set to 0 for the color red, which gives a good impression of the lateral distribution of the overlay image relative to the base image. Alternatively, S can remain fixed to a maximum, and the two source images separately encode V and H, which highlights the intensity distribution of the overlay image in Figure 5b. Both methods are suitable for image arrays that have two or more images with large overlapping areas and that have one image with large areas and other images showing occlusions or border structures. If more than two images are to be fused, the base image is assigned the V parameter, and each additional overlay image is assigned a certain H (e.g., red for overlay image 1 and blue for overlay image 2). For each pixel in the fusion image, a classification algorithm decides which of the overlay images may contribute its H, resulting in differently colored areas depending on which overlay element is predominant in a certain region of the image (Figures 4e, 4f, 5d, and 5e). Again, the intensity distribution of the base image is preserved, as well as the lateral distribution of the overlay images. One restriction is that large overlapping areas cannot be handled in this way because of the use of the EITHER–OR classification, which can assign only one unique color to each pixel. One presupposition that is common for all HSV methods is that the base image, which is always used for assigning V, should have a large area intensity distribution because image areas with a low V do not show H and S variations of the overlay images. On the other hand, the big advantage of this method is that V completely retains the intensity distribution information from the original base image (Figures 5c and 5d). A problem arises when
FIGURE 5. Images are from a sample of niobium–tungsten–zirconium alloy coated with a silicon–chromium–iron layer for better oxidation resistance (17). (a) Occlusions of zirconium (red) within the large-area niobium phase. This image was produced using HSV fusion with constant H (red). As an alternative, (b) shows the same two elements, but H varies with the intensity of zirconium, that is, the base structure shows the niobium, and the colors show the intensity of zirconium within niobium, ranging from yellow (low concentration) to green to blue (high concentration). (c) Distribution of boron and chromium in niobium using RGB fusion. The niobium phase is red, chromium is green, and boron is blue. Mixture colors are difficult to assign to elements. If the same three elements are fused using HSV with constant H (red and blue), the result is (d), in which the gray-scale base structure shows the niobium distribution, and the two other elements are superimposed using different colors (red is chromium, and blue is boron). (e) Distribution of sodium (red) and aluminum (blue) in tungsten (gray-scale base image) using HSV fusion with constant H. Because of the necessary decision of which overlay image contributes its H and S for each pixel, the V value of the other image at that pixel is lost. However, with nonoverlapping features, this is not a problem. (f) Fusion result of two large overlapping areas using HSVmix. The normal HSV fusion process produces unsatisfying results because of its exclusive “OR” character (the OR in EITHER– OR); however, by mixing H and S values of the overlay images, the spatial location of all three elements can be ascertained while keeping most of the intensity information.
three images with large overlapping areas have to be fused, because the classification algorithm can only make an OR decision. Therefore, a new “HSVmix” algorithm is used, which calculates the average H and S of the overlay images and allows color fusion of multiple large areas (Figure 5f). Another major advantage of HSV fusion is the way
A P R I L 1 , 2 0 0 0 / A N A LY T I C A L C H E M I S T R Y
287 A
color is mixed, which corresponds to the way the human eye interprets colors in terms of tone, color intensity, and brightness. Each of these qualities serves a certain defined purpose and repFIGURE 6. Typical examples of color image fusion for different analytical images. resents the information of one source image. There are no mix(a) RGB fusion of three EPMA element distribution images of a soldering joint containing nickel (red), iron (green), and silicon (blue). This image system is ideal for RGB fusion because there are three major nonoverlapping phases and no ture colors that cannot be base image. (b) HSV fusion with constant saturation of two EPMA images of corroded glass, highlighting sulfur crystals assigned to a corresponding (shades of blue that correspond to intensity) in calcium (green). (c) Fusion of multiple images with HSV and constant H. source. The gray-scale intensity A secondary-electron base image recorded with SEM is superimposed with the element distribution images of lead (red), copper (blue), manganese (violet), and tin (green) (18). distribution of an HSV fusion image with fixed hues, for example, always clearly corresponds to (1) Hutter, H.; Brunner, C.; Nikolov, S. G.; Mittermayr, C.; Grasserbauer, M. the intensity distribution of the base source image, whereas Fresenius’ J. Anal. Chem. 1996, 355, 585–590. the S of the colored areas clearly corresponds to the inten(2) Hawkes, D. J.; Hill, D. L. G.; Bracey, E. C. M. L. In Cardiovascular Nuclear sity distribution of the overlay source images. This also Medicine and MRI; Reiber, J. H., van der Wall, E. E., Eds.; Kluwer Acareduces the total number of colors used in the fused image demic Publishers: Dordrecht, Germany, 1992; 113–130. (13). (3) Rehm, K.; Strother, S. C.; Anderson, J. R.; Schaper, K. A.; Rottenberg, Generally, it is best to use gray values for fine detail D. A. J. Nucl. Med. 1994, 35, 1815–1821. and reserve chromatic color for attracting attention (14), (4) Maitre, H. Proceedings of the 9th SCIA, Scandinavian Conference on Image which is possible with HSV fusion. This is because the Analysis, Borgefors, G., Ed.; World Scientific: Uppsala, Sweden, 1995; luminance component in a color is critical for carrying Vol. 1, pp 139–153. information about high spatial frequencies (i.e., the highly (5) Bloch, I. IEEE Trans. Syst., Man, and Cybernetics Soc., A: Systems and varying “detail” components of an image submitted to freHumans 1996, 26, 52–67. quency analysis), whereas the H and S components carry (6) Data Fusion in Robotics and Machine Intelligence; Abidi, M. A., Gonzalez, information about low spatial frequencies (i.e., the littleR. C., Eds.; Academic Press: New York, 1992. varying “smoothed” base structure of an image submitted (7) Freiherr, G. Diagnostic Imaging 1988, 146–154. to frequency analysis) (15), which is why HSV fusion is so (8) Kamman R. L.; Stomp, G. P.; Berendsen, H. J. C. Magn. Reson. Med. successful in many cases. In Figure 6c, the base image is a 1989, 9, 240–253. high-resolution secondary-electron image, and the overlay (9) Colorimetry, 2nd ed.; Publication CIE No. 15.2; Central Bureau of the images are low-resolution element distribution images of Commission Internationale de L’Eclairage: Vienna, Austria, 1986. the same area. The figure also shows further examples of (10) DeMarsh, L. E.; Giorgianni, E. J. Phys. Today 1989, 44–52. color fusion applications with images from different sur(11) Munsell, A. H. A Color Notation; Munsell Color Co.: Baltimore, MD, 1946. face sources.
And the winner is … Although it is possible to increase the information density of multispectral analytical images by fusing their information using color modeling, there is no ideal color fusion approach. Human perception is such an intricate process that there appears to be no ubiquitous best strategy for color assignment. Nevertheless, “intuitive” color systems such as HSV usually have big advantages over the hardware-oriented color systems such as RGB and are also much more flexible. In any case, the characteristics of the images must be considered when evaluating the ideal color fusion method, because their color representation requirements can differ significantly. This work was financially supported by the Austrian Fund for Science and Research (FWF Project P13,160-CHE).
References 288 A
A N A LY T I C A L C H E M I S T R Y / A P R I L 1 , 2 0 0 0
(12) (13) (14) (15)
(16) (17) (18)
Smith, A. R. Computer Graphics 1978, 12, 12–19. Foley, J. D.; van Dam, A.; Feiner, S. K.; Hughes, J. F. Computer Graphics, Principles and Practice; Addison-Wesley: Reading, MA, 1990. Murch, G. Tekniques 1984, 8, 4–9. Rogowitz, B. E.; Treinish, L. A. In Proceedings of 1996 Data Explorer Symposium; IBM Thomas J. Watson Research Center: Yorktown Heights, NY, 1996. Pollak, C.; Kriszt, B.; Hutter, H. In Conference Proceedings of 1999 Solid State Physics Vienna, Microchim. Acta, submitted for publication, 1999. Gritsch, M.; Piplits, K.; Barbist, R.; Wilhartitz, P.; Hutter, H. Materials and Corrosion, submitted for publication, 1999. Schreiner, M.; Woisetschläger, G.; Schmitz, I.; Wadsak, M. J. Anal. At. Spectrom. 1999, 14, 395–403.
Thomas C. Stubbings is a research associate and Herbert Hutter is an associate professor at the Vienna University of Technology. Stubbings’ research interests include multispectral data analysis, neural networks, image processing, and visualization. Hutter’s research interests include characterizing solid-state materials and applied surface
