Anal. Chem. 2007, 79, 5424-5428
Augmenting Spectroscopic Imaging for Analyses of Samples with Complex Surface Topographies Michael K. Gilbert and Frank Vogt*
Department of Chemistry, University of Tennessee, Knoxville, Tennessee 37996
Spectroscopic imaging has become a widely used tool for analyses of heterogeneous samples. Focal plane array detectors are incorporated into spectrometers that acquire a large number of spectra from different sample locations in parallel. This sensing technique facilitates analyses of spatial distributions of chemical information in an X-Y plane at high time resolution. In many cases, chemical reactions proceed in three spatial dimensions (X-Y-Z) and require the acquisition of spectroscopic information in an X-Y plane plus topographic (Z-dimension) information. However, capturing two-dimensional (2D, i.e., X-Y) images from three-dimensional (3D, i.e., X-Y-Z) samples inherently loses Z-dimension information. This technical note describes an augmented spectroscopic imager that gains both types of data, i.e., spatially resolved spectroscopic information and topography. For the latter purpose, a regular light pattern is generated and projected onto a sample. Due to its 3D topography, this light pattern is distorted. After extracting these distortions, the topography can be determined since the height structure is encoded in the light pattern. Because topographic probing must not affect infrared measurements, different wavelength ranges are used. Here spectroscopic information is acquired in the mid-IR while the light pattern probing the topography is generated in the visible. For relating distortions to physical height structures, the setup needs to be calibrated. For this purpose, calibration objects of known dimensions have been manufactured onto which the light pattern is projected. Determining distortions introduced by objects of known height derives a transform from distortions to topographies. Due to mechanical restrictions, the light pattern can only achieve a certain spatial resolution. In order to enhance the spatial resolution the topography is probed with, scanning the light pattern in X- and Y-direction is proposed. In traditional optical spectroscopy, the light beam interacts with only a small portion of the sample. If heterogeneous samples are analyzed, measurement data gained by such a localized technique are nonrepresentative for the majority of the sample. This renders such an approach inadequate for investigations involving large areas of heterogeneous samples. In order to overcome this limitation, spectrometers are equipped with two-dimensional (2D) * Corresponding author. Phone: +1 (865) 974-3465. Fax: +1 (865) 974-3454. E-mail:
[email protected].
5424 Analytical Chemistry, Vol. 79, No. 14, July 15, 2007
detector arrays. In spectroscopic (or hyperspectral) imaging,1-17 spectrometers capture stacks of images at consecutive wavelength positions. In other words, a spectrum is acquired at every single pixel of the detector array, resulting in data sets containing thousands of spectra. Since these spectra are measured from different sample locations, spatial distributions of chemical information in an X-Y plane can be investigated. However, when acquiring 2D images from nonflat (i.e., 3D) objects, topographic information (Z-dimension) is inherently lost. This information is important for studies of chemical reactions proceeding in three spatial dimensions such as growth of biomedical tissue.14-16,18 Another example is industrial quality monitoring where objects with complex shapes need to be analyzed, for instance, to detect local contaminations.7,8,10 For these and many more measurement tasks, spectroscopic imaging must be expanded to determine surface topographies in addition to spectroscopic data. Stereo imaging19 has been introduced to reconstruct surface topographies. This technique is based on acquiring images of samples from two different positions and then reconstructing 3D information. Applications of stereo vision range from remote sensing for tracking of objects20 to scanning electron microscopy imaging to study surface roughness;21 the latter technique tilts the sample to image it from a different angle. Another approach, (1) Geladi, P.; Grahn, H. Multivariate Image Analysis; John Wiley & Sons: Chichester, 1996. (2) Colarusso, P.; Didder, L.; Levin, I.; Fraser, J.; Arens, J.; Levis, E. Appl. Spectrosc. 1998, 52, 106A-120A. (3) Wold, J.; Kvaal, K. Appl. Spectrosc. 2000, 54, 900-909. (4) Shaw, G.; Manolakis, D. IEEE Signal Process. Mag. 2002, 19, 12-16. (5) Landgrebe, D. IEEE Signal Process. Mag. 2002, 19, 17-28. (6) Manolakis, D.; Shaw, G. IEEE Signal Process. Mag. 2002, 19, 29-43. (7) Lawrence, K.; Windham, W.; Park, B.; Buhr, R. J. Near Infrared Spectrosc. 2003, 11, 269-281. (8) Liu, Y.; Windham, W.; Lawrence, K.; Park, B. Appl. Spectrosc. 2003, 57, 1609-1612. (9) Tran, C. Appl. Spectrosc. Rev. 2003, 38, 133-153. (10) Yu, H.; Macgregor, J. Chemom. Intell. Lab. Syst. 2003, 67, 125-144. (11) Vogt, F.; Dable, B.; Cramer, J.; Booksh, K. Analyst 2004, 129, 492-502. (12) Vogt, F.; Banerji, S.; Booksh, K. J. Chemom. 2004, 18, 350-362. (13) Vogt, F.; Cramer, J.; Booksh, K. J. Chemom. 2005, 19, 510-520. (14) Beleites, C.; Steiner, G.; Sowa, M.; Baumgartner, R.; Sobottka, S.; Schackert, G.; Salzer, R. Vib. Spectrosc. 2005, 38, 143-149. (15) Fernandez, D. C.; Bhargava, R.; Hewitt, S. M.; Levin, I. W. Nat. Biotechnol. 2005, 23, 469-474. (16) Romeo, M. J.; Diem, M. Vib. Spectrosc. 2005, 38, 115-119. (17) Vogt, F. Curr. Anal. Chem. 2006, 2, 107-127. (18) Diem, M.; Romeo, M.; Boydston-White, S.; Miljkovic, M.; Mattha¨us, C. Analyst. 2004, 129, 880-885. (19) Cheung, Y.; Zhang, Y. Meas. Sci. Technol. 2006, 17, 3221-3228. (20) Lee, J.-S.; Deo, C.-W.; Kim, E.-S. Opt. Commun. 2001, 200, 73-85. (21) Pouchou, J.-L.; Boivin, D.; Beaucheˆne, P.; Le Besnerais, G.; Vignon, F. Mikrochim. Acta 2002, 139, 135-144. 10.1021/ac070518m CCC: $37.00
© 2007 American Chemical Society Published on Web 05/26/2007
Figure 1. (left) Step 1: FT-IR spectroscopic imaging in reflection, transmission, or ATR mode probes samples at high spatial resolution. Step 2: Samples are illuminated with a light pattern in the visible-wavelength range; this pattern gets distorted upon projection onto 3D samples (see image in top left) and topographic information will be gained from these distortions. (right) Picture of the illumination optics for generating and projecting a light pattern onto samples.
spatial heterodyne interferometry,22 utilizes a Mach-Zehnder interferometer to split a laser beam in two; one beam is reflected from a flat reference surface and the other from the surface whose topography is analyzed. Due to different sample heights at different locations, a phase shift is introduced into the measurement beam relative to the reference beam. Scanning the laser beam over the sample and evaluating the induced phase shifts derives information about surface topographies. However, these imaging techniques do not involve spectroscopic information and thus cannot determine the spatial distribution of different analytes. In order to facilitate spectroscopic imaging of objects with complex topographies, we propose to perform spectroscopic imaging in two steps: (step 1) Conventional Fourier transform infrared (FT-IR) spectroscopic imaging will be used to derive analyte distributions at high spatial resolution. (step 2) Custommade illumination optics project a regularly shaped light pattern in the visible-wavelength region onto 3D samples (Figure 1). This light pattern, i.e., alternating illuminated and shadowed areas, is generated by placing a micromesh into the beam path. If the samples are flat, the light pattern is preserved. If the objects have a nonflat surface structure, the light pattern is distorted accordingly. After extracting the distorted light pattern from images, the distortions can be related to sample topographies. Combining both types of information will enable the determination of analyte distributions on the surface of samples with (22) Tobin, K.; Bingham, P. Proc. SPIE 2006, 6162, paper 6162-03.
complex topographies. Because spectroscopic imaging setups have become commercially available, this study focuses on the second step, i.e., deriving topographic information. EXPERIMENTAL SETUP AND FIRST RESULTS The setup used in this study is based on a commercial FT-IR spectroscopic imager. This system consists of a Bruker Vertex 70 spectrometer to which a Hyperion 1000 microscope is connected; this microscope features a 64 × 64 pixels focal plane array (FPA) and a CCD camera for the visible-wavelength region (Sony Exwave HAD digital video camera). The microscope can be operated in reflection or transmission (15× magnification) mode as well as in attenuated total reflection (ATR) (20× magnification) mode. An illumination unit projecting a light pattern onto samples has been developed in-house and is depicted in Figure 1 (right). The diverging light emitted from a blue LED is collimated into a parallel beam, which is first transmitted through a micromesh (Figure 2, left) and then focused onto the sample by a second microscope objective. This illumination unit operates in the visiblewavelength range in order not to affect infrared measurements. For this same reason, the mesh must be kept outside of the IR beam path because otherwise artifacts would be contained in the IR images. This would make extraction of the pattern difficult and would block a certain amount of pixels on the IR detector array. Another reason for choosing the visible-wavelength range is that CCD cameras usually have larger FPAs (here, 370 × 280 pixels), Analytical Chemistry, Vol. 79, No. 14, July 15, 2007
5425
Figure 2. (left) Image of the micromesh used in these studies. The hole dimensions are 100 µm × 100 µm, and wire thickness is 30 µm. The transmission of such micromeshes typically ranges from 30 to 70%. (center) This image shows a piece of a pharmaceutical without mesh in the illumination path. (right) The mesh placed in the beam path reduces the overall light level but retains sufficient contrast. The white dotted line (a) highlights a distortion of the light pattern, and (b) points out a discontinuity in the pattern induced by the height differences between sample and substrate.
Figure 3. (top left) Image of a microscopic sample acquired by the CCD camera (step 2 in Figure 1). (top right, gray graph) Signal measured along the column indicated by a white arrow in the top left image; the locations marked by (1)-(8) correspond to local signal minimums and shadow areas which receive less light; (top right, black graph) signal after 2D FFT bandpass filtering the image (see text). Now the local minimums are clearly visible at highly improved signal-to-noise ratios; (bottom left) the light pattern extracted from the raw image is marked by white dots overlaying the image; (bottom right) extracted light pattern marked by white lines overlaying the image after utilizing a 2D FFT bandpass filter.
which facilitate a better resolution of the topography than the lower resolution infrared-FPAs. The micromesh (Figure 2, left) also acts as a gray filter that reduces the light level for all wavelengths. Due to this, it first needs to be ensured that the remaining light level still facilitates a good contrast between shadowed and illuminated areas. In Figure 2 (center and right), images of a microscopic pharmaceutical sample are shown with and without the mesh being present. The mesh somewhat reduces the light level, but nonetheless, enough light is transmitted through the mesh to ensure sufficient contrast. Furthermore, one has to keep in mind that the mesh is only involved in topographic analyses, which does not affect the intensity of the IR radiation. The micromesh chosen for these studies was a good compromise between light reduction and pronunciation of the light pattern. Using a micromesh with thinner wires and larger apertures may transmit more light but would also generate a less pronounced shadow. This would make the detection and extraction of the light pattern less reliable. Also, the mesh shown has a limited ruggedness and thinner wires could cause an increased sensitivity to airflows. On the other hand, thicker wires and smaller apertures would cover too much of the 5426
Analytical Chemistry, Vol. 79, No. 14, July 15, 2007
sample with shadow and thus would reduce contrast and the spatial resolution of the topography. Figure 2 (right) is a first example demonstrating the distortions of the light pattern due to the sample’s surface topography; the curved white line highlights the pattern generated by a selected wire. Once image acquisition is complete, the distorted light pattern must be extracted. In some cases, it might be advantageous to average over several images in order to enhance the signal-tonoise ratio. Although the extraction algorithm as described below could determine the light pattern well without averaging, 20 images have been averaged. As the camera has a frame rate of several tens of hertz, this does not reduce the time resolution of the analysis significantly. The image shown in Figure 3 (top left) is an example by means of which the custom-made extraction algorithm will be discussed and assessed. The gray graph in Figure 3 (top right) shows the signal along the column marked with a white arrow in the top, left picture. The numbers 1-8 given in both parts indicate the local signal minimums caused by shadows of different mesh wires. In order to search for these local minimums, the extraction algorithm moves a window covering N measurement points stepwise over a column. This is done for each
Figure 4. Enhancing spatial resolution of the surface structure, the micromesh by scanning stepwise over the sample in Xm- and Ymdirection. The subscript m(esh) refers to the Xm-Ym plane of the mesh (Figure 1, center), which does not coincide with the X-Y plane shown in Figure 1 (bottom, center). After extracting the light pattern at each position, all extracted patterns are combined. This enhances the spatial resolution of the sample’s topography (compare Figure 5) and allows a better assessment of (here) the sample’s ridge structure.
column consecutively; then an identical approach moves the window over all rows one at a time. At each window position, the minimal signal reading inside this window is determined. There are three different types of minimums (see gray arrows in Figure 3, top right): (a) The minimum signal reading is found at the right border of the window; such a window covers a downhill part of the signal and a local minimum has not yet been reached. (b) The minimum is found at the left border of the window; here the window covers an uphill portion of the signal and a local minimum has been passed. (c) If the minimum value is found inside the window, a local minimum is contained inside this window. In this case, the pixel having the smallest value is marked. It is important to adapt the window length to the given data. If the window length is too long, it may cover more than one local minimum and the detection is ambiguous. Conversely, if the window is too narrow, small noise spikes might be misinterpreted as mesh shadow. In preliminary studies, a window length of N ) 10 measurement points was found to be adequate for the setup used here. Figure 3 (bottom, left) shows the light pattern extracted from Figure 3 (top, left). The local minimums found by the aforementioned extraction algorithm have been marked with white dots. Obviously, the general shape of the light pattern has been derived. However, the white dots do not line up to a continuous line and many minimums have been found outside the mesh’s shadow. These misclassifications are due to noisy image data, showing that improvements in the signal-to-noise ratio must be made. One approach is to bandpass filter the images. The underlying idea is that the mesh induces a more or less regular, sine-shaped signal (Figure 3, top right, gray graph) along the rows and columns. Due to distortions of the light pattern, the frequency of this sine may change somewhat from row to row or from column to column. In order to enhance the signal-to-noise ratio, frequency components that are clearly different from the frequency of the “sine” are not due to the light pattern and must be suppressed. Only frequencies close to that of the sine contain relevant information about the light pattern. Frequencies much higher are certainly noise; frequencies much lower are not related to the light pattern either. Images are first 2D fast Fourier transformed (FFT)23 and then have a bandpass filter applied in the Fourier domain to (23) Press, W.; Teukolsky, S.; Vetterling, W.; Flannery, B. Numerical Recipes in C, 2nd ed.; Cambridge University Press: New York, 1992.
Figure 5. Demonstration of enhancing the spatial resolution of the topography probing by stepwise scanning the mesh in the Xm-Ym plane. The mesh in the bottom image is shifted to the right relative to the top image; the white dashed lines serve as markers. The arrow shown at the top gives an approximate length scale; spatial resolutions on the micrometer scale are feasible.
enhance the relevant and to suppresse the disturbing frequency components. Such a bandpass filter has to be empirically finetuned regarding center and width to the light pattern generated by certain illumination optics. In Figure 3 (top, right), the signal of the same column is shown in black after bandpass filtering. The enhanced signal-to-noise ratio is clearly visible; now the signal minimum induced by the mesh’s shadow also becomes better defined. Due to this, the search for local minimums is much less ambiguous and the extracted light pattern (Figure 3, bottom, right) is better defined. Furthermore, since the lines of the light pattern are now continuous, the determination of the surface topography will be more precise. The number of shadow lines projected onto a sample determines the spatial resolution with which the topography can be determined. In Figure 4, the distortion of the light pattern reveals a ridge running across the piece of pharmaceutical; however, only a few lines cover the sample. In order to better probe the topography, more shadow lines must be projected onto the samples. In order to achieve this, a finer mesh could be chosen that would introduce mechanical challenges as mentioned before. Alternatively, one could choose a large magnification factor of the second microscope objective. The resulting, smaller focal depth, however, requires the second microscope objective being placed very close to the sample. Both objectives, i.e., IR microscope and illumination optics, have typical working distances of only a few millimeters. Bringing them close together is often not possible since there is simply not enough space for both objectives. Another more feasible solution is to scan the mesh. For this Analytical Chemistry, Vol. 79, No. 14, July 15, 2007
5427
Figure 6. (left) Schematic depicting the use of calibration objects for relating their known heights (in µm) to distortions of the light patterns (compare Figures 3-5) induced by them. (right) These microscopic calibration objects (only two are shown) have been milled out of a white Teflon slide. The footprint (∼250 µm × 250 µm) and the heights of the pillars have been determined under a microscope. After projecting a light pattern onto one or several of these pillars, the pixel shift-to-height relation can be determined.
purpose, a U-shaped holder of the micromesh was mounted onto an XY translation stage. While the mesh is scanned stepwise in Xm- and Ym-direction (see Figure 1, Figure 4), images are acquired at each position. From each of these images, the (distorted) light pattern is extracted and combined to a final topographic image featuring an enhanced spatial resolution. In Figure 5 (top), extracted distortions generated by the unshifted mesh are shown. Dashed white lines are used as markers to visualize the shift of the light pattern to the right in the bottom image. The depicted shift has been obtained after five steps to the right. Thus, a spatial resolution of the topography on the micrometer scale is feasible. Once the light pattern and its distortions have been extracted, these distortions measured in pixels must be related to the samples’ true physical height (i.e., topography) in micrometers. For this purpose, one or several “calibration objects” of known dimensions are placed on the microscope stage and are illuminated with the light pattern. Now the extraction algorithm determines the induced distortions and derives a translation from measured pixel shifts to known heights (Figure 6, left). Using a flat substrate as a “baseline” will establish the undisturbed light pattern (no height). From this the amount of distortion caused by the calibration, objects can be measured and related to calibration heights. For preliminary investigations, pillars of different heights (115 ( 2.4, 200 ( 1.3, 211 ( 1.5, 250 ( 2.5, and 310 ( 1.4 µm) have been milled out of a Teflon slide (Figure 6, right). The footprint and the heights of these pillars have been determined under a microscope in topview and sideview, respectively. CONCLUSIONS A microspectroscopic imaging setup has been augmented in order to gain topographic information of the samples along with spectroscopic data. Determining the topography of samples is (24) Martens, H.; Næs, T. Multivariate Calibration, 2nd ed.; John Wiley & Sons: New York, 1991. (25) Massart, D.; Vandegonste, B.; Buydens, L.; DeJong, S.; Lewi, P.; Smeyers, J. Handbook of Chemometrics and Qualimetrics; Elsevier: Amsterdam, 1997. (26) Jolliffe, I. Principal Component Analysis, 2nd ed.; Springer: New York, 2002. (27) Malinowski, E. Factor Analysis in Chemistry, 3rd ed.; John Wiley & Sons: New York, 2002.
5428
Analytical Chemistry, Vol. 79, No. 14, July 15, 2007
important for chemical sensing of reactions that proceed in three spatial dimensions, such as the growth of biological material. In industrial quality control, samples with complex surface structures have to be probed; one example is detection of contaminations in pharmaceutical or food products. In a first analysis step, conventional spectroscopic imaging data are acquired from which chemical information (composition, concentrations) is derived by means of chemometric methods.11,24-27 In a second step, the samples’ topographies are determined. For the second analysis step, additional illumination optics for the visible-wavelength range have been developed that project a light pattern onto the 3D samples. This light pattern is generated by placing a micromesh into the illumination beam path. If this light pattern is projected onto the flat substrate, it is preserved. If, however, the samples have a 3D surface topography, the regular light pattern is distorted. Information about surface structures is encoded into these distortions. By means of a novel algorithm, the light pattern is extracted from images acquired with a CCD camera. A calibration is required for translating these distortions into physical height information. This calibration is derived by means of calibration objects of known dimensions placed under the same illumination optics. Since these calibration objects also induce a distortion of the light pattern, their known heights can be related to induced shifts. Due to space limitations under the microscope objective and due to the fragility of micromeshes used, the light patterns are limited to length scales on the order of tens of micrometers. If higher resolutions are required, scanning the mesh by means of a microstage has been shown to enhance the topographical spatial resolution. ACKNOWLEDGMENT We appreciate helpful discussions with Tom Tague (Bruker Optics, Billerica, MA). Tim Free (University of Tennessee) is acknowledged for preparing the “calibration objects” shown in Figure 6. Received for review March 13, 2007. Accepted April 20, 2007. AC070518M