Correlative Microscopy Combining Secondary Ion Mass Spectrometry

Subscriber access provided by University of Sussex Library

Article

Correlative Microscopy combining Secondary Ion Mass Spectrometry and Electron Microscopy: Comparison of IntensityHue-Saturation and Laplace Pyramid Methods for Image Fusion Florian Vollnhals, Jean-Nicolas Audinot, Tom Wirtz, Muriel Mercier-Bonin, Isabelle Fourquaux, Birgit Schroeppel, Udo Kraushaar, Varda Lev-Ram, Mark H. Ellisman, and Santhana Eswara Anal. Chem., Just Accepted Manuscript • DOI: 10.1021/acs.analchem.7b01256 • Publication Date (Web): 13 Sep 2017 Downloaded from http://pubs.acs.org on September 20, 2017

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a free service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are accessible to all readers and citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

Analytical Chemistry is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Analytical Chemistry

Florian Vollnhals†*, Jean-Nicolas Audinot†, Tom Wirtz†, Muriel Mercier-Bonin‡, Isabelle Fourquaux§, Birgit Schroeppel, Udo Kraushaar, Varda Lev-Ram║, Mark H. Ellisman∇ ,O, Santhana Eswara†* †

Advanced Instrumentation for Ion Nano-Analytics (AINA), MRT Department, Institute of Science and Technology (LIST), 4422 Belvaux, Luxembourg ‡ Toxalim (Research Centre in Food Toxicology), Université de Toulouse, INRA, ENVT, INP-Purpan, UPS, 31027 Toulouse, France § Centre de Microscopie Électronique Appliquée à la Biologie, Faculté de Médecine de Rangueil, 31062 Toulouse, France 

NMI Natural and Medical Sciences Institute at the University of Tübingen, 72770 Reutlingen, Germany Department of Pharmacology, University of California San Diego, La Jolla California, USA 92093 ∇ Departments of Neurosciences and Bioengineering, University of California San Diego, La Jolla California, USA 92093 O National Center for Microscopy and Imaging Research, University of California San Diego, La Jolla California, USA 92093 ║

ABSTRACT: Correlative microscopy combining various imaging modalities offers powerful insights to obtain a comprehensive understanding of physical, chemical and biological phenomena. In this article, we investigate two approaches for image-fusion in the context of combining the inherently lower-resolution chemical images obtained using Secondary Ion Mass Spectrometry (SIMS) with the high-resolution ultrastructural images obtained using Electron Microscopy (EM). We evaluate the image fusion methods with three different case-studies selected to broadly represent the typical samples in life science research: (i) histology (unlabeled tissue), (ii) nanotoxicology, and (iii) metabolism (isotopically labelled tissue). We show that the Intensity-HueSaturation Fusion method often applied for EM-sharpening can result in serious image artifacts, especially in cases where different contrast mechanisms interplay. Here, we introduce and demonstrate Laplace Pyramid Fusion as a powerful and more robust alternative method for image fusion. Both physical and technical aspects of correlative image overlay and image-fusion specific to SIMS-based correlative microscopy are discussed in detail alongside the advantages, limitations and the potential artifacts. Quantitative metrics to evaluate the results of image fusion are also discussed.

Secondary Ion Mass Spectrometry (SIMS)10-15 is an attractive analytical technique due to key advantages such as (i) high sensitivity, (ii) ability to distinguish all elements and isotopes and (iii) high dynamic range. SIMS is employed across all domains of science7,16-18 and complements common analytical techniques available in electron microscopes such as Energy-Dispersive X-ray Spectroscopy (EDX) and the Electron Energy-Loss Spectroscopy (EELS). The lateral resolution of SIMS images is limited by technological aspects of the ion sources (mainly their brightness) and ion-optical system (mainly induced aberrations) as well as fundamental aspects like the size of the ion-sample interaction volume. The lateral resolution is around 50 nm in the state-of-the-art imaging SIMS instruments,19 while new exciting developments involving high-resolution SIMS chemical imaging using Helium Ion Microscopy (HIM) are approaching the physical limit around 10 nm.7,20 In comparison, Scanning and Transmission Electron Microscopy (SEM/TEM) routinely

Complementary multi-technique characterization methods are indispensable for the development of a comprehensive understanding of diverse phenomena in physical and life sciences. Correlative microscopy (CM) is being increasingly recognized as a powerful approach in both materials and life sciences for synergizing the strengths and overcoming the inherent limitations of individual techniques.1 For example, the Correlative Light-Electron Microscopy (CLEM)2,3 method combines live-cell imaging using low-resolution light microscopy to track fluorescent proteins and subsequent high resolution (HR) electron microscopy to establish structurefunction correlations. Several other correlative studies combining a multitude of techniques have been reported in many disciplines of the life sciences.4,5 Similarly, in materials sciences, multiple techniques are applied to correlate microstructure, chemistry, topography, crystallography and other physicochemical properties.6-9 1

ACS Paragon Plus Environment


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

allows far superior lateral resolution of few nanometers down to the atomic scale in TEM. However, spectromicroscopic analysis of isotopes or trace elements is not possible with EDX or EELS. Hence, correlation of SIMS imaging and Electron Microscopy (EM) emerges as a complementary and powerful paradigm in which relatively low-resolution chemical images from SIMS are correlated with high-resolution structural images from EM from the exact same regions of interest (ROI). Indeed, in-situ TEM-SIMS combination was recently demonstrated to enable correlative characterization in a single instrument.21 A natural extension of the correlative EM-SIMS workflow is the application of image fusion in order to improve the visualization of the available data, and thus facilitate human or automated interpretation. In the simplest case, high-resolution electron micrographs (HR-EM) and low-resolution spectromicroscopic (LR-SM) images are overlaid in image processing software, but more sophisticated methods are being proposed by various groups for the fusion of SIMS data with electron, ion, or light microscopy images. 1,22-27 Most of the proposed methods for image fusion have been developed in the context of remote sensing and satellite image processing. Examples for image fusion algorithms are Intensity-HueSaturation (IHS), Principle Component Analysis (PCA), Laplace Pyramid techniques, like Generalized Laplacian (GLP), Discrete Cosine/Wavelet Transform (DCT/DWT) or High-Pass filtering / modulation, (HPF/HPM). An overview of image fusion mainly in remote sensing can be found in recent review articles.28-31 While their value for remote sensing applications has been evaluated in great detail, they may not be directly applicable to the problems posed by multimodal image fusion in correlative microscopy. In the case of correlative EM-SIMS, the image fusion methodologies need to be adapted to reflect the physical and technological limitations of SIMS and EM. To this end, recent studies have investigated new domains of application for the well-known and relatively straight-forward image fusion procedure known as ‘pan-sharpening’ using the IHS transformation.23,32-34 In order to distinguish the application in remote sensing and correlative microscopy, sharpening of SIMS or other spectral images via electron micrographs will be referred to as EM-sharpening henceforth. Until now, the effect of different contrast mechanisms manifested in the source images and its impact in the image fusion have received limited attention. Specifically, in Electron Microscopy, different imaging modes, such as Bright-Field, Dark-Field, High-Angle Annular Dark-Field, Secondary Electron and Backscattered Electron need to be considered. Several contrast mechanisms contribute simultaneously in many cases, e.g. those due to topography, thickness variations, diffraction, atomic-number and phase contrast. All of these have an influence on the fusion results and may lead to artifacts. In addition, the lateral resolutions observed in SIMS and electron microscopy can differ by orders of magnitude, which is usually not the case in remote sensing. Hence, there is a clear need to explore the image fusion methodologies in an effort to identify the best practice and to avoid potentially misleading artifacts. Therefore, the goal of the present article is to evaluate the methodological aspects of image fusion specific for EM-SIMS correlative microscopy. We focus exclusively on applications in life

Page 2 of 10

sciences as correlative microscopy is being increasingly employed in this field,2,3 but the analysis presented here is applicable more generally across all fields of science. Three different case-studies were selected to represent a broad variety of samples commonly used in life sciences research: (i) histology, (ii) nanotoxicology and (iii) metabolism. SIMS images were subjected to EM-sharpening using two different image fusion workflows, namely Intensity-Hue-Saturation (IHS) fusion23 and Laplace pyramid fusion (LPF) (this contribution, based on 35). In addition, the concept of hybrid fusion will be discussed. The results are analyzed and compared, followed by a discussion on the (i) relevance and applicability of the EM-sharpening, (ii) sources of potential artifacts and (iii) suggested strategies to overcome the potential pitfalls associated with these methods for image fusion.

Sample Preparation and Analysis. The details of the sample preparation and analysis can be found in the supporting information (SI-1 and SI-2). Data Processing. ImageJ/Fiji36 was used for image and spectral data processing. SIMS data was processed via the OpenMIMS plugin. SIMS and EM micrographs were manually aligned in TrakEM237 using an affine transformation. IHS and LPF fusion were implemented using ImageJ’s Python scripting system. Unless explicitly stated, the HSB color space was used for IHS-based sharpening, as better results could be achieved in EM sharpening contexts. The IHSColorTransforms plugin was used for tests of the IHS color space (ij.ms3d.de/ihs_transforms.php). Laplace Pyramid fusion used ImageJ’s Gaussian blur filter with a radius of 2.0 (standard deviation) for low-pass filtering and three or four level pyramids. LPF calculations were performed on 32 bit floating point data. Additional details of the image processing can be found in the supporting information (SI-3).

As correlative microscopy requires the acquisition of multiple images using a number of tools, the workflow has to be optimized. As a general rule, high resolution nondestructive techniques like SEM/TEM should be used first to guarantee the integrity of finer structural details before using destructive techniques like SIMS. In addition, the fusion outcome depends on the quality of the input data. The intrinsic image properties (contrast, sharpness, etc.) of the individual imaging techniques as well as any kind of artifacts (due to particular sample type, sample preparation, etc.) will be naturally carried over to the fused image. Therefore, the individual images should fulfill the accepted standards of data acquisition in the respective technique and be validated before fusion. Intensity-Hue-Saturation Fusion (IHS). In IHS based EM-sharpening, the preprocessed SIMS image (24-bit RGB) containing the spectral information is transformed into a color model referred to as IHS, Intensity-hue-saturation. The hue component (and to a lesser extent the saturation) represent the color information of the initial image, while the achromatic 2


Page 3 of 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


satellite image enhancement,23 but we found limited effect on the fusion outcome for the presented case studies (Mean Average Error, MAE, less than 3; cf. values in Table 1). The examples presented in this contribution were processed using the closely related HSB (Hue-Saturation-Brightness) color model rather than the actual IHS, as it fulfills the same purpose but offers some advantages (c.f. SI-3). We will refer to the technique as IHS fusion nonetheless, as this is an established concept in literature. An illustration of IHS-based EM-sharpening can be found in the supporting information (SI-3).

intensity component carries information of the image contrast. In standard IHS sharpening, the intensity channel of the low resolution SIMS image is simply replaced by the highresolution EM image. Finally, the IHS representation is transformed back to RGB. The fused image then exhibits characteristics of both input data sets23,34: SIMS information is present in the color code, while the high resolution contribution allows for easier identification of the respective structural features of the sample. It was proposed that the fusion result can be improved by an adjusted brightness calculated from original SIMS brightness and EM image using algorithms developed for

Figure 1: Illustration of the Laplace Pyramid Fusion (LPF) concept. Left: Construction of Laplace pyramid (L 0-L2) from input highresolution image (SEM = S0) via Gauss-filtering, subtraction and subsampling. Color code in the Laplace pyramid indicates positive and negative values in blue and red respectively. Top right: SIMS image to be sharpened is injected into the algorithm (B). Note that in this case, the input image is the brightness channel of an HSB transformed RGB image (IHS – LPF hybrid fusion). Bottom right: Repeated steps of summation of Laplace and input image followed by up-sampling generate the sharpened version of the image (B*). HSB* to RGB transformation yields the final result LPF (EM+SIMS).

resolution SIMS image (LR). From the pyramid (L1, L2, …, Ln, Gn), Gn is replaced with the low-resolution SIMS image and the set (L1, L2, …, Ln, LR) is used to synthesize the sharpened version LR* using the reverse decomposition algorithm (see below). Figure 1 illustrates the process. After pyramid decomposition of the SEM image (Figure 1, left), the SIMS image to be sharpened can be directly introduced as LR. In this example, however, the input image is the brightness component of the HSB representation of the original SIMS image (see next section: IHS hybrid fusion). The corresponding Laplace image (L3) is added to LR to give LR’. LR’ is up-sampled by first inserting rows and columns of zeroes between existing rows and columns followed by a Gaussian blur. This blurred version is equivalent to the Gaussian image of the next higher level (in this case G2, not shown).35 L2 is added to this up-sampled intermediate to give LR’’. Summation and up-sampling are continued until the last Laplacian image L1 is added to the corresponding up-sampled intermediate to give LR*. In this example, the sharpened image LR* is then used as B* to synthesize the LPF-sharpened version of the input SIMS image (LPF (EM+SIMS)). The blur radius had little influence in the presented cases (MAE below 2 for radii between 1 and 4 px) and set to 2px as a standard value.

Laplace Pyramid Fusion (LPF). In fusion techniques like IHS, the pixel values of the fused image are a function of the pixel values of the input images at the same image coordinates, i.e., ifused(x,y) = f(iEM(x,y), ispectral(x,y)) with x and y being the x- and y-coordinates of the pixels. This limits the fused information to the individual pixel level, ignoring the local and global surroundings. Laplace pyramid fusion (LPF) operates on the level of spatial frequencies and aims to include the local and global information present in the high-resolution image. This is achieved by extraction of high-frequency information from the high-resolution EM image, which is added to the low-resolution SIMS image. The Laplace Pyramid Fusion technique described here is based on the Laplace pyramid concept proposed by Burt and Adelson. 35 While more refined models of LPF exist for applications in remote sensing28,29,38 there is not yet a clear understanding of their applicability in the context of EM-sharpening. Hence, for the current study, only the basic LPF was evaluated for EMsharpening. A detailed description can be found in the supporting information (SI-3). In order to use this algorithm for image fusion in the context of pan- or EM-sharpening, the high-resolution image (HR = S1) is decomposed into the Laplace pyramid until the residual Gaussian image Gn is the same pixel array size as the low3



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Hybrid Fusion. Some instances require that a full color RGB image is sharpened rather than raw data, e.g., in the case of the HSI-representation of the 15N enrichment map presented in Figure 4. In such a case, it is possible to perform an HSB transformation of the input image, followed by LPF sharpening of the brightness channel by the corresponding high resolution image, and a subsequent back-transformation from HSB to RGB. In this IHS-LPF hybrid approach, the brightness component representation is sharpened by the highresolution micrograph rather than replaced by it, which limits the color distortions caused by a possible contrast mismatch (Figure 1). Note that the hybrid fusion approach can be extended to other substitution-based fusion techniques, e.g. those based on principle component analysis (PCA). Instead of replacing the first principle component (PC1) by the high resolution micrograph and performing inverse PCA, PC1 can be EMsharpened by the HR micrograph via another algorithm, e.g., LPF, to the sharpened version PC1*, which can then be used in the inverse PCA. Fusion Quality Assessment. After fusion, it is important to evaluate the quality of the resulting images with respect to the aim of fusion. In this case, facilitating human interpretation is considered to be the most relevant aspect. The fusion result can be evaluated by visual comparison as well as through the use of computed fusion metrics. The former is important to address subjective aspects, while the later serves as an objective measure (of a certain aspect). While there is ample literature on image fusion metrics, especially in the context of remote sensing and satellite imaging, to our knowledge, none of these deal specifically with fusion of data sets comprised of electron and secondary ion mass spectromicroscopic images. In addition, it has been noted that even for well-defined problems different fusion metrics are often not in agreement.39 Therefore, we will focus on the visual inspection of the results and use generalized metrics for numerical image quality comparison. In order to compare the results from different image-fusion methods quantitatively, we computed the image-fusion quality metrics, namely Mean Absolute Error (MAE, lower value indicates better fusion), Mutual Information (MI, higher value indicates better fusion) and ERGAS (Erreur Relative Globale Adimensionelle de Synthèse, “relative adimensional global error in synthesis”)40 (lower value indicates better fusion). The details related to the computation of these metrics are given in the Supplements (SI-3). As we focus on enhancing the SIMS data, it is important to assure that the spectral characteristics of the fused images are similar to the initial SIMS data (indicated by lower MAE, ERGAS and higher MI). During this investigation, we found that the assessment provided by the metrics confirms the visual quality assessment when comparing different fusion algorithms for a given data set. In addition to the relative comparison of fusion metrics, we assessed the performance of the fusion algorithms with respect to a reference SIMS image. In the absence of higher resolution data for the SIMS experiments due to the resolution limit of the NanoSIMS instrument, we follow a strategy proposed by Wald et al., which uses the best available spectral data as a reference for the fusion of artificially degraded data. The approach is described in the supplemental (SI-5). The results suggest that (a) LPF is superior to IHS for use in the fusion of

Page 4 of 10

SIMS and EM-data, (b) neither technique yields artifact-free fused images, and (c) image fusion is beneficial for the interpretation of correlative data.

Case Study 1: Histology. A rat heart tissue sample was investigated using correlative TEM-SIMS analysis as an example for applications of EM sharpening in histology. A region of interest containing a cell nucleus was selected and imaged by TEM in high resolution and subsequently analyzed with SIMS. Figure 2 shows the bright-field TEM image (a), inverted TEM (b) and NanoSIMS 12C14N- chemical map (c/d) of the same ROI.

Figure 2. Comparison between IHS and LPF based EMsharpening. Original and inverted TEM (iTEM) image of a tissue sample (a, b); corresponding CN- SIMS data with gray (c) and bicolor LUT (d); IHS (e, f) and LPF (g, h) fusion results of SIMS and TEM/iTEM. Dark areas in the original TEM image (a) are dominant and suppress the intense SIMS signal at the same position in the IHS fused image (e). In the contrast-inverted TEM case (b), the now bright areas are dominant after fusion (f); other areas of equally intense SIMS signal (bottom right) are suppressed. LPF results (g, h) are less affected by the contrast

4


Page 5 of 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


mismatch (compare g/h and d). Insets show that small features in the TEM/iTEM are readily discernable in fused images. Pixel size TEM: 4.4 nm; SIMS: 39 nm.

fused image as well. It is clearly apparent from Figure 3c, g) that the pink color attributed to Ti-containing nanoparticles in the SIMS micrograph (b) is suppressed due to the lack of brightness in the TEM image. In addition, the contrast mismatch leads to a general loss of color fidelity. On the other hand, the LPF sharpened version (d, h) exhibit better color fidelity and are able to correctly represent the TiO2 nanoparticles.

The TEM image shows very detailed sub-cellular ultrastructure including nucleus and mitochondria. Parts of the cell nucleus are very dark due to uranyl acetate staining and hence clearly recognizable. As uranyl acetate binds amongst others highly to nucleic acid phosphate groups of DNA, uranyl acetate staining results in high contrast for chromatin which contains DNA within cell nuclei. However, the lateral resolution of the SIMS images is not sufficient to differentiate the various sub-cellular structures present in the sample. Therefore, IHS and LPF were used to sharpen the data set; the results are juxtaposed in Figure 2e-h.The contrast in the IHSfused image is dominated by the contrast of the TEM/inverted TEM (iTEM) (Figure 2e, f). Bright features in the EM produce bright features in the fused image; the reverse is true for dark features. In addition, there is a noticeable color shift between the IHS-fused images (e, f) and the input SIMS data (d). Both of these effects make evaluation of the IHS-fused images prone to misinterpretation in the absence of the raw SIMS data. In LPF, the sharpening is achieved by stepwise addition of high frequency contributions to the initial SIMS micrograph, and thus the initial intensity distribution of low resolution chemical image is better preserved. This leads to a reduced, yet not completely suppressed, influence of the contrast differences between the SIMS and EM micrographs on the fusion result (g, h) and also preserves the color balance of the SIMS raw data (a, b) to a higher degree. Yet, LPF still adds high resolution structural information, which facilitates the interpretation of the data. An additional example of the CN-and P--distribution using LPF sharpening for this case study can be found in the supporting information (SI-4). Case Study 2: Nanotoxicology. This case study is representative of the class of samples typically investigated in gut nanotoxicology through the new “microbiota” target. Toxicity associated with nanoparticles is a subject of intense scientific research.41 In particular, the effects on the bacteria of the gut microbiota have been recently identified as a challenging question.42-45 In the present study, Escherichia coli was chosen as representative for intestinal bacteria. 42 The analytical challenge associated with this category of samples is due to the coexistence of both organic and inorganic material which contributes to a large variety of contrast mechanisms. Specifically, the TEM imaging of organic material typically requires strong defocus of the objective lens to obtain sufficient image contrast.However, the inorganic nanoparticles are usually crystalline and require only weak defocus for best image contrast. Hence, in samples where both organic and inorganic material is present, the ideal focus condition is compromised. Furthermore, several contrast mechanisms are involved such as staining contrast, diffraction contrast and absorption contrast. The BF-TEM image of the sample and the corresponding NanoSIMS images are shown in Figure 3. Note that the TiO2 nanoparticles appear dark in the BF-TEM image. As the intensity of the high-resolution image is incorporated into the fused image in a simple HSI image fusion, these dark particles would erroneously result with less intensity on the

Figure 3. Bacteria (E. coli) exposed to TiO2 nanoparticles; embedded in resin. a) BF-TEM micrograph; b) NanoSIMS analysis: 12C14N- (green) and 48Ti16O- (red); (c, d) Sharpened versions using IHS and Hybrid LPF; (e-h) magnified versions of (a-d). Scale bar in inset e) represents 500 nm. Pixel size TEM: 6.3 nm; SIMS: 39 nm.

Case Study 3: Metabolism. This case study represents the broad category of research using isotopic labeling such as in the investigation of metabolism and protein turnover. An ROI containing a node of Ranvier was selected to investigate by correlative EM-SIMS. The myelinated axon near the node of Ranvier is also visible in Figure 4. In order to visualize the areas rich in 15N, the SIMS ratio image of 12C15N-/12C14N- was obtained and shown in (b) using the HSI color scheme option in OpenMIMS. The SIMS ratio image enables to derive quantitative insights about the isotopic enrichment and depletion. The lower limit of the color scale in (b) is set to the natural abundance of 15N (0.37%). The maximum value in the ratio image corresponds to an abundance of 9.25%, which indicates that the 15N content in the hotspots is 25 times above the natural abundance. The SEM image contains very fine structural details which are not captured in the relatively lower resolution SIMS image. A detailed evaluation of the biological significance of this result is beyond the scope of the current study and therefore, the discussion will be limited to the methodological aspects of correlative EM-SIMS image fusion for this type of input data set. As the previous case studies have demonstrated the beneficial aspects of LPF over IHS image fusion, we will use only the LPF-sharpened version of this data set to discuss the details of this process in Figure 4, while the comparison of different fusion approaches will be discussed in Figure 6. Figure 4c shows the end result of LPF image fusion combining the original SEM and SIMS input images (a, b). A specific feature is magnified and shown in the 5



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

inset (e) alongside the corresponding area from the original SEM (d) and SIMS (f) images. There is little noticeable color shift between the input and sharpened image (b and c), confirming the preservation of color fidelity in LPF fusion. The high resolution SEM image (d), the pixelated, low resolution SIMS image (f) and the fused version (e) demonstrate the benefits of image fusion in the localization and identification of features of interest. The chemical information is preserved via the hue component, while the structural information is present via the image contrast. Note that in this case the pixel sizes of SEM (3.5 nm) and SIMS (78 nm) differ by a factor of greater than 22, while common factors in remote sensing are two or four. Nevertheless, the fused image allows for easier visual identification and correlation of microscopic features and their chemical nature.

Page 6 of 10

Evaluation: Choice of Color Code. In contrast to satellite imaging, the color information in EM-sharpening in correlative microscopy is entirely user-defined and thus can be arbitrarily chosen. This is especially critical in IHS fusion, which can only fuse RGB representations of raw data rather than the raw data itself. SIMS images are often displayed in a single color channel, e.g. from black to green with black, (0, 0, 0)RGB, representing zero SIMS intensity and bright green, (0, 255, 0)RGB, representing the maximum SIMS intensity; all possible pixel colors are (0, i, 0)RGB with i being the SIMS intensity mapped to the 8-bit range [0…255]. While such images are certainly useful for presenting chemical maps, their application in IHS-based image fusion will result in serious image artifacts. As this issue is not adequately addressed in the literature, we illustrate this point with Figure 5. The SIMS image (a) is colored by application of a green look-up table (LUT) and converted to HSB. The pixel values are converted from (0, i xy, 0)RGB to (85, 255, ixy)HSB. All pixels exhibit the same hue (h = 85) and saturation (s = 255), and vary only in their brightness component i. After replacing the brightness channel by a high resolution image with pixel values of mxy in the actual IHS fusion step the pixel values will be (85, 255, mxy)HSB (Figure 5 b, c). Converting back to RGB yields pixel values of (0, m xy, 0)RGB, which is identical to the high resolution image displayed with a green LUT (Figure 5d, e). Thus, all SIMS information is lost in the replacement step. This kind of artifact will occur for any color scale that is represented as a single hue (or a very limited range of hues) in its HSB representation and is intrinsic to IHS-based fusion due to the use of an RGB representation of the data. Fusion algorithms operating directly on numerical data (i.e., intensity matrices), like standard LPF, are not affected. This type of artifact is an intrinsic problem of IHS fusion of data that is represented in arbitrary color. If IHS is to be used despite this issue, this type of artifact can be avoided by the use of a hue range rather than a brightness range. The ideal case is a wide hue spread, e.g. a “rainbow” type color scale (c.f. Figure 4), but this is limited to cases where only a single spectral channel is to be sharpened. If multiple channels are to be presented in a single image, a reasonable approach is the use of a non-black color, e.g., dark blue, (0, 0, 127)RGB for the representation of zero intensity, in combination with a color of high visibility like red, green, yellow, pink, orange, etc. for high intensities (c.f. Figure 3). Evaluation: Contrast Mismatch. In IHS based image fusion, the contrast of the pan/EM channel is directly transferred to the final image due to the way the HSB/RGB conversion is handled. An area that appears dark in in the EM will produce a corresponding dark area in the sharpened image. The lack of brightness affects the hue component and may cause it to become almost invisible in the fused image. Vice versa, a high brightness may yield a brightly colored feature in the sharpened image, which can over-emphasize the SIMS intensity in that region or cause color shifts. These aspects become especially apparent when the contrasts of the input EM and SIMS micrographs differ greatly, i.e., if the different physical contrast mechanisms of the techniques generate signals of high intensity in one modality and low intensity in the other. A carbon-rich particle might appear dark

Figure 4. Hybrid LPF based EM-sharpening of a HSI map of the 15 N/14N enrichment in tissue. The color scale (Hue) indicates values between natural 15N abundance (blue, 0.37%) and 15N enrichment (magenta, 9.25%). (a) SEM image of the sample; (b) 15 N/14N ratio NanoSIMS dataset displayed as HSI map; (c) Sharpened version combining (a) and (b) using Laplace Pyramid Fusion; (d-f) magnified areas of interest, corresponding to SEM (d), LPF sharpened HSI map (e), and raw (f) HSI map (Field of view: 1 μm). Pixel size SEM: 3.5 nm; SIMS: 78 nm.

6


Page 7 of 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


15

in scanning electron microscopy due to lower secondary electron yield, while it may be a prominent feature in a 12CSIMS micrograph. Figure 6 juxtaposes IHS and IHS-based hybrid fusion based EM sharpening schemes applied to the

N-enrichment case study (c.f. Figure 4) for both SEM and inverted SEM images (the inverted SEM serves as an approximation of contrast observed in BF-TEM).

Figure 5. Illustration of the pitfall associated with single-hue LUTs in EM-sharpening. For EM-sharpening, the initial SIMS intensity of 12 14 C N (a, top) is converted to a false color image via a LUT, in this case black-to-green (a, bottom). The color image is then converted to the HSB color model (b, H component in RGB). Pixels in the hue channel are either green, or of no color (black). SIMS information is contained in the brightness channel (B). When B is replaced by the high-resolution EM (HR = B*), the SIMS information is essentially discarded (c). Back-transformation to RGB yields an invalid “sharpened” image (d). This image is essentially identical to the EM-image (e) displayed with the same LUT as (a).

IHS fusion without pre-processing (d, g) results in color distortions, especially for the case of the inverted SEM image (bottom row). Note also the absence of the dark regions in the center of the SIMS micrographs (red arrow). The Munechika algorithm,22,46-48 which has been shown to be able to improve the spatial resolution in SIMS EM-sharpening, yields loss of overall brightness compared to the input image, which can be counteracted by histogram matching at the cost of a reduced contrast bandwidth (f, i). In contrast, LPF conserves the visual appearance of the RGB data: color distortions are reduced while a high contrast bandwidth is preserved (e, h) and fine structure remains visible (yellow circles).

In order to address this type of artifact, the contrast range of the electron micrograph can be adjusted to the spectral data, e.g., by histogram matching, band-pass filtering or manual adjustment of brightness and contrast. While these preprocessing steps are valid in principle, they may lead to further artifacts in the fusion result. Evaluation: Fusion Metrics. The quality metrics corresponding to the case studies are listed in Table 1; the best scores are highlighted per case study. For comparison, the respective scores for Munechika-type fusion using preprocessed EM data are included as well.

7



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 8 of 10

Figure 6. Comparison of different processing approaches applied to the 15N enrichment analysis. Top row: input data; bottom rows: image fusion output data using the SEM (left) or inverted SEM (iSEM, right). Fusion algorithms are indicated in the image. See text for discussion.

The comparison shows that Laplace pyramid fusion consistently yields fused images with better relative scores compared to IHS and Munechika. Only if additional preprocessing is applied, Munechika yields comparable or better scores. Fusion Algorithm IHS Case Study 1: Histology Case Study 2: Nanotoxicology

Case Study 3: N enrichment in tissue

15

EM

applications, e.g. IHS or PCA-based algorithms, in hybrid fusion. In general, LPF was found to be the more robust and reliable method for image fusion than the IHS method. As a final remark, it is strongly advised that the input images of the image fusion process always be presented in addition to the final EM-sharpened image to avoid incorrect interpretation.

Fused vs. SIMS ERGAS MI 5.61 0.20

TEM

MAE 38.3

LPF

TEM

18.5

2.42

0.27

IHS

iTEM

29.2

4.03

0.17

LPF

iTEM

18.3

2.41

0.33

IHS

TEM

63.5

33.28

0.53

LPF

TEM

17.6

8.04

1.07

IHS

SEM

18.0

2.73

0.29

LPF

SEM

16.7

2.43

0.42

Munechika

SEM

28.0

5.60

0.62

IHS

iSEM

29.3

5.29

0.09

LPF

iSEM

17.6

2.58

0.34

Munechika

iSEM

22.9

7.39

0.29

Munechika + Preprocessing

SEM

8.6

1.3

0.75

iSEM

17.0

3.0

0.31

The Supporting Information is available free of charge on the ACS Publications website. A detailed description of the sample preparation (SI-1) and analysis (SI-2), image processing (SI-3) as well as additional data for the histology case study (SI-4) can be found in the supporting PDF document.

Table 1: Quality metrics of the fusion outcomes. Best scores per data set are highlighted (excluding preprocessed Munechika fusion).

* E-mail: [email protected] (F. V.)

This work was funded by the National Research Fund Luxembourg (FNR) within the framework of the HELION project through grant C14/MS/8345352 and the C4HEALTH project through grant INTER/MERA/14/9822270. We thank Tom Deerinck and Eric Bushong of NCMIR for preparing samples from the isotopic fed mouse brain samples and carrying out the high resolution SEM imaging. NCMIR is supported by the US NIH P41GM103412 (to MHE). The 15N labeled mice were produced as part of work on an award from the US NIH to study long-lived molecules and memory mechanisms in the brain, NIH RO1 NS027177I (to Roger Y. Tsien, MHE and VL). The authors also thank Marie-Pierre Duviau (LISBP Toulouse, FranceI) and Christel Cartier (Toxalim Toulouse, France) for technical help. Ria Knittel and Gabi Frommer-Kästle (University Hospital Tübingen, Germany) are thanked for their skillful technical assistance. This work was made possible in part by the OpenMIMS software whose development is funded by the NIH/NIBIB National Resource for Imaging Mass Spectrometry, NIH/NIBIB 5P41 EB001974 (to Claude Lechene).

The rise of multimodal correlative microscopy in material science and biology has sparked interest in suitable methods for data analysis, correlation and visualization of data. Specifically, methods to reliably combine high resolution structural image with lower resolution chemical image are often needed. One of the promising approaches to achieve this is EM-sharpening. The current study on this subject leads to the following conclusions: (1) Pan-sharpening algorithms used in remote sensing cannot be directly applied in EM-sharpening. The differences in input data requires careful evaluation, testing and most likely adaptation before use in EM based correlative microscopy. This includes in particular the evaluation of quality metrics used in the analysis of the fusion performance. (2) The Intensity-Hue-Saturation (IHS) approach suffers from a number of limitations, with the three most severe being: (i) operation on an RGB representation of the data rather than the raw data itself, (ii) limited capability of dealing with contrast mismatch in input images and (iii) risk of misrepresentation due to the selection of an invalid color scheme. The beneficial aspects of the IHS approach are its simplicity and ease of implementation in image processing software like ImageJ/Fiji, and the fact that it can serve as basis for hybrid fusion approaches (e.g. IHS-LPF). (3) The Laplace pyramid fusion introduced in this publication overcomes or mitigates these limitations, especially the pronounced effects of contrast mismatch and risk of choosing a misleading color scheme. Furthermore, LPF can be used in combination with other image fusion

REFERENCES (1) Jiang, H.; Kilburn, M. R.; Decelle, J.; Musat, N. Curr. Opin. Biotechnol. 2016, 41, 130-135. (2) Lucas, M. S.; Günthert, M.; Gasser, P.; Lucas, F.; Wepf, R. In Methods in Cell Biology, Correlative Light and Electron Microscopy; Thomas Müller-Reichert, Paul Verkade, Eds.; Academic Press: 2012; pp 325-356. (3) de Boer, P.; Hoogenboom, J. P.; Giepmans, B. N. G. Nat. Methods 2015, 12, 503-513.

8


Page 9 of 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


(4) Biesemeier, A.; Eibl, O.; Eswara, S.; Audinot, J. N.; Wirtz, T.; Pezzoli, G.; Zucca, F. A.; Zecca, L.; Schraermeyer, U. J. Neurochem. 2016, 138, 339-353. (5) Jiang, H.; Favaro, E.; Goulbourne, C. N.; Rakowska, P. D.; Hughes, G. M.; Ryadnov, M. G.; Fong, L. G.; Young, S. G.; Ferguson, D. J. P.; Harris, A. L.; Grovenor, C. R. M. Methods 2014, 68, 317-324. (6) Kumar, P.; Pfeffer, M.; Willsch, B.; Eibl, O.; Yedra, L.; Eswara, S.; Audinot, J. N.; Wirtz, T. Sol. Energy Mater. 2017, 160, 398-409. (7) Wirtz, T.; Philipp, P.; Audinot, J. N.; Dowsett, D.; Eswara, S. Nanotechnology 2015, 26, 434001. (8) Ovchinnikova, O. S.; Tai, T.; Bocharova, V.; Okatan, M. B.; Belianinov, A.; Kertesz, V.; Jesse, S.; Van Berkel, G. J. ACS Nano 2015, 9, 4260-4269. (9) Eswara, S.; Hu, R.; Yedra, L.; Audinot, J. N.; Schwedt, A.; Tasan, C.; Mayer, J.; Raabe, D.; Wirtz, T. In European Microscopy Congress 2016: Proceedings; Wiley-VCH Verlag GmbH & Co. KGaA: 2016; (10) Williams, P. Ann. Rev. Mater. Sci. 1985, 15, 517-548. (11) Benninghoven, A.; Werner, H. W.; Rüdenauer, F. G. Secondary ion mass spectrometry: Basic concepts, instrumental aspects, applications and trends; Wiley-Interscience: New York, Chichester, Brisbane, Toronto, Singapore, 1987. (12) Zalm, P. C. Vacuum 1994, 45, 753-772. (13) Rüdenauer, F. G. Anal. Chim. Acta 1994, 297, 197-230. (14) McDonnell, L. A.; Heeren, R. M. A. Mass Spectrom. Rev. 2007, 26, 606-643. (15) McPhail, D.; Dowsett, M. In Surface Analysis - The Principal Techniques; Vickerman, J. C., Gilmore, I. S., Eds.; John Wiley & Sons, Ltd: 2009; pp 207-268. (16) Steinhauser, M. L.; Bailey, A. P.; Senyo, S. E.; Guillermier, C.; Perlstein, T. S.; Gould, A. P.; Lee, R. T.; Lechene, C. P. Nature 2012, 481, 516-519. (17) Wild, E. M.; Teschler-Nicola, M.; Kutschera, W.; Steier, P.; Trinkaus, E.; Wanek, W. Nature 2005, 435, 332-335. (18) Floss, C.; Stadermann, F. J.; Bradley, J.; Dai, Z. R.; Bajt, S.; Graham, G. Science 2004, 303, 1355-1358. (19) Hoppe, P.; Cohen, S.; Meibom, A. Geostand. Geoanal. Res. 2013, 37, 111-154. (20) Gratia, P.; Grancini, G.; Audinot, J. N.; Jeanbourquin, X.; Mosconi, E.; Zimmermann, I.; Dowsett, D.; Lee, Y.; Grätzel, M.; De Angelis, F.; Sivula, K.; Wirtz, T.; Nazeeruddin, M. K. J. Am. Chem. Soc. 2016, 138, 15821-15824. (21) Yedra, L.; Eswara, S.; Dowsett, D.; Wirtz, T. Sci. Rep. 2016, 6, 28705. (22) Milillo, T.; Hard, R.; Yatzor, B.; Miller, M. E.; Gardella, J. Surf. Interface Anal. 2015, 47, 371-376. (23) Tarolli, J. G.; Jackson, L. M.; Winograd, N. J. Am. Soc. Mass Spectrom. 2014, 25, 2154-2162. (24) Tyler, B. J.; Rayal, G.; Castner, D. G. Biomaterials 2007, 28, 2412-2423. (25) Van de Plas, R.; Yang, J.; Spraggins, J.; Caprioli, R. M. Nat. Methods 2015, 12, 366-372.

(26) Stubbings, T. C.; Nikolov, S. G.; Hutter, H. Microchim. Acta 2000, 133, 273-278. (27) Sobol, O.; Holzlechner, G.; Holzweber, M.; Lohninger, H.; Boellinghaus, T.; Unger, W. E. S. Surf. Interface Anal. 2016, 48, 474478. (28) Vivone, G.; Alparone, L.; Chanussot, J.; Mura, M. D.; Garzelli, A.; Licciardi, G. A.; Restaino, R.; Wald, L. IEEE Trans. Geosci. Rem. Sens. 2015, 53, 2565-2586. (29) Aiazzi, B.; Alparone, L.; Baronti, S.; Garzelli, A. IEEE Trans. Geosci. Rem. Sens. 2002, 40, 2300-2312. (30) Amro, I.; Mateos, J.; Vega, M.; Molina, R.; Katsaggelos, A. K. EURASIP J. Adv. Signal Process. 2011, 2011, 79. (31) Pohl, C.; Van Genderen, J. L. Int. J. Rem. Sens. 1998, 19, 823854. (32) Tarolli, J. G.; Bloom, A.; Winograd, N. Biointerphases 2016, 11, 02A311. (33) Tarolli, J. G.; Naes, B. E.; Garcia, B. J.; Fischer, A. E.; Willingham, D. J. Anal. At. Spectrom. 2016, 31, 1472-1479. (34) Tarolli, J.; Tian, H.; Winograd, N. Surf. Interface Anal. 2014, 46, 217-220. (35) Burt, P.; Adelson, E. IEEE Trans. Commun. 1983, 31, 532540. (36) Schindelin, J.; rganda-Carreras, I.; Frise, E.; Kaynig, V.; Longair, M.; Pietzsch, T.; Preibisch, S.; Rueden, C.; Saalfeld, S.; Schmid, B.; Tinevez, J. Y.; White, D. J.; Hartenstein, V.; Eliceiri, K.; Tomancak, P.; Cardona, A. Nat. Methods 2012, 9, 676-682. (37) Cardona, A.; Saalfeld, S.; Schindelin, J.; Arganda-Carreras, I.; Preibisch, S.; Longair, M.; Tomancak, P.; Hartenstein, V.; Douglas, R. J. PLoS ONE 2012, 7, e38011. (38) Garzelli, A.; Nencini, F. Inf. Fusion 2005, 6, 213-224. (39) Jagalingam, P.; Hegde, A. V. Aquatic Procedia 2015, 4, 133142. (40) Wald, L. Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions; Presses de l'Ecole, Ecole des Mines de Paris, Paris, France: 2002. (41) Bettini, S.; Boutet-Robinet, E.; Cartier, C.; Coméra, C.; Gaultier, E.; Dupuy, J.; Naud, N.; Taché, S.; Grysan, P.; Reguer, S.; Thieriet, N.; Réfrégiers, M.; Thiaudière, D.; Cravedi, J. P.; Carrière, M.; Audinot, J. N.; Pierre, F. H.; Guzylack-Piriou, L.; Houdeau, E. Sci. Rep. 2017, 7, 40373. (42) Fröhlich, E. E.; Fröhlich, E. Int. J. Mol. Sci. 2016, 17, 509. (43) Karavolos, M.; Holban, A. Pharmaceuticals 2016, 9, 62. (44) Mercier-Bonin, M.; Despax, B.; Raynaud, P.; Houdeau, E.; Thomas, M. Crit. Rev. Food Sci. Nutr. 2016, 00. (45) Pietroiusti, A.; Magrini, A.; Campagnolo, L. Toxicol. Appl. Pharmacol. 2016, 299, 90-95. (46) Zhang, Y. System and method for image fusion. U.S. Patent 7,340,099, Mar 4, 2008. (47) Munechika, C. K.; Warnick, J. S.; Salvaggio, C.; Schott, J. R. Photogramm. Eng. Remote Sens. 1993, 59, 67. (48) Milillo, T. M.; Miller, M. E.; Fischione, R.; Montes, A.; Gardella, J. A. Biointerphases 2016, 11, 02A308.

9



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Page 10 of 10

Insert Table of Contents artwork here


10

Correlative Microscopy Combining Secondary Ion Mass Spectrometry

Recommend Documents