Deep Learning Approach for Enhanced Detection of Surface Plasmon

Subscriber access provided by KEAN UNIV

Article

Deep Learning Approach for Enhanced Detection of Surface Plasmon Scattering Gwiyeong Moon, Taehwang Son, Hongki Lee, and Donghyun Kim Anal. Chem., Just Accepted Manuscript • DOI: 10.1021/acs.analchem.9b00683 • Publication Date (Web): 09 Jul 2019 Downloaded from pubs.acs.org on July 20, 2019

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Analytical Chemistry

Deep Learning Approach for Enhanced Detection of Surface Plasmon Scattering

Gwiyeong Moon*, Taehwang Son*, Hongki Lee, and Donghyun Kim†

School of Electrical and Electronic Engineering Yonsei University, Seoul, Korea, 120-749

Abstract: A deep learning approach has been taken to improve detection characteristics of surface plasmon microscopy (SPM) of light scattering. Deep learning based on convolutional neural network algorithm was used to estimate the effect of scattering parameters mainly the number of scatterers. The improvement was assessed on a quantitative basis by applying the approach to SPM images formed by coherent interference of scatterers. It was found that deep learning significantly improves the accuracy over conventional detection: the enhancement in the accuracy was shown to be significantly higher by almost six times and useful for scattering by polydisperse mixtures. This suggests that deep learning can be used to find scattering objects effectively in the noisy environment. Furthermore, deep learning can be extended directly to label-free molecular detection assays and provide considerably improved detection in imaging and microscopy techniques.

Keywords: Surface plasmon microscopy; Surface plasmon scattering; Deep learning; Convolutional neural network

* †

These authors contributed equally to this work. Corresponding author: [email protected]

1

ACS Paragon Plus Environment

Analytical Chemistry 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

1. Introduction Label-free imaging and detection have provided rich information on intrinsic physical characteristics of cellular and molecular events without labeling process. One of the techniques developed to achieve label-free detection is surface plasmon microscopy (SPM) that measures intensity changes as the excitation of electron density waves or surface plasmon (SP) takes place in metal dielectric interface by momentum-matching with incident photon [1,2,3,4]. SPM has been used in many applications, e.g., highly multiplexed high-throughput measurement of biomolecular interactions [5,6,7,8], label-free imaging of viruses [9], single nanoparticle characterization and detection [10], measurements of DNA hybridization adsorption [11], analysis of surface properties [12], and measurement of cell-to-substrate separation [13,14,15]. Unlike SP-enhanced fluorescence microscopy, imaging resolution of SPM often becomes much worse than the diffraction-limit due to severe SP scattering. Therefore, a large part of SPM research has been devoted to the enhancement of spatial resolution, for example, by deconvolution of SPM images [16], control of plasmon scattering with nanostructures [17], use of incident light patterned with a digital light projector [18], and multi-channel momentum sampling with minimum filtering [19]. Most of these approaches rely on hardware-based improvement with moderate success. With recent emergence of machine learning techniques under intensive development, many disciplines of research have explored machine learning for vast and diverse possibilities that may have previously been overlooked [20,21,22]. Applications to image [23,24,25,26] and speech recognition [27] and optimization of diverse systems [28, 29, 30] have been in rapid progress with increased interests. In the field of biotechnology, use of machine learning techniques has been actively investigated while research has been carried out for specification of DNA sequences [31] and classification of cells in label-free detection [32] and for diagnosis of skin cancer [33]. Machine learning algorithms were applied to solve optical phase tomography problems [34]

2


Page 2 of 29

Page 3 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


and used for analysis of images obtained through scattering media [35,36] and aberration compensation in digital holographic microscopy [37]. Computational sensing framework was also constructed by applying machine learning to plasmonic sensors [38]. In this work, we explore a deep learning-based approach to understanding SP scattering of light, which recently has found numerous applications in molecular detection [39] and ultimately to imaging objects in noisy environment dominated by SP scattering. To this goal, we have used deep learning to estimate scattering parameters, mainly the number and the positions of scatterers in SPM, based on the classification of images produced by multiple scatterers. The performance was compared with a conventional procedure without deep learning and confirmed both theoretically and experimentally that deep learning can significantly enhance the accuracy of the data measured by conventional SPM systems and furthermore provide considerably improved detection in imaging techniques beyond SPM.

2. Numerical method and model 2.1 Image formation model by SP scattering We have employed a simple model to generate SPM images [40,41]: p-polarized light incident on a thin metal film and BK7 glass substrate becomes evanescent and scattered by a particle scatterer placed on the film to form a SPM image, as shown in Figure 1a. The scattered field Es can be expressed as a decaying spherical wave, i.e., |

|

|

|

𝐸𝑠(𝒓) = 𝜇𝑠𝐸𝑠𝑝(𝒓𝟏)𝑒 ―𝜅 𝒓 ― 𝒓𝟏 𝑒 ―𝑖𝑘𝑠𝑝𝑝 𝒓 ― 𝒓𝟏 ,

(1)

where r1 represents the scatterer location. s is a scattering coefficient dependent on the scatterer polarizability [41].  denotes absorption coefficient. Esp(r1) represents a plasmonic wave formed by the momentum-matching between incident photon and SP wave vector. SP wave number kspp is well known to be given by

3



𝑘𝑠𝑝𝑝 =

(

Page 4 of 29

1 2

).

𝜔 𝜀𝑚 𝜀𝑑 𝑐 𝜀𝑚 + 𝜀𝑑

(2)

Here,  is the angular frequency of an incident wave, c speed of light in vacuum, m and d the permittivity of metal and dielectric material. SPM image intensity I(r) can be described by the sum of reflected field at glass (Er) and Es from which a background image in the absence of a scatterer can be subtracted: 𝐼(𝒓) = |𝐸𝑟(𝒓) + 𝐸𝑠(𝒓)|2-|𝐸𝑟(𝒓)|2.

(3)

Equation (3) can be generalized for a system of N scatterers (N > 1) by considering all scattered fields and ignoring multiple scattering as follows:

|

𝑁

|

2

𝐼(𝒓) = 𝐸𝑟(𝒓) + ∑𝑖 = 1𝐸𝑠,𝑖(𝒓,𝒓𝒊) ― |𝐸𝑟(𝒓)|2 .

(4)

ES,i can be described as the convolution of Object and PSF in the linear imaging regime, i.e., 𝐼(𝒓) = |𝐸𝑟(𝒓) + 𝑂𝑏𝑗𝑒𝑐𝑡 ∗ 𝑃𝑆𝐹|2 ― |𝐸𝑟(𝒓)|2,

(5)

where Object denotes the distribution of point scatters and, ultimately, that of refractive index on the metal surface. Note also that PSF in Equation (5) is not the point-spread function of far-field optics. Instead, it represents the scattering of a point scatterer without multiple scattering. PSF thus is identical to the field formed when propagating SP is scattered by a point scatterer, which allows an image I(r) to be constructed from Equations (4) and (5). Regarding the reconstruction of the object distribution, we have followed the method detailed previously [16]. The method involves the reconstruction of Object from the acquired image using Fourier filtering as well as the deconvolution of a complex electric field.

4


Page 5 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Figure 1 (a) Schematics of the particle scattering model by propagating SP waves. (b) Diagram of a unit simulation in an area of 129 × 129 pixels (R1) with schematic in this case corresponding to N = 2 and 3. The position of N scattering particles was limited to within 31 × 31 pixels (R2). R2 was located at the center of an entire simulation area (R1). A pixel is a square of 100-nm sides. (c) Distribution of the inter-scatterer distance d defined as the distance between the nearest scatterers in the R2 region. The number of scatterers (N) and the average inter-scattering distance (davg) are also shown. Unit of davg: μm. (d) Schematic diagram of the CNN architecture. It consists of Convolution layer (Conv), Batch normalization (BN), Rectified linear unit (Relu), Max pooling layer (Pooling), Fully connected layer (FC) and output layer.

2.2 Method of SPM image formation N scatterers were placed on the gold film in water medium and plasmonic fields formed by the scatterers were produced by the illumination of p-polarized light ( = 680-nm) in a unit area of 129  129 pixels (R1). The location of scatterers was limited to a square (R2). R2 consists of 31  31 pixels

5



centered in R1. Each pixel was set to be 100  100 nm2 and modeled as 20  20 subpixels to improve the precision of locating scatterers. In total, R2 consists of 620  620 subpixels, where N scatterers were randomly distributed. A scatterer is assumed to be a polystyrene nanoparticle of 100-nm diameter (ns = 1.58 and scattering coefficient s = 0.027 cm-1) [41]. If multiple scatterers are present in R2, the distance between scatterers (𝑑) is a random variable between 0 and 3 2 µm, as shown in Figure 1b. We have considered the scatterer number N up to Nmax = 9, which is equivalent to a number concentration of 1 #/µm3 ( = N/(3  3) µm3), assuming that 2D distribution characteristics of particles remains identical in 3D and that scatterers are randomly distributed. The accuracy is affected by the range of N in the way that the accuracy corresponding to Nmax can be overestimated because N cannot be larger than Nmax and therefore the case of N = Nmax is more predictable than other cases. For this reason, we performed the calculation up to N = 10 yet took the results with Nmax = 9. With more scatterers in the region of interest for N > 9, the region can be split into sub-regions so that the current approach remains valid without the loss of generality. Figure 1c shows the distribution of inter-scatterer distances (d) when repeating the task of randomly placing N scatterers in the R2 region. Inter-scatterer distance d was defined as the distance between the nearest two out of N scatterers and used as a metric in the context that the scattering by the nearest two scatterers would cause the most significant distortion to be addressed in image characterization and classification. d corresponds to the minimum of the random variable 𝑑. N and average inter-scatterer distance (davg) are shown on each histogram. Expectedly, a larger N, i.e., more scatterers in the region R2, decreases davg. At N = 9, davg is reduced to 259 nm. A SPM image can be obtained using Equation (4). The intensity was normalized in 8-bit after an image was subtracted by baseline and Fourier-filtered. Poisson noise was then added to model the shot noise that may be present under experimental conditions [42, 43]. Signal-to-noise ratio (𝑆𝑁) as an independent image parameter was varied from 30 to 1 dB for least noisy to noisiest environment.

2.3 Deep learning and performance metrics The convolution neural network (CNN) was employed as a deep learning algorithm for

6


Page 6 of 29

Page 7 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


classification of plasmonic images with respect to the number of scatterers (N). The architecture of the CNN used in this study was modified from VGG19 [44] and is illustrated in Figure 1d. Network consists of functional components of layers: Convolution (Conv), Batch normalization (BN), Rectified linear unit (Relu), Max pooling (max pool), Fully connected (FC), and output. The BN layer was added after Conv to improve training and a dropout technique was used in the FC layer to prevent over-fitting. Stochastic gradient descent with momentum was employed as the optimization with an initial learning rate of 0.001, a mini-batch size of 32, and the number of epochs at 15. With trained CNN, about 2000 SP scattering images can be classified in one second, i.e., classification can be carried out in real time after training. Classification accuracy was used to present the performance of the classification algorithm and defined as the ratio of correct prediction to the total number of test samples, expressed in percentage and calculated with N. The total number of the test samples is 1500. The correct prediction represents the number of cases when N is predicted correctly.

7



Figure 2 (a) SPM images as the number of randomly distributed scatterers (N) changes: N = 2 ~ 9 from top left to bottom right. Each column consists of SPM images (left) without and (right) with the addition of background Poisson noise (𝑆𝑁 = 13 dB): the addition marked by an arrow at the top of each column. Unit of d: μm. The scatterer positions are marked by red dots. (b) Distribution of per-pixel intensity (Ip) in each of the images. Each circle corresponds to one image and 100 circles are represented per N. The average of Ip appears below the cluster of circles. Error bars represent the standard deviation. Intensity average at each N was connected in line without interpolation to show the trend.

2.4 Experimental assessment Experimental verification of SP scattering was conducted on an inverted microscope with a 100 high numerical aperture (NA: 1.49) oil immersion objective with  = 671 nm illuminated on the gold surface for beads of 100 nm diameter as scatterers. Reflected intensity from the gold surface was recorded at a frame rate of 58 fps. Image stacks were processed by taking differential images and averaging three consecutive frames. Signal-to-noise ratio SN of experimental images of SP scattering is estimated to be in the range of 18.5 ~ 24.1 dB when the noise power was compared to the simulation (see Supporting Information S1 for details). Note that the nature of noise in experimental conditions is different from the noise added to SP scattering images used in theoretical studies for training neural network. Therefore, images for training neural network have been generated by experimentally producing SP scattering patterns that correspond to N = 1 ~ 5 and performing image synthesis using toolbox of MATLAB. In this way, 10,000 images were generated, among which 70% (= 7,000 images) were used for training. 15% of the images (= 1,500 images) were used for validation and another 15% for the test. The scatterer number N was constrained to five, considering that the concentration of scatterers under experimental conditions was not so high enough to justify N > 5. Then, we have tested experimental images using trained CNN and augmented them for test into 235, 99, 60, and 72 images corresponding to N = 1 ~ 4 by cropping and randomly

8


Page 8 of 29

Page 9 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


translating partial images and adding noise.

9



3. Results and discussion 3.1 Intensity-based analysis of SPM images Figure 2a shows SPM images of scatterers in the absence and presence of the background noise in the left and the right of the two columns. Addition of noise was marked with an arrow. The number of scatterers in each image (N) is varied between N = 2 and 9. The signal strength in the presence of noise was assessed in terms of SN = 10log(PS/PN) in dB, where PS and PN are signal and noise power calculated by the respective intensity sum over the pixels. In other words, without noise in a signallimited scenario (PS >> PN), SN  . In a noise-limited environment (PS  PN), SN  0. In the example of Figure 2a, 𝑆𝑁 was 13 dB in the 8-bit representation. The exact scatterer positions are marked by red dots. N and d also appear in the images. A SPM image generated by a single particle creates a unique pattern of parabola. When multiple scatterers are present in the R2 region, coherent interference between scattered fields produces distortion in the pattern and causes individual scattering characteristics such as the number of scatterers to be difficult to identify. Background noise, which is introduced to mimic experimental conditions, lowers the image signal-to-noise ratio (SNR) to SN =13 dB, as shown on the right in each pair of images in Figure 2a, and makes it even more challenging to identify scatterer characteristics. Figure 2b shows the distribution of per-pixel intensity (Ip), which was evaluated as the total intensity summed in the R2 region of SPM images divided by the number of pixels (31  31). Ip increases significantly up to N = 4. However, for N > 4, little change is observed with N and most of the intensity values fall within an error bar with a significant overlap. Figure 2b suggests that classification of SPM images with N by simple analysis of image intensity would be difficult. This confirms the need of an approach based on more elaborate algorithm for efficient classification of SPM images.

3.2 Conventional detection of monodisperse scatterers without deep learning We first present conventional detection method which we compare with the performance of the

10


Page 10 of 29

Page 11 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


proposed classification using deep learning based on the CNN. The conventional image reconstruction method relies on image processing in the Fourier space of SPM images followed by the deconvolution [16]. The image reconstruction method in itself cannot discriminate polydisperse scatterers. For comparison, we performed the image reconstruction and then predicted the number of scatterers by finding intensity peaks in a reconstructed image. Test images were obtained in the same way as described in 2.2. Figure 3a shows SPM images reconstructed by processing in the Fourier space when 𝑆𝑁 = 9 and 30 dB. For N = 2 (upper row), as the two scatterers become closer with the distance d reduced from d = 0.87 to 0.69 μm, two clearly separate peaks corresponding to the scatterer positions merge into one that is somewhat visible yet distorted. The reconstructed image then becomes completely distorted, which hampers correct prediction of the scatterer number. For the images in the bottom row, the number of scatters increases (thus smaller d). Even with much higher SNR at SN = 30 dB, peaks corresponding to the scatterer positions become much blurrier. In other words, the complexity of object structure, which is represented as an increase in the number of scatterers, remains difficult to resolve despite improved SNR. This feature emphasizes the limitation of the conventional reconstruction method in terms of accuracy. Figure 3b shows the performance estimation of accuracy from conventional image reconstruction by which intensity peaks are found, i.e., the accuracy is significantly degraded when the estimated scatterer number increases and if the estimation is performed in the presence of more severe background noise, i.e., lower SN, because the reconstruction method is prone to the noise and the interference between scattered fields, which increasingly affects the accuracy with more scatterers present in the region of interest. The data of accuracy are listed in Supporting Information Table S1. Even at the highest SN (SN = 30 dB), the classification results in almost zero accuracy for N > 4. In the worst background noise (SN = 1 dB), only the case of N = 1 can be classified with practical accuracy based on conventional detection.

11



Figure 3 (a) Reconstructed SPM images for N = 2, 4, and 5 at SN = 9 and 30 dB. Inset shows each SPM image before reconstruction. The number of scatterers (N) and the inter-scatterer distance d are also shown. Accuracy results in 3D with respect to SN and N by the prediction for monodisperse scatterers based on: (b) conventional reconstruction method and (c) deep learning. (d) SPM images, which is correctly predicted by the CNN, in the presence of background noise. The images are labeled with N, 𝑆𝑁 and inter-scatterer distance d. (e) Accuracy averaged over SN for conventional reconstruction (AR) and deep learning (AD). Accuracy enhancement factor (AEF) and differential accuracy ΔAR/ΔN and ΔAD/ΔN are also shown. (f) t-SNE visualization of the last hidden layer in the trained CNN for the nine

12


Page 12 of 29

Page 13 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


classes of the test set. Each colored clouds represent N = 1 ~ 9. (g) SPM images as the angle of incidence (θ) is varied to approach the resonance angle θSP = 56.1º. (h) Accuracy obtained at various angles of light incidence. Accuracy remains largely unaffected by θ up to N = 4. (i) Experimental images of monodisperse scatterers and augmentation by translation. (j) Classification accuracy of deep learning applied to experimental and theoretical images.

3.3 Training and test of SPM images using CNN We now investigate how deep learning overcomes the limit of conventional detection. Deep learning based on the CNN was first used to train and test with SPM images of monodisperse scatterers. The approach was then extended to the detection of polydisperse scatterers.

3.3.1 Classification of monodisperse scatterers In order to train and test the CNN, we have assumed 1 to 10 scatterers (N = 1 ~ 10) in the R2 region and generated 10,000 SPM images of randomly distributed scattering particles in each case of N. We have divided the image data into the set of training (70%), validation (15%), and test (15%). Due to the SPM image characteristics, the orientation along which a parabolic pattern is generated was determined by the direction of SP wave propagation: for this reason, the process of augmentation by rotating an image was omitted. Figure 3c shows the accuracy of classification with the number of monodisperse scatterers as the statistical nature of the noise is varied for SN = 1 ~ 30 dB. If noise increases from SN = 30 dB down to 1 dB, classification accuracy is reduced along with the SNR of an image. Because of more severe distortion by interference in the presence of more scatterers, the accuracy obviously decreases with N. As a special case of N = 1, the lack of inter-scatter interference leads to the accuracy reaching 100% for SN > 1 dB (the accuracy is 99.8% when SN = 1 dB). The accuracy that corresponds to a specific combination of background noise SN and scatterer number N is listed in Supporting Information Table S2. As an example, SPM images corresponding to a few combinations of N and SN are presented in

13



Page 14 of 29

Figure 3d for the correctly predicted cases by deep learning while Figure S2 presents incorrectly predicted images. Each image is labeled with N, SN, and d. We emphasize that deep learning can accurately predict the scatterer number even in the noisy environment and complex scattering patterns caused by multiple scatterers. In Figure 3e, accuracy averaged over SN was calculated for conventional reconstruction based method (AR) and deep learning (AD). AD is much higher than AR: quantitative estimation of the enhancement by deep learning was performed by defining accuracy enhancement factor AEF = AD/AR. Note that the way of prediction by the reconstruction and deep learning based on CNN is fundamentally different. Deep learning makes prediction between N = 1 and 10 through classification. On the other hand, the reconstruction method finds peaks in the reconstructed images and then counts the number of scatterers to predict N, therefore excludes a classification procedure. To take the difference into account, AEF was modified as 𝐴𝐸𝐹 =

∑ 𝑆 𝐴 𝐷 (𝑆 𝑁 ) 𝑁

∑𝑆 𝑀𝑎𝑥[𝐴𝑅(𝑆𝑁),10%], which sets the effective 𝑁

conventional accuracy as 10% if AR < 10% (10% is the probability to predict N correctly, when making a random prediction among 10 choices). Both AD and AR are the function that depend on SN. The compensated AEF is shown in Figure 3e. For N < 3, the accuracy improvement produced by deep learning is not significant with AEF < 2. However, for N  4 , AEF is drastically improved with a maximum at AEFmax = 5.9 when N = 5, suggesting that deep learning performs significantly better in terms of accuracy when interference becomes severe in the presence of multiple scatterers to complicate the identification of the number of scatterers. In contrast to AEF, which is a ratio of the accuracy averaged

over

SN,

we

have

also

defined

enhancement

factor

(EF)

as

𝐸𝐹(𝑆𝑁) =

𝐴𝐷(𝑆𝑁) 𝑀𝑎𝑥[𝐴𝑅(𝑆𝑁),10%] for each value of SN. For reference, we have plotted the EF with respect to SN in Figure S3. Insights can further be obtained with differential accuracy with respect to the scatterer number (ΔA/ΔN), which is a measure of how much the accuracy becomes worse when N increases by one. In case of deep learning, ΔAD/ΔN does not vary significantly with N and is largely maintained at about –10%. In contrast, the differential accuracy of the reconstruction method, ΔAR/ΔN, shows a very

14


Page 15 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


clear maximum of –34.8% when N increases from 3 to 4. Overall, the accuracy varies quite significantly with N, or equivalently, with d which decreases with N. Figure 3f shows internal features of the training results learned by the CNN for N = 1 ~ 9 and investigated through t-SNE (t-distributed Stochastic Neighbor Embedding), which is an algorithm applied to visualization by dimensional reduction [45]. The t-SNE algorithm tends to map datasets of similar features to a nearby point. Each point corresponds to a two-dimensional set reduced from a 1024-dimensional output of the last hidden layer. The clouds that correspond to N = 1 cluster on the opposite side of the ones corresponding to N = 9, while the cloud (N = 1) clusters close to those of N = 2 and 3. The nine distinct clouds clearly show the capability of the CNN algorithm of distinguishing SPM images with respect to the number of scatterers. Angle of light incidence as well as refractive index of scatterers was also found to affect the accuracy. Figure 3g presents SPM images acquired at an incident angle (θ) ranging from θ = 54.9° to 55.5° approaching the resonance angle θSP = 56.1°. Although the scattering pattern itself does not change, the contrast in the image decreases due to a narrower dynamic range in the reflectance. This eventually reduces SNR and leads to the lower accuracy, as shown in Figure 3h. Interestingly, the accuracy remains almost unchanged regardless of the angle of incidence up to N = 4, which is degraded quite visibly for N ≥ 5. The reduction of accuracy may be addressed by training neural network with SPM images obtained in a range of incident angles. Effect of scatterer refractive index can be understood in a similar context, i.e., reduced index contrast may decrease SNR and the prediction accuracy. The proposed deep learning was applied to the classification of monodisperse experimental data. Figure 3i presents the experimental images in the left column of each row augmented into right ones by translation following the procedure described in Section 2.4. In Figure 3j, the blue and the orange plot shows the accuracy obtained with theoretical and experimental images. Obviously, deep learning applied to experimental images in general confirms what we observed with theoretical data. Reduction of accuracy does occur in the case of experimental images from 100% to 97% for N = 1 and from 69% to 61% for N = 4, because of many factors that were not considered in the theoretical model. For

15



example, the intensity and the angle of light incidence are not uniform across the sample surface, making the pattern of SP scattering of individual scatterers non-uniform. Also, the contrast of SP scattering pattern is usually low for various reasons, i.e., motion artifacts due to vibration may make it difficult to remove the background, which results in lower contrast. Despite the potential limitation, the overall results strongly suggest that the proposed deep learning method may be applicable to the classification of experimental data. Deep learning can be made to be more efficient by employing more parameters to cope with diverse experimental environment and combining the method with traditional deconvolution [16].

Figure 4 (a) SPM images when randomly distributed obstructors were added. The scatterer and obstructor positions are marked by red and orange dots. The number of scatterers and obstructors are also shown. (b) Accuracy based on deep learning presented in 3D with respect to SN and N for the prediction of didisperse scatterers. N is the number of original scatterers. (c) Histogram of prediction results with respect to inter-scatterer distances d for N = 2 ~ 5: blue bars for the case of correct prediction (CP) and orange bars for incorrect prediction (IP) when SN = 13 dB. The plot (red) is the ratio of CP to IP. Columns represent the estimation of the scatterer number based on SPM image reconstruction (left column) and the classification by the CNN (center for monodisperse and right didisperse scatterers). (d)

16


Page 16 of 29

Page 17 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


The ratio of correct predictions to incorrect ones with the number of original scatterers (N) when d is smaller than the diffraction limit in the conventional and deep learning-based image reconstruction.

3.3.2 Classification of polydisperse scatterers We now consider a scenario in which polydisperse scatterers with disparate scattering coefficients are classified in SPM. For the convenience of analysis, we have assumed a didisperse mixture of two different types of particles that consist of originals and obstructors with s = 0.027 and 0.010 cm-1. The number of obstructors (Nobs) was randomly chosen between Nobs = 0 and 2 while their positions in the R2 are as random as the original scatterers, i.e., the total number of scatterers Ntot = N + Nobs (N: number of original scatterers). After SPM images were generated, background noise was added in a manner identical to the case of monodisperse scattering described in the previous section. Figure 4a shows SPM images generated with 𝑆𝑁 = 17 dB. Red and orange dots represent the position of original scatterers and obstructors. For convenience, the polarizability of a scatterer was assumed to be identical to the case of monodisperse scatterers, although the polarizability in the presence of obstructors can be estimated by solving an optimization problem between simulated and experimental SP scattering images. Overall the scattering images have become more complicated than those of monodisperse scatterers, because the presence of obstructors leads to stronger light scattering and also the number of scatterers increases by the amount of obstructors. The SPM images were trained to the identical CNN architecture using the number of original scatterers (N). Figure 4b shows the classification results in terms of accuracy when obstructors are present. Obviously, the accuracy is reduced with the introduction of obstructors, while the accuracy at N = 1 was not maintained at near 100% in contrary to the case of monodisperse scattering. The results indicate that the accuracy for didisperse scatterers is overall lower and, moreover, decreases much faster than for monodisperse scatterers, if we compare Figures 3c and 4b. The complete set of data presented in Figure 4b is listed in Table S3. The results, combined with repeated prediction after randomly removing scatterers and/or obstructors, suggest that deep learning by the CNN can extract useful information of scatterers in the presence of obstructors in polydisperse mixture (see Supporting

17



Information Figure S4).

3.4 Comparison and discussion More direct comparison between the performance of the conventional reconstruction and deep learning by the CNN is presented in Figure 4c. Figure 4c shows the histogram of prediction results for inter-scatterer distance d at 𝑆𝑁 = 13 dB, when correctly (blue bars) or incorrectly (orange bars) predicted by the conventional reconstruction in the left column and deep learning based on the CNN (center column for prediction of monodisperse scatterers and right one for didisperse scatterers). The line plot that connects red dots represents the ratio of correct predictions (CP) to incorrect ones (IP), i.e., CP/(CP + IP), for each d, as a measure of accuracy. The plot confirms increased IP with a shorter inter-scatterer distance when estimated with reconstruction method. In contrast, classification based on the CNN does not have a clear trend of prediction vs. inter-scatterer distance, suggesting in general much enhanced accuracy over the reconstruction method, particularly when N is large. To emphasize the differences, accuracy was plotted for d smaller than the diffraction limit ( ~ 230 nm) and presented in Figure 4d. In reconstruction method, the accuracy with N was 10% or less. On the other hand, the deep learning approach shows much higher accuracy ranging from 21.7% to 100%. This confirms that the deep learning approach is substantially more efficient to characterize scatterers in close proximity. If we further compare the CNN and the reconstruction method, the latter resorts to simple digital image processing without learning algorithm. Therefore, it does not require time and other computational resources needed to train data. However, as we have seen above, the CNN-based deep learning is found to perform fundamentally better, i.e., more accurate in the presence of multiple scatterers, while affected less by noise. Also, the CNN algorithm can handle classification of polydisperse scattering, which the reconstruction method cannot. For the same performance, deep learning can lead to simple and economic hardware architecture. On the other hand, the detection of objects of more complicated nature, e.g., polydisperse mixture of DNA and proteins with random shapes, would involve largely random scattering images, making it extremely difficult to estimate the characteristics of objects correctly because uniqueness of

18


Page 18 of 29

Page 19 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


an image cannot be guaranteed and the reconstruction process becomes ill-posed. Use of deep learning in this case can still be achieved by a sophisticated neural network model with more training and additional constraints. Note that CNN per se used in this work does not directly reveal which scatterers are counted. In regard to this point, Layer-wise Relevance Propagation (LRP) based on neural network may be employed, which allows visualization of how neural network learns and solves a problem by determining how much each pixel contributes to prediction [46]. Finally, a few remarks on the smallest distance of resolvable scatterers or effect of deep learning on image resolution are worth a note. As shown in Supporting Information Figure S5, the ability to resolve scatterers is not determined by a simple function of parameters, nor is it directly related to the Rayleigh criterion. For example, it is affected by the distribution of scatterers relative to the direction of propagation of SP waves and also the number of scatterers in the field-of-view. Yet the resolution that is made feasible by deep learning is much smaller than the propagation length of SP (~ 4.64 μm at  = 680 nm), which typically determines the resolution of SPM. Because neural network was trained with images of randomly positioned scatterers, the distribution of inter-scatterer distance (d) in the training images takes a bell-shaped distribution, rather than a constant distribution. For this reason, neural network tends to be trained less with a small d. If neural network is trained more in this range, we expect that scatterers can be better resolved. A variety of strategies in addition to more training may be taken to achieve significant enhancement in the resolution, for example, with a hybrid approach combining deep learning and conventional method.

4. Concluding remarks In this study, we have applied CNN-based deep learning to SPM of light scattered by nanoparticles. The number of scatterers among many scattering parameters was estimated in mono- and didisperse particle mixtures with a deep learning algorithm used for the image classification. It was found that deep learning-based SPM enhances the accuracy significantly over image reconstruction method by almost six times. The results are expected to help understand scattering characteristics and

19



can be directly extended to other label-free molecular detection assays that involve particle scatterers. This study can be generalized to scattering by more scatterers that form a complicated interference pattern in a much larger field of view and also for the classification by the CNN in diverse architecture.

Supporting information Supporting Information Available: S1 Assessment of experimental signal-to-noise ratio SN; Table S1 Accuracy for monodisperse scatterers based on the conventional reconstruction method; Table S2 Accuracy for monodisperse scatterers based on the deep learning method; Figure S2 SPM images that were incorrectly predicted by the deep learning method; Figure S3 Enhancement factor (EF) for each value of SN; Table S3 Accuracy for didisperse scatterers based on the deep learning method; Figure S4 Repeated prediction of the number of scatterers after randomly removing scatterers and obstructors in the didisperse mixture; Figure S5 Effect of deep learning on the smallest distance of resolvable scatterers. This material is available free of charge via the Internet at http://pubs.acs.org.

Acknowledgment This work was supported by the National Research Foundation (NRF) grants funded by the Korean Government (2018R1D1A1B07042236 and 2019R1F1A1063602).

References [1] Brockman, J. M.; Nelson, B. P.; Corn, R. M. Surface Plasmon Resonance Imaging Measurements of Ultrathin Organic Films. Annu. Rev. Phys. Chem. 2000, 51, 41-63. [2] Huang, B.; Yu, F.; Zare, R. N. Surface Plasmon Resonance Imaging Using a High Numerical Aperture Microscope Objective. Anal. Chem. 2007, 79, 2979–2983. [3] Son, T.; Kim, D. Theoretical Approach to Surface Plasmon Scattering Microscopy for Single Nanoparticle Detection in Near Infrared Region. Proc. SPIE 2015, 9340, 93400W.

20


Page 20 of 29

Page 21 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


[4] Jiang, Y.; Wang, W. Point Spread Function of Objective-Based Surface Plasmon Resonance Microscopy. Anal. Chem. 2018, 90, 9650-9656. [5] Nelson, B. P.; Frutos, A. G.; Brockman, J. M.; Corn, R. M. Near-Infrared Surface Plasmon Resonance Measurements of Ultrathin Films. 1. Angle Shift and SPR Imaging Experiments. Anal. Chem. 1999, 71, 39283934. [6] Shumaker-Parry, J. S.; Aebersold, R.; Campbell, C. T. Parallel, Quantitative Measurement of Protein Binding to a 120-Element Double-Stranded DNA Array in Real Time Using Surface Plasmon Resonance Microscopy. Anal. Chem. 2004, 76, 2071-2082. [7] Lindquist, N. C.; Lesuffleur, A.; Im, H.; Oh, S.H. Sub-Micron Resolution Surface Plasmon Resonance Imaging Enabled by Nanohole Arrays with Surrounding Bragg Mirrors for Enhanced Sensitivity and Isolation. Lab Chip 2009, 9, 382-387. [8] Halpern, A. R.; Chen, Y.; Corn, R. M.; Kim, D. Surface Plasmon Resonance Phase Imaging Measurements of Patterned Monolayers and DNA Adsorption onto Microarrays. Anal. Chem. 2011, 83, 2801-2806. [9] Wang, S.; Shan, X.; Patel, U.; Huang, X.; Lu, J.; Li, J.; Tao, N. Label-Free Imaging, Detection, and Mass Measurement of Single Viruses by Surface Plasmon Resonance. Proc. Natl. Acad. Sci. U.S.A. 2010, 107, 1602816032. [10] Nizamov, S.; Scherbahn, V.; Mirsky, V. M. Detection and Quantification of Single Engineered Nanoparticles in Complex Samples Using Template Matching in Wide-Field Surface Plasmon Microscopy. Anal. Chem. 2016, 88, 10206–10214. [11] Halpern, A.R.; Wood, J. B.; Wang, Y.; Corn, R. M. Single-Nanoparticle Near-Infrared Surface Plasmon Resonance Microscopy for Real-Time Measurements of DNA Hybridization Adsorption. ACS Nano 2013, 8, 1022-1030. [12] Nizamov, S.; Scherbahn, V., Mirsky, V. M. Ionic Referencing in Surface Plasmon Microscopy: Visualization of The Difference in Surface Properties of Patterned Monomolecular Layers. Anal. Chem. 2017, 89, 3873-3878. [13] Giebel, K. F.; Bechinger, C.; Herminghaus, S.; Riedel, M.; Leiderer, P.; Weiland, U.; Bastmeyer, M. Imaging of Cell/Substrate Contacts of Living Cells with Surface Plasmon Resonance Microscopy. Biophys. J. 1999, 76, 509-516. [14] Wang, W.; Wang, S.; Liu, Q.; Wu, J.; Tao, N. Mapping Single-Cell–Substrate Interactions by Surface

21



Plasmon Resonance Microscopy. Langmuir 2012, 28, 13373-13379. [15] Son, T.; Seo, J.; Choi, I.; Kim, D. Label-Free Quantification of Cell-to-Substrate Separation by Surface Plasmon Resonance Microscopy. Opt. Commun. 2018, 422, 64-68. [16] Yu, H.; Shan, X.; Wang, S.; Tao, N. Achieving High Spatial Resolution Surface Plasmon Resonance Microscopy with Image Reconstruction. Anal. Chem. 2017, 89, 2704-2707. [17] Kim, D.; Kim, D.; Subwavelength Grating-Based Nanoplasmonic Modulation for Surface Plasmon Resonance Imaging with Enhanced Resolution. J. Opt. Soc. Am. B 2010, 27, 1252-1259. [18] Peterson, A. W.; Halter, M.; Tona, A.; Plant, A. L. High Resolution Surface Plasmon Resonance Imaging for Single Cells. BMC Cell Biol. 2014, 15, 35. [19] Son, T.; Lee, C.; Seo, J.; Choi, I.; Kim, D. Surface Plasmon Microscopy by Spatial Light Switching for Label-Free Imaging with Enhanced Resolution. Opt. Lett. 2018, 43, 959-962. [20] Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res. 2014, 15, 1929-1958. [21] Ioffe, S.; Szegedy. C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. 2015, arXiv:1502.03167. arXiv.org e-Print archive. https://arxiv.org/abs/1502.03167v3 (accessed Mar 2, 2015). [22] Henriques, J. F.; Caseiro, R.; Martins, P.; Batista, J. High-Speed Tracking with Kernelized Correlation Filters. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 583-596. [23] Krizhevsky, A.; Sutskever, I.; Hinton, G. E. Imagenet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 1, Lake Tahoe, NV, December 3-6, 2012, 1097-1105. [24] He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, June 27-30, 2016; 770-778. [25] Noh H.; Hong S.; Han B. Learning Deconvolution Network for Semantic Segmentation. Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, December 7-13, 2015; 1520-1528. [26] Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. 2015, arXiv:1606.00915. arXiv.org e-Print archive. https://arxiv.org/abs/1606.00915v2 (accessed May 12, 2017). [27] Hinton, G.; Deng, L.; Yu, D.; Dahl, G. E.; A. R. Mohamed.; Jaitly, N.; Senior, A.; Vanhoucke, V.; Nguyen,

22


Page 22 of 29

Page 23 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


P.; Sainath, T. N.; Kingsbury, B. Deep Neural Networks for Acoustic Modeling in Speech Recognition: the Shared Views of Four Research Groups. IEEE Signal Process. Mag. 2012, 29, 82-97. [28] Ma, W.; Cheng, F.; Liu, Y. Deep-Learning Enabled On-Demand Design of Chiral Metamaterials. ACS Nano 2018, 12, 6326-6334. [29] Mazaheri, H.; Ghaedi, M.; Azqhandi, M. A.; Asfaram, A. Application of Machine/Statistical Learning, Artificial Intelligence and Statistical Experimental Design for the Modeling and Optimization of Methylene Blue and Cd(II) Removal from a Binary Aqueous Solution by Natural Walnut Carbon. Phys. Chem. Chem. Phys. 2017, 19, 11299-11317. [30] Baltz, E. A.; Trask, E.; Binderbauer, M.; Dikovsky, M.; Gota, H.; Mendoza, R.; Platt, J. C.; Riley, P. F. Achievement of Sustained Net Plasma Heating in a Fusion Experiment with the Optometrist Algorithm. Sci. Rep. 2017, 7, 6425. [31] Alipanahi, B.; Delong, A.; Weirauch, M. T.; Frey, B. J. Predicting the Sequence Specificities of DNA-and RNA-Binding Proteins by Deep Learning. Nat. Biotechnol. 2015, 33, 831-838. [32] Chen, C. L.; Mahjoubfar, A.; Tai, L. C.; Blaby, I. K; Huang, A.; Niazi, K. R.; Jalali, B. Deep Learning in Label-Free Cell Classification. Sci. Rep. 2016, 6, 21471. [33] Esteva, A.; Kuprel, B.; Novoa, R. A.; Ko, J.; Swetter, S. M.; Blau, H. M.; Thrun S. Dermatologist-Level Classification of Skin Cancer with Deep Neural Networks. Nature 2017, 542, 115-118. [34] Kamilov, U. S.; Papadopoulos, I. N.; Shoreh, M. H.; Goy, A.; Vonesch, C.; Unser, M.; Psaltis, D. Learning Approach to Optical Tomography. Optica 2015, 2, 517-522. [35] Satat, G.; Tancik, M.; Gupta, O.; Heshmat, B.; Raskar, R. Object Classification through Scattering Media with Deep Learning on Time Resolved Measurement. Opt. Express 2017, 25, 17466-17479. [36] Horisaki. R.; Takagi. R.; Tanida, J. Learning-Based Imaging through Scattering Media. Opt. Express 2016, 24, 13738-13743. [37] Nguyen, T.; Bui, V.; Lam, V.; Raub, C. B.; Chang, L. C.; Nehmetallah, G. Automatic Phase Aberration Compensation for Digital Holographic Microscopy Based on Deep Learning Background Detection. Opt. Express 2017, 25, 15043-15057. [38] Ballard, Z. S.; Shir, D.; Bhardwaj, A.; Bazargan, S.; Sathianathan, S.; Ozcan, A. Computational Sensing Using Low-Cost and Mobile Plasmonic Readers Designed by Machine Learning. ACS Nano 2017, 11, 22662274.

23



[39] Liang, W.; Wang, S.; Festa, F.; Wiktor, P.; Wang, W.; Magee, M.; Labaer, J.; Tao, N. Measurement of Small Molecule Binding Kinetics on a Protein Microarray by Plasmonic-Based Electrochemical Impedance Imaging. Anal. Chem. 2014, 86, 9860-9865. [40] Yu, H.; Shan, X.; Wang, S.; Chen, H.; Tao, N. Plasmonic Imaging and Detection of Single DNA Molecules. ACS Nano 2014, 8, 3427-3433. [41] Yu, H.; Shan, X.; Wang, S.; Chen, H.; Tao, N. Molecular Scale Origin of Surface Plasmon Resonance Biosensors. Anal. Chem. 2014, 86, 8992-8997. [42] Ober, R. J.; Ram, S.; Ward, E. S. Localization Accuracy in Single-Molecule Microscopy. Biophys. J. 2004, 86, 1185-1200. [43] Aikens, R. S.; Agard, D. A.; Sedat, J. W. Solid-State Imagers for Microscopy. In Methods in Cell Biology; Taylor, D. L., Wang, Y.-L., Eds.; Academic Press: Cambridge, MA, 1988; Vol. 29, pp 291-313. [44] Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. 2014, arXiv:1409.1556. arXiv.org e-Print archive. https://arxiv.org/abs/1409.1556v6 (accessed Apr 10, 2015). [45] Maaten, L. V. D.; Hinton, G. Visualizing Data Using T-SNE. J. Mach. Learn. Res. 2008, 9, 2579-2605. [46] Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K. R.; Samek, W. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation. PLoS One 2015, 10, e0130140.

24


Page 24 of 29

Page 25 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Figure 1 (a) Schematics of the particle scattering model by propagating SP waves. (b) Diagram of a unit simulation in an area of 129 × 129 pixels (R1) with schematic in this case corresponding to N = 2 and 3. The position of N scattering particles was limited to within 31 × 31 pixels (R2). R2 was located at the center of an entire simulation area (R1). A pixel is a square of 100-nm sides. (c) Distribution of the inter-scatterer distance d defined as the distance between the nearest scatterers in the R2 region. The number of scatterers (N) and the average inter-scattering distance (davg) are also shown. Unit of davg: μm. (d) Schematic diagram of the CNN architecture. It consists of Convolution layer (Conv), Batch normalization (BN), Rectified linear unit (Relu), Max pooling layer (Pooling), Fully connected layer (FC) and output layer. 156x113mm (300 x 300 DPI)



Figure 2 (a) SPM images as the number of randomly distributed scatterers (N) changes: N = 2 ~ 9 from top left to bottom right. Each column consists of SPM images (left) without and (right) with the addition of background Poisson noise (SN = 13 dB): the addition marked by an arrow at the top of each column. Unit of d: μm. The scatterer positions are marked by red dots. (b) Distribution of per-pixel intensity (Ip) in each of the images. Each circle corresponds to one image and 100 circles are represented per N. The average of Ip appears below the cluster of circles. Error bars represent the standard deviation. Intensity average at each N was connected in line without interpolation to show the trend. 156x180mm (300 x 300 DPI)


Page 26 of 29

Page 27 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Figure 3 (a) Reconstructed SPM images for N = 2, 4, and 5 at SN = 9 and 30 dB. Inset shows each SPM image before reconstruction. The number of scatterers (N) and the inter-scatterer distance d are also shown. Accuracy results in 3D with respect to SN and N by the prediction for monodisperse scatterers based on: (b) conventional reconstruction method and (c) deep learning. (d) SPM images, which is correctly predicted by the CNN, in the presence of background noise. The images are labeled with N, SN and interscatterer distance d. (e) Accuracy averaged over SN for conventional reconstruction (AR) and deep learning (AD). Accuracy enhancement factor (AEF) and differential accuracy ΔAR/ΔN and ΔAD/ΔN are also shown. (f) t-SNE visualization of the last hidden layer in the trained CNN for the nine classes of the test set. Each colored clouds represent N = 1 ~ 9. (g) SPM images as the angle of incidence (θ) is varied to approach the resonance angle θSP = 56.1º. (h) Accuracy obtained at various angles of light incidence. Accuracy remains largely unaffected by θ up to N = 4. (i) Experimental images of monodisperse scatterers and augmentation by translation. (j) Classification accuracy of deep learning applied to experimental and theoretical images. 153x157mm (300 x 300 DPI)



Figure 4 (a) SPM images when randomly distributed obstructors were added. The scatterer and obstructor positions are marked by red and orange dots. The number of scatterers and obstructors are also shown. (b) Accuracy based on deep learning presented in 3D with respect to SN and N for the prediction of didisperse scatterers. N is the number of original scatterers. (c) Histogram of prediction results with respect to interscatterer distances d for N = 2 ~ 5: blue bars for the case of correct prediction (CP) and orange bars for incorrect prediction (IP) when SN = 13 dB. The plot (red) is the ratio of CP to IP. Columns represent the estimation of the scatterer number based on SPM image reconstruction (left column) and the classification by the CNN (center for monodisperse and right didisperse scatterers). (d) The ratio of correct predictions to incorrect ones with the number of original scatterers (N) when d is smaller than the diffraction limit in the conventional and deep learning-based image reconstruction. 156x101mm (300 x 300 DPI)


Page 28 of 29

Page 29 of 29 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


ToC 68x44mm (300 x 300 DPI)


Deep Learning Approach for Enhanced Detection of Surface Plasmon

Recommend Documents