Anal. Chem. 1995, 67, 1381-1 389
Modified Algorithm for Standardization of Near-Infrared Spectrometric Instruments E. Bouveresse and D. L. Massart* ChemoAC, Pharmaceutical Institute, VrJe Universiteit Brussel, Laabeeklaan 103, B- 7 090 Brussels, Belgium P. Dardenne Station de Haute Belgique, Rue de Setpont 100, 8-6800Libramont-Chevigny, Belgium
The transfer of calibration models between near-Mared spectrometric instruments with the algorithm proposed by S h e d and Westerhaus as a standardization method is studied. This algorithm does not yield optimal results when standardizationsamples are used that are not of the same nature as the samples to be analyzed. In an attempt to improve the results obtained with this algorithmwhen one uses such standardization samples, five different modifications of the spectral intensity correction step of the algorithm are investigated. The use of locallyweighted regression,which gives more weight to the standardization samples close to the same spectral intensity range where the samples to be predicted are located and less weight to the farthest samples, seems to be the most suitable one. Because of the differences between instrumental responses of newinfrared spectrometers, standardization has become a necessary step to correct those differences and to avoid timeconsuming recalibration procedures. Among the standardization methods proposed in the literature, the patented method proposed by Shenk and Westerhaus1,2is widely applied. In this paper, this algorithm will be referred to as the “patented algorithm. It is based on the transfer of spectra from the “slave” instrument to the “master”one and involves two steps, namely wavelength index correction and spectral intensity correction. The spectral intensity correction involves a linear regression relating wavelength by wavelength the spectral intensities obtained on a first instrument with those obtained on a second one. In a previous study? calibrationtransfer involving standardization samples, independent from the samples to be predicted, was tested. Two different approaches were investigated: the first involved using an independent standardization set containing samples similar to those to be predicted, and the second involved using two independent standardization sets containing chemically more stable samples very different from those to be predicted. A mixed approach was studied by Wang and K~walski:~ they used two generic standards and a sample selected from the set of samples to be predicted as the standardization set and applied the piecewise direct standard(1) Shenk, J. S.; Westerhaus, M. 0. Crop Sci. 1991,31, 1694-1696. (2) Shenk, J. S.; Westerhaus, M. 0. US. Patent 4866644,Sept 12, 1989. (3) Bouveresse, E.; Massart, D. L.; Dardenne, P. Anal. Chim.Acta 1994,297, 405-416. (4) Wang, Y.; Kowalski, B. R Appl. Spectrosc. 1992,46, 764-771. 0003-2700/95/0367-1381$9.00/0 Q 1995 American Chemical Society
ization m e t h ~ d .As ~ far as we know, a pure generic standard approach using the patented algorithm has not been proposed yet. Our previous study3showed that the use of standardization samples similar to the samples to be predicted can lead to good standardization but is applicable only for those samples. The use of standardization samples of different nature, which cover a larger spectral intensity range, leads to poor results. The use of such standardization samples would be more suitable, because very stable standardization samples could then be measured to transfer spectra of dBerent sets of samples. To explain the results obtained, the first reason proposed is the different nature of the samples involved. Yet, it is also possible that the algorithm presents some limitations when the data sets involved exceed the usual framework of application. For instance, if the spectral variation range covered by the standardization samples is really wide, nonlinearities might occur, and the linear regression used in the patented algorithm might lead to bad results. Changing the regression step to a more suitable one may then improve standardization. Moreover, high spectral intensity values are probably more subject to noise. Since a low number of standardization samples are involved to compute standardization parameters, points located in high spectral intensity ranges and subject to relatively large noise can have a bad influence on the linear regression model. Changing the simple linear regression to a more suitable one may also improve standardization. As explained later, it is not possible to simply eliminate high absorbances from consideration as this would lead to artifacts (discontinuities) in the standardized spectra. In other words, if the patented algorithm enables good standardization in some cases, the bad results obtained in other cases can perhaps be improved by some modifications of the algorithm. To cope with the problems previously mentioned, the following techniques are investigated. Use of Univariate Quadratic Models. In order to better fit nonlinearities due to wide spectral intensity ranges, the spectral intensities of both instruments are related wavelength by wavelength with quadratic models. Iteratively Reweighted Least This iterative technique is applied in order to decrease the effect of spectral intensities subject to noise on the regression lines obtained. (5) Wang, Y.; Veltkamp, D. J.; Kowalski, B. R Anal. Chem. 1991,63, 27502756. (6) Mosteller, F.; Tukey, J. W. Data analysis and regression; Addison-Wesley: Reading, MA, 1977. (7) Phillips, G. R; Eyring, E. R Anal. Chem. 1983,55, 1134-1138.
Analytical Chemistry, Vol. 67, No. 8, April 15, 1995 1381
Least Median of Sq~ares.~-ll This robust regression technique is applied to cancel outlying spectral intensities, which can have strong influences on the regression lines obtained. Locally Weighted R e g r e s ~ i o n . ~ ~This - - ' ~technique is applied to relate spectral intensities of the standardization samples measured on both instruments by taking into account the spectral intensity region where the samples from the set to be predicted are located. For this technique, different standardization parameters have to be computed for each different prediction set, since the spectra of the samples to be predicted are used to compute those "local" standardization parameters. THEORY Notations. Samples from standardization set are measured on both instruments: the X, and X, matrices give the spectra of the N, standardization samples measured at N, wavelengths on the slave and master instruments. The Npsamples to be predicted are measured only on the slave instrument, and the spectra obtained are stored in the X, matrix. For each matrix X which contains spectra, X j represents the element of the ith column and $h row of the X matrix and corresponds to the spectral intensity of the 3th sample at the ith wavelength. Xj represents the firow of the X matrix and corresponds to the spectnun of the jth sample. XIrepresents the ith column of the X matrix and corresponds to the spectral intensities of all samples at the ith wavelength. Methods. SW: Patented Standardization Mefhod. A detailed description of this algorithm is given in a previous paper.3 The patented method is based on two main steps: wavelength index correction, followed by spectral intensity correction. All spectra from the standardization set are transformed by a first derivative mathematical treatment, and those first derivative spectra are used to compute the parameters of the wavelength adjustment. After that, wavelength adjustment of the initial spectra from X, is carried out, yielding X,# (matrix that contains spectra of the standardizationsamples measured on the slave instrument, after wavelength adjustment). For each wavelength i, spectral intensity correction is then obtained by linear regression of the responses of the slave instrument after wavelength adjustment X,# on the response of the master instrument at the corresponding wavelength X,,:
IRLS: Iteratively Reweighted Least Squares. IRLS is a robust method suggested by Mosteller and Tukep which has been successfully applied to chemical data IRLSconsists of an iterative application of weighted least squares, with the weights determined from the data. IRLS is applied for each wavelength i, relating the response obtained on the master instrument, X k , to the corresponding response on the slave one, Xst#. X,P and X,i are kst centered. At each iteration, weighted least-squares regression is performed: the regression coefficients are computed according to (3) where W' is an N, x N, matrix, with WUGj) being the weight corresponding to standardization sample j after a iterations, and W'G,k) = 0 if k z j . The initial W is the identity matrix (all weights are set equal to 1). Predictions of X k are then computed, and residuals are calculated:
ri" = x,j - X,i#B"
(4)
From those residuals, weights for the next iteration are computed
Here, k is the tuning constant and 9 is defined as follows:
s" = median (Ir:[)
(6)
This weight function gives high weights to the points with low residuals and vice versa. Iteration continues until the difference between the regression coefficients a and satisfies a convergence criterion. Once this criterion is fulfilled, the B regression coefficients computed are stored in the standardization file, and the same calculations are carried out for the i lth wavelength. LMS: LRasf Median of Squares. LMS is a robust method introduced by Rou~seeuw.~ A detailed description of the method is available,1° and the k s t application of LMS to analytical chemistry is described in another article." LMS consists of computing the equations of all lines between all possible pairs of points. For each line, squared residuals for all the points are computed, the squared residuals are sorted in increasing order, and the median of those squared residuals is found. The line for which the median of the squared residuals is the minimum is chosen as the LMS linear model. For each wavelength i, LMS is applied for the Ns points (Xvt#,Xwl).A robust model is determined, relating the response obtained on the master instrument, X,,, with the response obtained on the slave one, X,!. LWR: Locally Weighted Regression. LWR is also a weighted regression method, but the weights applied to the calibration samples used are based not on the residuals but on the distance between the calibration samples and the samples to be predicted.
+
where intercept (a) and slope (b) are computed for each wavelength i. In order to improve the fit of the standardization samples by the computed model, four different regression methods are proposed in this paper. SWQ: Regression with Quadratic Modeki. Instead of a linear relation, a quadratic model is used, yielding (8) Vankeerberghen, P.; Vandenbosch, C.; Smeyers-Verbeke, J.; Massart, D. L. Chemom. Intell. Lab. Syst. 1991,12, 3-13. (9) Rousseeuw, P. J. 1.Am. Stat. Assoc. 1984,79,871-880. (10) Rousseeuw,P. J.; LeroyA M. Robust Regression and OutlierDetection; John Wiley & Sons, Inc.: New York, 1987. (11) Massart, D. L.; Kaufman, L;Rousseeuw, P. J.; Leroy, A M. Anol. Chim. Acta 1986,187, 171-179. (12) Cleveland, W. S.; Devlii, S. J. J. Am. Stat. Assoc. 1988,83, 596-610. (13) Wang, 2.;Isaksson,T.; Kowalski, B. R Anol. Chem. 1994,66, 249-260. (14) Naes, T.; Isaksson,T.Appl. Spectrosc. 1992,46, 34-43. (15) Naes. T.; Isaksson, T.; Kowalski, B. R Anal. Chem. 1990,62, 668-673.
1382 Analytical Chemistry, Vol. 67,No. 8,April 15, 1995
The expected main advantage of LWR is that more weight is given to the calibration samples that are the closest to the samples to be predicted. An important step of LWR is to select the local calibration samples by measuring distances to the samples to be predicted. As proposed by Cleveland and Devlin,lZ we applied ordinary Euclidean distance. In our case, locally weighted regression is applied to the standardization samples. LWR is applied for each wavelength i,relating the response obtained on the master instrument, Xmi, to the corresponding response on the slave one, X,?. The first step consists of computing ab), the sum of the squared Euclidean distances between the standardization sample j and each sample to be predicted:
The 6 6 ) values computed for each standardization sample j are then sorted in increasing order. The number of closest standardization samples to be used (referred to as A9 is chosen. All sorted 6 6)values are divided by the N lth 6 6)value, which is referred to as db); this rescaling enables us to obtain SO)/db) values between 0 and 1for the first N standardization samples. The other samples have SQ)/d(i) values equal to or higher than 1. The standardization sample closest to the prediction samples has the smallest 66)/db) value. The following cubic weight function is used to compute the weights, W(jJ'),corresponding to each standardization sample i:
+
WGj) = (1- ( d O / d 6 ~ ~ ) ~i f d 0 / d 6 ) WGJ) = 0 otherwise
