Anal. Chem. 1994,66, 4507-4513
Generalized Fourier Smoothing of Flow Injection Analysis Data Oliver Lee and Adrian P. Wade'
Department of Chemistry, University of British Columbia, Vancouver, British Columbia, Canada V6T 1Z1 Guy A. Dumont
Department of Electrical Engineering, c/o Pulp and Paper Centre, University of British Columbia, Vancouver, British Columbia, Canada V6T 124
The Fourier transform is widely used for smoothing data such as those from flow injection analysis (FIA). The effectiveness of this method can be enhanced if, in addition to the standard complex exponential functions, the Fourier transform is generalized to use other sets of complete, orthogonal functions such as the Gram or Mekner polynomials as its basis functions. The choice of which set of basis functions to use depends on its efficiency on a given peak. Using simulated noisy FIA peaks differing in degree of skewness, it was found that the standard complex exponential set is best-suited for symmetric or nearly symmetric peaks, and the Mekner set, for moderate to greatly skewed peaks. The Gram set weakly favors skewed peaks, but it is not more effective than both the complex exponential and Mekner sets over any portion of the skewness range studied. The problem of determiningthe optimal spectral cutoff point was cast in terms of hierarchical model selection, and a generalized Akaike information-theoretic criterion (GAIC) was evaluated for its ability to find the best filter order. Use of an efficient basis minimizes the chance of selecting a nonoptimal filter order. The combination of generalized Fourier filtering and the GAIC provides an attractive means to filter FIA data automatically. Filtering is often performed on raw analytical signals obtained from flow injection analysis (FIA). By increasing the signal-te noise ratio (S/N), a desired signal parameter, say, peak height, may be extracted by direct measurement. Over the years, the task of enhancing S/N has fallen increasingly on digital filters. Broadly defined, digital filters are numeric algorithms that transform an input data series into a (more useful) output series. Those designed for reduction of noise whose frequency components are above that of the signal's are often referred to as smoothers. Digital filtering via the discrete Fourier transform is one popular method for smoothing chemical data.'-8 In this, the data are converted to the frequency domain, where signal and noise (1) Horlick, G. Anal. Chem. 1972,44, 943. (2) Kirmse, D. W.; Westerberg, A W. Anal. Chem. 1971,43, 1035. (3) Maldacker, T.A; Davis, J. E.; Rogers, L. B. Anal. Chem. 1974,46, 637. (4) Bush, C. A Anal. Chem. 1974,46, 890. (5) Binkley, D. P.; Dessy, R E. Anal. Chem. 1980,52, 1335. 0003-2700/94/0366-4507$04.50/0 Q 1994 American Chemical Society
may be resolved on the basis of spectral differences. By multiplication, some appropriate filter function is used to sign& cantly attenuate, if not eliminate, those frequency components attributed largely to noise, but to preserve those ascribed to the signal. In smoothing, the idealized filter function is rectangular; that is, it has unit value for frequencies less than some specified cutoffrequency and zero for all higher frequencies. Finally, the inverse transform is performed on the result to construct a smooth (er) timedomain signal. The greater and/or the sharper the difference between the frequencies spanned by signal and noise, the higher the prospect of recovering the signal itself. It is known that the Fourier transform is simply general linear least-squares modeling with a hierarchical and complete set of orthogonal complex exponential polynomials as basis functions.9 Choosing orthogonal basis functions over arbitrary ones ensures numerical stability.l0 Clearly, Fourier filtering can be generalized to other complete, orthogonal sets of functions such as the Legendre or Laguerre polynomials, which are continuous, or analogously, the Gram or Meixner polynomials, which are discrete. Indeed, Hermite and Chebyshev polynomials have been used in this way to filter chromatographic data." Extensive lists of orthogonal polynomials can be found in the literature.12J3 Since Fourier smoothing amounts to signal approximation in the presence of noise, use of other basis functions (i) increases the repertoire of filtering tools to deal more effectively with a variety of peak shapes and (ii) is an elegant alternative to modifying classical Fourier spectra with nonrectangular filter functions, ad hoc or model-based. According to proper curve-fitting practice, the basis functions chosen should be similar to the function being fitted so that high efficiency is achieved,14that is, so that only a small number of basis functions is needed for high accuracy. (6) Lam, R B.; Isenhour, T.L. Anal. Chem. 1981,53, 1179. (7)Felinger, A; Pap, T. L.; Incddy, J. Anal. Chim. Acta 1991,248, 441. (8) Larivee, R J.; Brown, S. D. Anal. Chem. 1992,64, 2057. (9) Erdelyi,A Higher Transcendental Functions; McGraw-Hill: New York, 1953; VOl. II. (10) Bialkowski, S. E. Anal. Chem. 1989,61, 1308. (11) Scheeren, P. J. H.; Klous, Z.; Smit, H. C.; Doombos, D. AAnul. Chim. Acta 1985,171, 45. (12) Abramowitz, M.; Stegun, I. A Handbook ofMathematica1 Functions; National Bureau of Standards: Washington, DC, 1964. (13) Szego, S. Orthogonal Polynomkls, rev. ed.; American Mathematical Society Colloquium Publications 23; Providence, RI, 1959. (14) Deutsch, R System Analysis Techniques; RenticeHall: Englewood Cliffs, NJ, 1969; p 294. Analytical Chemistty, Vol. 66, No. 24, December 15, 1994 4507
A well-recognized problem associated with Fourier smoothing is that of determining the optimal spectral cutoff, where noise is removed as much as possible without compromising signal integrity.’+ When the signal and noise spectral distributions are completely separated,the cutoff is obvious,but when they overlap, which is often the case, it is less determinate. Spectraltruncation can be a haphazard operation unless there is accurate prior knowledge on both the signal and noise.15 Unfortunately, such information is usually not readily available. Thus a number of ad hoc methods have been devised in analytical chemistry to select the “best” ~ u t o f f . ~Felinger -~ et al.7compared some of these with simulated Gaussian peaks and found that the methods described in refs 2 and 6 tended to select cutoffs lower than the optimal, but the opposite was true for that described in ref 3. The drawback of these techniques is that either an arbitrary decision is r e q ~ i r e d ~or- ~a theoretical model is e ~ p l o i t e d . ~ Direct -~ (nonlinear) curve fitting may be more appropriate when a theoretical model is available. Recently, a criterion based on maximization of Shannon’s information entropy was proposed.* The use of information theory represents a signifcant conceptual advance over ad hoc methodology. A number of other statistical methods are documented in the literature.16-18 Interest in the use of automated flow injection analyzers for research, environmental monitoring, and process control has grown of late.19%20 On such systems, it is desirable that filtering be performed not only automatically but also optimally. Given the variety of peak shapes found in FIA, a filtering method based on general polynomial models promotes wide applicability and facilitates model identification. The latter is signifcant because, to date, theoretical models have been developed for very simple analyzers only, and even these models are not sufticiently accurate for filteringSz1Furthermore, it is not difficult to optimize a singlefilter parameter. By comparison, optimization of Savitzky-Golay filters is complicated by many local minima in the filter lengthfilter order parameter space.s In this paper, the performance of digital Fourier smoothers based on the discrete complex exponential, Gram, and Meixner polynomials was evaluated with both simulated and real FIA peaks. A generalized Akaike information-theoretic criterion has been adopted and evaluated for automatic near-optimal spectral truncation. THEORY GeneralizedFourier Smoothing. Given that a set of N data points y[kl of equal variance is to be fitted to the general linear
model M- 1
given by the normal equations 1
(2) where * is the complex conjugate operator. If the basis functions &[kl are orthogonal, and in particular, orthonormal, that is, n-1 1 ifm=n
~&[kI4,[kl
k=O
(15) Bialkowski, S. E.Anal. Chem. 1988,60, 355A. (16)Sijderstrom, T. Int. J. Control 1977,26, 1. (17) Leonhitis, I. J.; Billings, S. A. Int. J. Control 1987,45, 311. (18)Bozdogan, H.Aychometrika 1987,52, 345. (19) Lee, 0.;Dumont, G. A; Toumier, P.; Wade, A. P.Anal. Chem. 1994,66, 971.
(20) Wentzell, P. D.; Hatton, M. J.; Shiundu, P. M.; Ree, R M.; Wade, A P.; Betteridge, B.; Sly, T. J. J Autom. Chem. 1989,11, 227. (21) Ruzicka,J.; Hansen, E. H. Flow Injection Analysis, 2nd ed.; Wdey: New York, 1988.
4508 Analytical Chemistry, Vol. 66, No. 24, December 15, 1994
{
= 0 otherwise
(3)
then eq 2 reduces to the simple form, N-1
n = 0, ...,M - 1
(4)
k=O
The Fourier approach uses an homologous series of complete, orthogonalfunctionsas basis functions. By homologous, we mean that the series increases in complexity in some well-defined incremental manner. Hence eq 1 becomes a set of hierarchical models, and Fourier smoothing implies signal approximation by one model in this set. Since FIA signals are simple, a low-order model (A4
