Expert Systems for Environmental Applications - ACS Publications

Chapter 4

Neural Networks and Environmental Applications Joseph Schmuller

Downloaded by COLUMBIA UNIV on January 21, 2018 | http://pubs.acs.org Publication Date: July 5, 1990 | doi: 10.1021/bk-1990-0431.ch004

Expert Systems Team, CDM Federal Programs Corporation, 13135 LeeJackson Memorial Highway, Fairfax, VA 22033

Neural network models are gaining popularity in a variety of areas, and are currently getting a lot of publicity. This paper i s a tutorial introduction to these models, and is intended for people with little or no background in the f i e l d . We begin with the fundamental concept of interconnections among simple computational elements, examine a simple neural net model, and discuss training and validation of neural networks. We outline differences between neural net models and traditional expert systems, and we present ideas for using them together. Major players i n the neural net f i e l d are enumerated, as well as the roles they play. We l i s t successful applications of this technology, and we indicate several potential environmental applications.

A n e u r a l network i s a p i e c e o f hardware o r s o f t w a r e ( o r a c o m b i n a t i o n o f t h e two) t h a t s i m u l a t e s what we t h i n k we know about how t h e b r a i n works. The o p e r a t i v e word i s " t h i n k " ; t o say t h a t such models do t h i n g s e x a c t l y as t h e b r a i n does would be much t o o presumptuous a t t h i s t i m e , as t h e r e a r e s t i l l a g r e a t many m y s t e r i e s c o n c e r n i n g t h e brain. To a v o i d any i m p l i e d presumption, some r e s e a r c h e r s i n t h i s f i e l d p r e f e r terms l i k e p a r a l l e l d i s t r i b u t e d p r o c e s s o r s , c o n n e c t i o n i s t systems, o r c o l l e c t i v e d e c i s i o n c i r c u i t s . N e u r a l network models and b r a i n s c o n t a i n sets o f elements, each o f w h i c h i s c o m p u t a t i o n a l l y s i m p l e . The e l e m e n t s , c a l l e d "neurons", a r e h i g h l y i n t e r c o n n e c t e d t o one a n o t h e r ; i n t h e human b r a i n t h e r e a r e about 100 b i l l i o n neurons, and each one i s connected to about 10,000 o t h e r neurons. N e u r a l network models o f t e n c o n t a i n l a r g e numbers o f s i m u l a t e d neurons, b u t n o t as many as a r e i n t h e brain. F o r t h e remainder o f t h i s paper, we w i l l r e f e r t o s i m u l a t e d neurons as " u n i t s " . 0097-6156/90/0431-0052$06.00A) © 1990 American Chemical Society

Hushon; Expert Systems for Environmental Applications ACS Symposium Series; American Chemical Society: Washington, DC, 1990.

4.

SCHMULLER

Neural Networks and Environmental Applications53

A C l o s e r Look How does a networked arrangement of u n i t s s o l v e problems? a t a simple network.

Let's look


Layers. F i g u r e 1 shows 15 u n i t s on the l e f t c o n n e c t e d w i t h 5 u n i t s on the r i g h t . L e t ' s suppose t h a t the u n i t s on the l e f t take a p a t t e r n of s t i m u l a t i o n from the o u t s i d e w o r l d , and t h a t t h i s p a t t e r n i s then r e p r e s e n t e d i n these u n i t s . L e t ' s r e f e r t o the 15 u n i t s as the " i n p u t l a y e r " . Each o f the 5 u n i t s on the r i g h t i s c o n n e c t e d t o every u n i t i n the i n p u t l a y e r ; t h u s , the a c t i v i t y o f these u n i t s depends on the p a t t e r n o f i n p u t . W e ' l l c a l l these 5 the "output l a y e r " . S t i m u l i and Responses. Imagine t h a t each u n i t i n the i n p u t l a y e r c o r r e s p o n d s t o a c e l l i n a 5 X 3 m a t r i x , and t h a t each o f these c e l l s can have the v a l u e "1" o r "0" ( c o r r e s p o n d i n g t o "on" o r " o f f " ) ; each u n i t i n the o u t p u t l a y e r c o r r e s p o n d s t o an uppercase v e r s i o n o f one o f the f i r s t 5 l e t t e r s o f the a l p h a b e t . Any p a t t e r n on t h i s m a t r i x can be r e p r e s e n t e d by a v e c t o r o f 15 numbers, each o f w h i c h i s e i t h e r 1 o r 0. The problem t h a t t h i s network has t o s o l v e i s t h i s : given a v e c t o r o f l ' s and 0's p r e s e n t e d t o t h e i n p u t l a y e r , w h i c h o f t h e l e t t e r s A t h r o u g h Ε does the v e c t o r c o r r e s p o n d to? F i g u r e 2 i l l u s t r a t e s such a 5 X 3 m a t r i x and 5 upper-case l e t t e r s t h a t c o u l d be c o n s t r u c t e d w i t h t h i s m a t r i x . I n terms o f l ' s and 0's, the m a t r i x ' s v e c t o r f o r the l e t t e r A i s [0,1,0,1,0,1,1,1,1,1,0,1,1,0,1] We want o n l y the f i r s t o u t p u t - l a y e r u n i t (which c o r r e s p o n d s t o "A") t o f i r e i f t h i s p a t t e r n i s p r e s e n t e d . That i s , i n terms o f l ' s and 0's, we would want the o u t p u t - l a y e r ' s response v e c t o r t o be [1,0,0,0,0]. U n i t 8 i n the i n p u t l a y e r c o r r e s p o n d s t o the c e n t e r i n A, B, and E. I f t h i s u n i t i s "on", the network has e v i d e n c e t h a t the i n p u t p a t t e r n i s n o t C o r D. I f t h i s u n i t i s " o f f " , t h e network has e v i d e n c e t h a t the i n p u t p a t t e r n i s not A, B, o r E. We c a n make s i m i l a r statements about o t h e r u n i t s and t h e l e t t e r s they a r e a s s o c i a t e d w i t h and the ones they e l i m i n a t e . I f a u n i t i s a s s o c i a t e d w i t h a l e t t e r , we w i l l a t t a c h a p o s i t i v e w e i g h t t o i t s i n t e r c o n n e c t i o n w i t h the o u t p u t - l a y e r u n i t c o r r e s p o n d i n g t o the a s s o c i a t e d letter. I f a u n i t ' s f i r i n g ( o r b e i n g "on", o r h a v i n g the v a l u e "1" i n t h e input pattern) e l i m i n a t e s a l e t t e r , w e ' l l a t t a c h a negative weight t o i t s i n t e r c o n n e c t i o n w i t h the l e t t e r ' s o u t p u t - l a y e r u n i t . Each o u t p u t - l a y e r u n i t , then, has a s e t o f w e i g h t s , and each w e i g h t c o r r e s p o n d s t o a c o n n e c t i o n w i t h an i n p u t - l a y e r u n i t . Each w e i g h t i s a p o s i t i v e o r n e g a t i v e number, and each i n p u t i s e i t h e r a one o r a zero. A c t i v a t i o n o f a U n i t . What does a u n i t ( i n t h i s c a s e , an o u t p u t - l a y e r u n i t ) do w i t h these i n p u t s and w e i g h t s ? I n s i m p l e networks l i k e o u r s , a u n i t j t a k e s an i n p u t ( x ^ and m u l t i p l i e s i t by the w e i g h t o f the i n t e r c o n n e c t i o n (w^j) t h r o u g h w h i c h the i n p u t came. I t does t h i s f o r a l l i t s i n p u t s , and then adds these p r o d u c t s t o g e t h e r . That i s ,


EXPERT SYSTEMS FOR ENVIRONMENTAL APPLICATIONS


54


4. SCHMULLER

Neural Network and Environmental Applications 55 SUM x

i = 1, 2,

i W i j

15

i n w h i c h A j i s t h e a c t i v a t i o n o f o u t p u t u n i t j . T h i s summation f o r m u l a i s one k i n d o f a c t i v a t i o n f u n c t i o n - - a r u l e w h i c h t e l l s t h e u n i t what t o do w i t h i t s i n p u t s . I n some models, the a c t i v a t i o n o f a u n i t depends p a r t l y on i t s previous a c t i v a t i o n . F o r example, a model c a l l e d BSB ("Brain S t a t e i n a Box") s e t s up a c t i v a t i o n f u n c t i o n s f o r i t s u n i t s such t h a t ( w i t h i n p r e - s e t upper and lower bounds)


A j ( t + 1 ) = A j ( t ) + SUM x ^ j i n w h i c h A j ( t ) i s j ' s a c t i v a t i o n a t time t and A j ( t + 1 ) i s i t s a c t i v a t i o n a t t+1 ( 1 ) . Other models depart from s i m p l y summing t h e w e i g h t e d i n p u t s t o each u n i t . One c l a s s o f models c o n t a i n s " c o n j u n c t s " -- two o r more input units w i t h a single interconnection t o an o u t p u t u n i t . The s i g n a l s from t h e u n i t s i n a c o n j u n c t a r e m u l t i p l i e d t o g e t h e r b e f o r e they a r e m u l t i p l i e d by t h e w e i g h t o f t h e i n t e r c o n n e c t i o n , thus producing the conjunct's net input t o the output u n i t . I f a set of c o n j u n c t s feeds i n t o a u n i t , t h e i r n e t i n p u t s a r e summed t o produce that unit's a c t i v a t i o n . Output o f a U n i t . I n g e n e r a l , u n i t s i n t e r a c t by s e n d i n g o u t s i g n a l s t o o t h e r u n i t s . The s i z e o f a u n i t ' s o u t p u t s i g n a l depends on t h e u n i t ' s a c t i v a t i o n . I n a s i m p l e model, i f t h e a c t i v a t i o n exceeds some p r e - s e t " t h r e s h o l d " v a l u e , t h e u n i t " f i r e s " ( i . e., i t p r o v i d e s an o u t p u t ) ; i f n o t , i t doesn't. I n t h e s i m p l e s t c a s e , i f a u n i t f i r e s , i t s output i s 1; i f n o t , i t s o u t p u t i s 0. The r u l e w h i c h t u r n s a u n i t ' s a c t i v a t i o n i n t o i t s output i s c a l l e d t h e u n i t ' s transfer function. I n o u r l e t t e r - i d e n t i f i e r , an i n p u t - l a y e r u n i t f i r e s when i t s a s s o c i a t e d m a t r i x c e l l i s s t i m u l a t e d -- w h i c h we assume produces an a c t i v a t i o n g r e a t e r than t h e i n p u t - l a y e r u n i t ' s t h r e s h o l d . An o u t p u t l a y e r u n i t f i r e s when i t s a c t i v a t i o n exceeds i t s t h r e s h o l d . An o u t p u t - l a y e r u n i t ' s f i r i n g means t h a t t h e u n i t p r e s e n t s i t s a s s o c i a t e d l e t t e r as an i d e n t i f i c a t i o n o f the i n p u t p a t t e r n . The t h r e s h o l d arrangement can be more m a t h e m a t i c a l l y complex than i s t h e case i n o u r l e t t e r - i d e n t i f i e r . Suppose S i s t h e sum o f w e i g h t e d i n p u t s t o a u n i t j whose t h r e s h o l d i s T. Some models p o s i t a l o g i s t i c f u n c t i o n such t h a t j ' s o u t p u t i s Oj =

1 / (1

+ e"(

S

+

J

h

The arrangement c a n be s t i l l more complex. F o r example, i n a c l a s s o f n e u r a l networks c a l l e d thermodynamic models, each u n i t c a n o u t p u t 0 o r 1, and a s t o c h a s t i c f u n c t i o n o p e r a t i n g on t h e u n i t ' s i n p u t s determines t h e p r o b a b i l i t y t h a t i t s o u t p u t w i l l be 1 ( 2 ) . T r a i n i n g A Network When we f i r s t s e t up the model, how do we know t h e v a l u e s t o a s s i g n f o r the weights? O f t e n , we do n o t . We might s t a r t w i t h some


56



a r b i t r a r y v a l u e s and then " t e a c h " t h e network what t o do. F o r o u r l e t t e r - i d e n t i f i e r , we would expose t h e network t o m a t r i x p a t t e r n s w h i c h r e p r e s e n t each o f t h e f i v e uppercase l e t t e r s and n o t e t h e network's responses (the p a t t e r n s p r e s e n t e d t o t h e model a t t h i s p o i n t are c a l l e d t h e " t r a i n i n g s e t " ) . Some o f t h e responses may be erroneous a t f i r s t , and we w o u l d have t o p r o v i d e feedback t o t h e network; t h e r e s u l t o f t h e feedback i s t h e a l t e r i n g o f t h e w e i g h t s a c c o r d i n g t o some r u l e s p e c i f i e d by t h e model. One well-known p r o c e d u r e f o r p r o v i d i n g feedback and a l t e r i n g t h e w e i g h t s i s c a l l e d t h e d e l t a r u l e ( 3 ) . I t works i n t h e f o l l o w i n g way. Suppose, i n t h e i n i t i a l t r a i n i n g t r i a l , o u r w e i g h t s a r e s e t up so t h a t when our network i s p r e s e n t e d w i t h t h e a f o r e m e n t i o n e d v e c t o r f o r "A", i t responds t h a t t h e s t i m u l u s c o u l d have been e i t h e r A, B, o r E. That i s , i t s output v e c t o r i s [1,1,0,0,1] i n s t e a d of the d e s i r e d t a r g e t v e c t o r [1,0,0,0,0]. We s u b t r a c t t h e o b t a i n e d v e c t o r from t h e d e s i r e d v e c t o r by s u b t r a c t i n g c o r r e s p o n d i n g elements, y i e l d i n g t h e v e c t o r o f " d e l t a s " [0,-1,0,0,-1]. L e t us g e n e r i c a l l y l a b e l an i n p u t v e c t o r as I ( i n w h i c h "p" stands f o r " p a t t e r n " ) , an o u t p u t v e c t o r as 0 , and t h e t a r g e t v e c t o r as T . The v e c t o r o f d e l t a s i s t h e n T - 0 . F u r t h e r , l e t us l a b e l an element o f an i n p u t v e c t o r as I ( " i " denotes an i n p u t u n i t ) , an element o f an o u t p u t v e c t o r as 0 j , and an element o f t h e d e l t a s v e c t o r as Tpj-O j ( " j " denotes an o u t p u t u n i t ) . A c c o r d i n g t o t h e d e l t a r u l e , a f t e r a t r a i n i n g t r i a l t h e change (Cj^) i n t h e w e i g h t o f an i n t e r c o n n e c t i o n between i n p u t u n i t i ( l i k e our u n i t s 1-15) and o u t p u t u n i t j ( o u r u n i t s 16-20) depends on t h e activation I o f t h e i n p u t u n i t and t h e d e l t a T j - O j o f t h e o u t p u t unit: p

p

p

p

p

p

i

p

p

p

i

p

C

ji

"

r ( T

p

) I

P J " °PJ pi

i n w h i c h r i s t h e p r e - s e t l e a r n i n g r a t e o f t h e network, a number w h i c h i s t y p i c a l l y between 0 and 1. The e x a c t v a l u e we a s s i g n t h i s r a t e determines how q u i c k l y t h e network converges on i t s i d e a l s t a t e . I t s h o u l d r e f l e c t t h e degree o f " n o i s e " i n o u r t r a i n i n g p a t t e r n s . F o r example, some o f o u r t r a i n i n g "A's" might n o t be e x a c t "A's". To t h e e x t e n t t h a t they v a r y from o u r p r o t o t y p e "A", we would a s s i g n a lower v a l u e ( l i k e .1 o r .2) t o r ; i f o u r i n p u t s a r e c l o s e t o o u r p r o t o t y p e l e t t e r s , we would a s s i g n a h i g h e r v a l u e (.8 o r .9) t o r . I f we a r e n ' t sure how n o i s y our d a t a a r e , we'd p i c k an in-between v a l u e . I f t h e v a l u e we p i c k i s t o o low, t h e network w i l l need many t r a i n i n g t r i a l s to converge on i t s t a r g e t ; i f t h e v a l u e i s t o o h i g h , i t may o v e r s h o o t the t a r g e t . Suppose o u r l e a r n i n g r a t e i s .9. A c c o r d i n g t o t h e d e l t a r u l e , i n our example t h e i n t e r c o n n e c t i o n between u n i t 8 and u n i t 17 (the o u t p u t u n i t f o r "B") s h o u l d change by


4. SCHMULLER

Neural Network and Environmental Applications 57 c

17,8

"

.9(-l)(D

= -.9

as s h o u l d t h e o t h e r i n t e r c o n n e c t i o n s w h i c h caused t h e wrong o u t p u t units to f i r e . A f t e r a network has been t r a i n e d , i t s performance i s a s s e s s e d by e x p o s i n g i t t o a s e t o f s t i m u l i w h i c h resemble t h e t r a i n i n g s t i m u l i but a r e n o t i d e n t i c a l t o them. The g o a l i s f o r t h e t r a i n e d network t o c l a s s i f y t h e t e s t s t i m u l i i n t h e same way t h a t i t c l a s s i f i e s t h e t r a i n i n g s t i m u l i . The network i s j u d g e d by i t s degree o f success i n a c h i e v i n g t h i s g o a l , as a network i s u s e f u l t o t h e e x t e n t t h a t i t can g e n e r a l i z e beyond i t s t r a i n i n g .


Hidden L a y e r s To t h i s p o i n t , o u r d i s c u s s i o n has c e n t e r e d around a network w i t h j u s t an i n p u t - l a y e r and an o u t p u t l a y e r . Most networks w h i c h s o l v e problems o f p r a c t i c a l i m p o r t a n c e , however, have a t l e a s t one more l a y e r o f u n i t s between t h e i n p u t l a y e r and t h e o u t p u t l a y e r . L a y e r s between t h e i n p u t and o u t p u t a r e s a i d t o be h i d d e n . A unit i n a h i d d e n l a y e r a c t s l i k e a u n i t i n any o t h e r l a y e r ; i t t a k e s one o r more i n p u t s , and i t passes o u t p u t s t o o t h e r u n i t s . I f we were t o add a h i d d e n l a y e r t o o u r model, i t w o u l d l o o k l i k e F i g u r e 3 ( f o r t h e sake of c l a r i t y , s t r o n g l y - w e i g h t e d c o n n e c t i o n s a r e t h e o n l y ones shown). I n a t r a i n e d network, h i d d e n - l a y e r u n i t s s h o u l d c o r r e s p o n d t o component f e a t u r e s o f t h e s t i m u l i . Our l e t t e r s , f o r example, can be thought o f as b e i n g c o n s t r u c t e d from v e r t i c a l l i n e s and h o r i z o n t a l l i n e s , each o f w h i c h i s formed from s e v e r a l c e l l s i n t h e i n p u t m a t r i x : a l e f t - s i d e v e r t i c a l l i n e i s i n d i c a t e d by u n i t s 1, 4, 7, 10, and 13 b e i n g on, a c r o s s b a r by u n i t s 7, 8, and 9, a r i g h t - s i d e v e r t i c a l l i n e by u n i t s 3, 6, 9, 12, and 15, a t o p h o r i z o n t a l l i n e by 1, 2, and 3, and a bottom h o r i z o n t a l l i n e by 13, 14, and 15. A h i d d e n - l a y e r u n i t w i t h h e a v i l y - w e i g h t e d c o n n e c t i o n s t o 7, 8, and 9 would a c t as a crossbar detector. T h i s u n i t , i n t u r n , would have s t r o n g c o n n e c t i o n s w i t h t h e o u t p u t - l a y e r c e l l s w h i c h c o r r e s p o n d t o A, B, and E. One method o f t r a i n i n g a network w i t h h i d d e n l a y e r s i n v o l v e s an extension of the d e l t a r u l e c a l l e d backpropagation (4). This procedure computes w e i g h t changes f o r h i d d e n u n i t s by f i r s t f i n d i n g d i f f e r e n c e s between t h e o b s e r v e d o u t p u t s o f o u t p u t - l a y e r u n i t s and t h e d e s i r e d o u t p u t s , and t h e n p r o p a g a t i n g t h e s e d i f f e r e n c e s back t o t h e u n i t s w h i c h send o u t p u t v a l u e s t o them. Most u n i t s i n h i d d e n - l a y e r models o p e r a t e v i a complex a c t i v a t i o n f u n c t i o n s ( l i k e t h e l o g i s t i c f u n c t i o n mentioned e a r l i e r ) w h i c h a r e d i f f e r e n t i a b l e and n o n - d e c r e a s i n g . I n these models, t h e w e i g h t change f o r an o u t p u t u n i t j g i v e n an i n p u t pattern ρ i s C

PJ

-

( T

PJ - Opj'f'j^PJ'

i n w h i c h f * j ( n e t j ) i s t h e d e r i v a t i v e o f j ' s a c t i v a t i o n f u n c t i o n . The w e i g h t change f o r a h i d d e n u n i t h i s p

c

p h

- fVnetphJSUMjCpjWhj



58


INPUT LAYER

HIDDEN LAYER

OUTPUT LAYER

Stimulus Pattern

Possible Outputs

F i g u r e 3. A N e u r a l Network With a Hidden

Layer



4.

SCHMULLER


i n w h i c h the d e r i v a t i v e o f h's a c t i v a t i o n f u n c t i o n i s m u l t i p l i e d by the w e i g h t e d sum o f changes t o u n i t s j t o w h i c h h sends s i g n a l s . T h i s e q u a t i o n r e c u r s e s backward t h r o u g h a l l l a y e r s o f the network. The o v e r a l l p i c t u r e t h a t emerges, t h e n , i s a s e t o f s i g n a l s t r a n s m i t t e d f o r w a r d (from i n p u t u n i t s t h r o u g h h i d d e n u n i t s t o output u n i t s ) , and w e i g h t adjustments t r a n s m i t t e d backward (from output u n i t s t h r o u g h hidden u n i t s t o input u n i t s ) . Hidden l a y e r s are f o u n d a t i o n a l t o contemporary work on n e u r a l networks. I n some models, they a l l o w l e a r n i n g t o t a k e p l a c e i n the absence o f t r a i n i n g s e s s i o n s -- i . e . , w i t h no feedback from a p e r s o n . T h i s type o f l e a r n i n g i s called " u n s u p e r v i s e d " . One way o f a c c o m p l i s h i n g t h i s i s t o p a r t i t i o n the h i d d e n l a y e r i n t o c l u s t e r s o f mutually i n h i b i t o r y u n i t s ( 5 ) . Learning takes place v i a c o m p e t i t i o n among u n i t s i n a c l u s t e r . Each c l u s t e r e v e n t u a l l y r e c o g n i z e s a d i f f e r e n t f e a t u r e o f the i n p u t p a t t e r n . I f we add such a p a r t i t i o n e d h i d d e n l a y e r t o our t w o - l a y e r l e t t e r i d e n t i f i e r , we might f i n d t h a t i n a 2 - u n i t c l u s t e r , one u n i t l e a r n e d t o respond when a c r o s s b a r was p r e s e n t , w h i l e the o t h e r responded when a c r o s s b a r was a b s e n t . V a l i d a t i n g A N e u r a l Network. One i m p o r t a n t a s p e c t o f n e u r a l network construction i s validation. I t i s not enough t o t r a i n a network on a s e t o f s t i m u l u s p a t t e r n s and then r e p o r t the l e v e l o f a c c u r a c y w i t h w h i c h t h e network u l t i m a t e l y comes t o i d e n t i f y those p a t t e r n s . A l t h o u g h network b u i l d e r s a l s o r e p o r t r e s u l t s o f t e s t i n g s t i m u l u s p a t t e r n s not i n the t r a i n i n g s e t , and a l t h o u g h these r e s u l t s r e f l e c t the r e l i a b i l i t y o f the network, these d a t a do not show the u n d e r l y i n g reasons f o r the d e c i s i o n s t h a t the network makes. I n o t h e r words, a f t e r a network has been t r a i n e d , what e x a c t l y has i t l e a r n e d ? A network may be making c o r r e c t d e c i s i o n s by s y s t e m a t i c a l l y d e t e c t i n g f e a t u r e s of i t s i n p u t s t i m u l i , o r by some o t h e r p r o c e s s t h a t we cannot i n t e r p r e t as b e i n g r e l a t e d t o the i n p u t i n any m e a n i n g f u l way. I n the former case, we would t e n d t o b e l i e v e i n the v a l i d i t y o f the network as a model o f r e c o g n i t i o n ; i n the l a t t e r case we would n o t . F o r any g i v e n network, how can we make the d i s t i n c t i o n ? A n a l y z i n g Hidden U n i t A c t i v i t y . Elman and Z i p s e r ' s (6) study o f h i d d e n - l a y e r networks f o r speech r e c o g n i t i o n i s i n s t r u c t i v e . These i n v e s t i g a t o r s c o n s t r u c t e d b a c k p r o p a g a t i o n networks t o c l a s s i f y spoken s y l l a b l e s . Each s y l l a b l e was a c o m b i n a t i o n o f one o f t h r e e consonants and one o f t h r e e vowels. The networks c o n s i s t e d o f 320 i n p u t u n i t s , between two and s i x h i d d e n u n i t s ( i n d i f f e r e n t c o n d i t i o n s ) , and t h r e e or n i n e output u n i t s . The i n p u t s t i m u l i were s p e c t r a l r e p r e s e n t a t i o n s of the s y l l a b l e s as spoken by a male speaker. Each spectrum c o n s i s t e d of d a t a f o r 16 frequency ranges t a k e n over 20 time-segments (each segment r e p r e s e n t e d 3.2 msec); each i n p u t u n i t c o r r e s p o n d e d t o one o f the r e s u l t i n g 320 c o m b i n a t i o n s o f f r e q u e n c y range and time segment. Each output u n i t c o r r e s p o n d e d t o one o f t h e s y l l a b l e s t o be i d e n t i f i e d ; i n one c o n d i t i o n , the t a s k was t o i d e n t i f y each o f the n i n e s y l l a b l e s , i n a n o t h e r , t o i d e n t i f y each o f the t h r e e v o w e l s , and i n a n o t h e r , t o i d e n t i f y each o f t h e t h r e e consonants. One p e r s o n r e c o r d e d about 56 r e p e t i t i o n s o f each consonant-vowel c o m b i n a t i o n , h a l f o f w h i c h were used f o r t r a i n i n g t h e network, and h a l f f o r t e s t i n g . S t a r t i n g w i t h r a n d o m l y - a s s i g n e d w e i g h t s , over 100,000 t r a i n i n g t r i a l s were run, d u r i n g w h i c h a network l e a r n e d t o r e c o g n i z e i t s t r a i n i n g s e t p e r f e c t l y . When p r e s e n t e d w i t h the t e s t s e t , t h e



60


w h o l e - s y l l a b l e r e c o g n i z e r averaged 162 e r r o r s ; t h e v o w e l - r e c o g n i z e r and the consonant-recognizer averaged 1.5Z and 7.9Z errors respectively. Going beyond these performance s t a t i s t i c s , Elman and Z i p s e r reasoned t h a t t h e p o s t - t r a i n i n g p a t t e r n s o f a c t i v i t y i n t h e h i d d e n u n i t s w o u l d p r o v i d e t h e b e s t i n d i c a t i o n o f t h e l e a r n i n g t h a t took place. Using a v i s u a l r e p r e s e n t a t i o n that depicted the a c t i v i t y of each h i d d e n u n i t g i v e n each s t i m u l u s p a t t e r n , they showed t h a t h i d d e n u n i t s l e a r n e d t o o u t p u t a v a l u e o f 1 f o r some sound t y p e s , and 0 f o r others. Thus, every h i d d e n u n i t became a s s o c i a t e d w i t h a subset o f sound t y p e s . These s u b s e t s were v o w e l l i k e o r c o n s o n a n t l i k e , i n t h a t a u n i t was on o r o f f f o r a p a r t i c u l a r consonant o r v o w e l . I n t h e case of a f o u r - u n i t h i d d e n l a y e r f o r f u l l - s y l l a b l e r e c o g n i t i o n , f o r example, one u n i t l e a r n e d n o t t o f i r e when s y l l a b l e s b e g i n n i n g w i t h "b" were p r e s e n t e d , and a n o t h e r l e a r n e d n o t t o f i r e when s y l l a b l e s b e g i n n i n g w i t h "g" were p r e s e n t e d . S i m i l a r r e s u l t s were found f o r t h e o t h e r two u n i t s ' responses t o v o w e l s . These c o r r e l a t i o n s between s t i m u l i and h i d d e n u n i t a c t i v i t y a r e e v i d e n c e t h a t t h e network l e a r n e d t o respond t o m e a n i n g f u l f e a t u r e s o f t h e s t i m u l u s s e t . R e n a i s and Rohwer (7) found s i m i l a r r e s u l t s . T h e i r s t u d y , an e x a m i n a t i o n o f n e u r a l networks f o r r e c o g n i z i n g v o w e l s , showed t h a t h i d d e n u n i t s l e a r n e d t o respond s e l e c t i v e l y t o members o f t h e s t i m u l u s set. T o u r e t s k y and Pomerleau (8) examined h i d d e n - c e l l a c t i v i t y i n a b a c k p r o p a g a t i o n network w h i c h had been t r a i n e d t o c l a s s i f y computerg e n e r a t e d p i c t u r e s o f road c o n d i t i o n s and serve as a n a v i g a t o r f o r an autonomous l a n d v e h i c l e . Analogous w i t h t h e p h o n e m e - r e c o g n i t i o n r e s u l t s , t h e i r a n a l y s i s i n d i c a t e d that the twenty-nine hidden u n i t s i n t h e network had l e a r n e d t o respond t o component f e a t u r e s o f t h e d e p i c t e d roads. These s t u d i e s suggest t h a t s t u d y i n g p o s t - t r a i n i n g h i d d e n u n i t a c t i v i t y i s a v a l u a b l e t e c h n i q u e f o r i n f e r r i n g what a network has l e a r n e d . B u i l d e r s o f n e u r a l networks might do w e l l t o c o n s i d e r t h i s technique as s t a n d a r d o p e r a t i n g procedure i n e v a l u a t i n g t h e i r n e t w o r k s ' performance. S c a l i n g The T r a i n e d Network's Responses. When c o g n i t i v e p s y c h o l o g i s t s t r y t o model p e o p l e ' s r e c o g n i t i o n p r o c e s s e s f o r a p a r t i c u l a r s e t o f s t i m u l i , they study t h e p a t t e r n s o f e r r o r s t h a t p e o p l e make. F o r example, when p e o p l e i d e n t i f y b r i e f l y - p r e s e n t e d ( e . g., 20 msec) lower-case l e t t e r s o f t h e a l p h a b e t they o f t e n m i s t a k e "q" f o r "p" and "o" f o r " c " . M i s t a k e s l i k e these have been t a k e n t o i n d i c a t e t h a t people a t t e n d t o component f e a t u r e s o f t h e s t i m u l i ; t h e q-p c o n f u s i o n suggests t h a t each l e t t e r ' s d e s c e n d i n g s t r a i g h t l i n e has a t t r a c t e d a t t e n t i o n , w h i l e t h e o-c c o n f u s i o n suggests t h e same f o r l e t t e r c u r v a t u r e . These p a t t e r n s o f e r r o r s a r e t h e f o u n d a t i o n f o r models o f the r e c o g n i t i o n p r o c e s s based on a n a l y s i s o f component f e a t u r e s . A c o n f u s i o n m a t r i x i s a c o n v e n i e n t way o f summarizing e r r o r p a t t e r n s . T h i s i s a m a t r i x whose rows a r e t h e members o f a s t i m u l u s s e t and whose columns a r e t h e p o s s i b l e responses t o those s t i m u l i (which, o r d i n a r i l y , a r e t h e same o b j e c t s as t h e s t i m u l i ) ; each c e l l o f t h e m a t r i x c o n t a i n s t h e frequency w i t h w h i c h t h e column's response was g i v e n t o t h e row's s t i m u l u s ( i . e., t h e " c o n f u s a b i l i t y " o f t h e presented stimulus and t h e r e s u l t i n g response). With perfect r e c o g n i t i o n , o n l y t h e c e l l s on t h e main d i a g o n a l a r e f i l l e d , i f t h e



4. SCHMULLER

Neural Network and Environmental Applications 61

p o t e n t i a l responses a r e l i s t e d i n t h e same o r d e r as t h e s t i m u l u s objects. Two statistical methods used i n behavioral science, m u l t i d i m e n s i o n a l s c a l i n g (9) and h i e r a r c h i c a l c l u s t e r i n g ( 1 0 ) , a r e used t o a n a l y z e the e r r o r p a t t e r n s r e p r e s e n t e d i n c o n f u s i o n m a t r i c e s . Together, they can be the b a s i s f o r a t e c h n i q u e w h i c h examines t h e n a t u r e o f a n e u r a l network's d e c i s i o n p r o c e s s . I n c o n t r a s t w i t h r e g r e s s i o n - b a s e d t e c h n i q u e s such as f a c t o r a n a l y s i s , m u l t i d i m e n s i o n a l s c a l i n g i s based on t h e metaphor o f " d i s t a n c e s " between p a i r s o f o b j e c t s . That i s , two h i g h l y c o n f u s a b l e o b j e c t s are c h a r a c t e r i z e d as b e i n g c l o s e t o one a n o t h e r i n space, two n o n - c o n f u s a b l e o b j e c t s as b e i n g f a r a p a r t . A c o n f u s i o n m a t r i x i s thus viewed as a m a t r i x o f i n t e r o b j e c t d i s t a n c e s , and m u l t i d i m e n s i o n a l s c a l i n g p r o c e d u r e s map the o r d i n a l i t i e s o f these d i s t a n c e s i n t o spaces of v a r y i n g numbers o f d i m e n s i o n s . Each space i s a s s o c i a t e d w i t h an e r r o r term, whose magnitude i s i n v e r s e l y r e l a t e d t o the number o f d i m e n s i o n s . When t h e number o f dimensions i s found such t h a t a d d i t i o n a l dimensions r e s u l t i n no a p p r e c i a b l e r e d u c t i o n o f the e r r o r term, the c o r r e s p o n d i n g space i s t a k e n t o be the b e s t r e p r e s e n t a t i o n o f the c o n f u s i o n m a t r i x . The l o c a t i o n s o f the s t i m u l u s o b j e c t s i n t h i s space enable an e x p e r i e n c e d a n a l y s t t o a t t a c h l a b e l s t o t h e dimensions and t o thus make i n f e r e n c e s about t h e n a t u r e o f t h e underlying r e c o g n i t i o n process. H i e r a r c h i c a l c l u s t e r i n g t r e a t s h i g h l y confusable p a i r s of objects as b e l o n g i n g t o a c l u s t e r . When an o b j e c t from one c l u s t e r i s c o n f u s a b l e w i t h an o b j e c t from a n o t h e r , the two c l u s t e r s are j o i n e d to form a l a r g e r c l u s t e r . I t e r a t i v e a p p l i c a t i o n s o f t h i s j o i n i n g process result i n a tree-like hierarchy of clusters called a dendogram. Dendograms have been used t o d e l i n e a t e the c l u s t e r i n g among o b j e c t s w i t h i n m u l t i d i m e n s i o n a l spaces (11), i n o r d e r t o f a c i l i t a t e explanation. A n e u r a l network's responses t o i t s s t i m u l i can be c a s t i n t o a c o n f u s i o n m a t r i x , so t h a t these t e c h n i q u e s c o u l d be a p p l i e d , r e s u l t i n g i n i n s i g h t s i n t o a n e u r a l network's l e a r n i n g . A f t e r h i g h l e v e l s o f a c c u r a c y have been a t t a i n e d as a r e s u l t o f many t r a i n i n g t r i a l s , a c o n f u s i o n m a t r i x w i l l have l i t t l e use; the s c a l i n g methods c o u l d be used throughout t r a i n i n g , however, t o show how l e a r n i n g e v o l v e s . On the o t h e r hand, h i g h l e v e l s o f a c c u r a c y on the t r a i n i n g s e t do not n e c e s s a r i l y i m p l y h i g h l e v e l s on t h e t e s t s e t , as S t e n t i f o r d and Hemmings found i n t h e i r study o f word r e c o g n i t i o n (12); t h e i r p o s t t r a i n i n g t e s t r e s u l t s o f 49Z-58Z a c c u r a c y suggest t h a t a c o n f u s i o n m a t r i x and subsequent a p p l i c a t i o n o f s c a l i n g t e c h n i q u e s would have been u s e f u l i n t h e i r work. Sawai, W a i b e l , M i y a t a k e and Shikano (13) r e p o r t e d such a m a t r i x , b u t d i d not use i t f o r the a p p l i c a t i o n o f s c a l i n g techniques. V a l i d a t i o n : G e n e r a l Remarks. W i t h the i n c r e a s i n g use o f m u l t i - l a y e r e d n e u r a l networks f o r s o l v i n g r e a l - w o r l d problems, u s e r s o f t h e t e c h n o l o g y w i l l become i n c r e a s i n g l y concerned about how and why a network a r r i v e d a t a d e c i s i o n . U n f o r t u n a t e l y , n e u r a l network d e s i g n e r s have t y p i c a l l y p a i d l i t t l e a t t e n t i o n t o t h e i m p l i c a t i o n s o f n o t validating t h e i r networks. I n a recent conference on speech r e c o g n i t i o n ( 1 4 ) , f o r example, over two dozen papers i n v o l v e d n e u r a l networks, but the p r e v i o u s l y - c i t e d paper o f Renais and Rowhrer was the


62


o n l y one w h i c h r e p o r t e d h i d d e n u n i t a c t i v i t y p a t t e r n s , and the paper by Sawai e t al» was the o n l y one w h i c h p r e s e n t e d a c o n f u s i o n m a t r i x .


N e u r a l Networks and E x p e r t Systems Because n e u r a l networks and e x p e r t systems are b o t h p r o b l e m - s o l v i n g d e v i c e s , they are o f t e n compared. There are s e v e r a l obvious differences. F i r s t , the knowledge i n a n e u r a l network i s r e p r e s e n t e d not by e x p l i c i t l y - s t a t e d h e u r i s t i c s (as i n e x p e r t s y s t e m s ) , but by the p a t t e r n o f n u m e r i c a l v a l u e s of the i n t e r c o n n e c t i o n s between p r o c e s s i n g elements. As these v a l u e s change, the knowledge r e p r e s e n t a t i o n changes. Thus, you c a n ' t p o i n t t o a " p i e c e o f knowledge" i n a n e u r a l net i n the same way t h a t you can p o i n t t o a p r o d u c t i o n r u l e i n a knowledge base. T h i s l e a d s d i r e c t l y t o another d i s t i n c t i o n : an e x p e r t system can e x p l a i n i t s r e a s o n i n g t o a u s e r , w h i l e a n e u r a l net cannot. A n o t h e r d i f f e r e n c e i s the s c e n a r i o i n v o l v e d i n b u i l d i n g each type of d e v i c e : an e x p e r t system comes i n t o b e i n g as knowledge i s g a i n e d from a human e x p e r t through r e p e a t e d knowledge a c q u i s i t i o n s e s s i o n s ; a n e u r a l network, on the o t h e r hand, l e a r n s t o c l a s s i f y p a t t e r n s , e i t h e r by i t s e l f o r by o b t a i n i n g feedback from a p e r s o n ( o r from a computer program w h i c h g e n e r a t e s the p a t t e r n s i n the t r a i n i n g s e t , p r e s e n t s them, and then checks the n e u r a l n e t ' s responses t o them). A s i d e from these d i f f e r e n c e s , s e v e r a l ways suggest themselves f o r u s i n g these d e v i c e s t o g e t h e r . One f r e q u e n t l y - m e n t i o n e d i d e a i s the i n t e g r a t e d i n t e l l i g e n t system, i n w h i c h a n e u r a l network's output (an i d e n t i f e d p a t t e r n ) i s p r e s e n t e d as i n p u t t o an e x p e r t system f o r further action. A n o t h e r type o f synergy may r e s u l t from n e u r a l networks coming i n t o i n c r e a s i n g use as a p p l i e d p r o b l e m - s o l v e r s . Some people w i l l become e x p e r t s a t d e s i g n i n g the r i g h t type o f network f o r a p a r t i c u l a r a p p l i c a t i o n ; t h i s e x p e r t i s e c o u l d be c a p t u r e d i n an e x p e r t system w h i c h , g i v e n the c h a r a c t e r i s t i c s of a problem, c o u l d a u t o m a t i c a l l y d e s i g n a n e u r a l network t o s o l v e i t . Once a network has been s e t up, an e x p e r t system c o u l d be the b a s i s f o r an i n t e l l i g e n t u s e r - i n t e r f a c e between a p e r s o n and a n e u r a l network. Such an i n t e r f a c e c o u l d h e l p the p e r s o n f o r m u l a t e and i n p u t the t r a i n i n g s e t and the t e s t s e t . Major P l a y e r s At p r e s e n t , many people from i n d u s t r y and from the academic w o r l d are w o r k i n g on n e u r a l networks. This s e c t i o n i s a b r i e f representative l o o k a t some of t h i s work, not an e x h a u s t i v e l i s t o f a l l r e s e a r c h e r s and groups. In the academic w o r l d , Rumelhart and h i s c o l l e a g u e s and s t u d e n t s have c r e a t e d an i m p r e s s i v e s e t o f models, and they have used these models t o study a wide range of c o g n i t i v e p r o c e s s e s ( 1 5 ) . Kohonen (16) has done some of the p i o n e e r i n g work i n systems w h i c h e x h i b i t u n s u p e r v i s e d l e a r n i n g . G r o s s b e r g (17) has f o r m u l a t e d a s e t o f n e u r a l network p r i n c i p l e s w h i c h model phenomena of l e a r n i n g , c o g n i t i o n , motor c o n t r o l , p s y c h o p h y s i o l o g y , and anatomy. H o p f i e l d (18) has shown how a s i n g l e - l a y e r e d network ( i n w h i c h a l l u n i t s are connected t o one another) can s t o r e p a t t e r n s a f t e r they have been p r e s e n t e d , and s u b s e q u e n t l y use these p a t t e r n s t o i d e n t i f y n e w l y - p r e s e n t e d ones.


4.

SCHMULLER



I n d u s t r y i s moving on a number o f f r o n t s . S e v e r a l companies manufacture s o f t w a r e s h e l l s f o r c o n s t r u c t i n g n e u r a l n e t w o r k s . Others b u i l d n e u r a l n e t hardware. S t i l l o t h e r s p r o v i d e c o n s u l t i n g and training. Software. N e u r a l network s h e l l s u s u a l l y c o n t a i n g r a p h i c s c a p a b i l i t i e s for i l l u s t r a t i n g u n i t a c t i v i t y . I n general, t h e i r cost i s d i r e c t l y r e l a t e d t o t h e number o f d i f f e r e n t types o f network models they support. C a l i f o r n i a S c i e n t i f i c S o f t w a r e ' s Brainmaker i s a l o w - c o s t MS/DOSbased program f o r c o n s t r u c t i n g m u l t i - l a y e r b a c k p r o p a g a t i o n networks based on s e v e r a l k i n d s o f t r a n s f e r f u n c t i o n s . I t comes w i t h a s e t o f t r a i n e d networks whose t a s k s range from shape r e c o g n i t i o n t o t e x t - t o speech c o n v e r s i o n . SAIC's n e u r a l network s o f t w a r e p r o d u c t s a r e more e x p e n s i v e : ANSim and S h e l l s a r e environments f o r i m p l e m e n t i n g f r e q u e n t l y - u s e d n e u r a l net models; ANSpec i s a language f o r s p e c i f y i n g and d e v e l o p i n g new models. Each package runs on a PC/AT o r c o m p a t i b l e , o r on an AT enhanced w i t h SAIC's D e l t a a c c e l e r a t o r c a r d . Neuralware I n c . ' s NeuralWorks P r o f e s s i o n a l I I i s a PC-based package f o r b u i l d i n g networks based on a wide range o f l e a r n i n g r u l e s and t h r e s h o l d f u n c t i o n s . The i n p u t data f o r these networks can be k e p t i n f i l e s whose formats a r e c o m p a t i b l e w i t h a number o f p o p u l a r s o f t w a r e packages. N e s t o r ' s NDS i s a h i g h - e n d development system f o r PC/AT's and f o r A p o l l o and Sun w o r k s t a t i o n s . U n l i k e t h e o t h e r n e u r a l n e t s h e l l s , NDS i s based on a p r o p i e t a r y model on w h i c h N e s t o r h o l d s a p a t e n t . N e s t o r c l a i m s t h a t networks based on i t s model can be t r a i n e d s i g n i f i c a n t l y f a s t e r than models based on b a c k p r o p a g a t i o n . Hardware• Computers b u i l t t o work l i k e n e u r a l n e t s a r e c a l l e d " p a r a l l e l p r o c e s s o r s " . A p a r a l l e l p r o c e s s o r uses a l a r g e number o f s m a l l , i n t e r c o n n e c t e d p r o c e s s i n g u n i t s r a t h e r t h a n a s i n g l e CPU. Prominent among these i s T h i n k i n g Machines C o r p o r a t i o n ' s "Connection Machine", w h i c h c a n r e a l i z e a v a r i e t y o f n e u r a l n e t models. I t i s programmable i n LISP, and i s w e l l - s u i t e d t o database t a s k s . HechtN i e l s e n produces the ANZA b o a r d , a c o - p r o c e s s o r w h i c h a l l o w s t h e PC/AT to emulate a p a r a l l e l p r o c e s s o r . C o n s u l t i n g . S e v e r a l f i r m s s u p p l y c o n s u l t i n g s e r v i c e s and r e p o r t s i n the a r e a o f n e u r a l n e t w o r k s . Perhaps t h e b e s t known o f these i s A d a p t i c s , whose p r e s i d e n t , Maureen C a u d i l l , has w r i t t e n a p o p u l a r series o f introductory a r t i c l e s beginning w i t h (19). Adaptics also p r o v i d e s t r a i n i n g f o r p u r c h a s e r s o f SAIC's n e u r a l n e t p r o d u c t s . New S c i e n c e A s s o c i a t e s has produced two u s e f u l r e p o r t s o r i e n t e d toward commercial a p p l i c a t i o n s (20,21). Some S u c c e s s f u l N e u r a l Network A p p l i c a t i o n s N e u r a l network models have been implemented f o r s o l v i n g r e a l - w o r l d problems i n a number o f a r e a s . A g a i n , t h i s i s n o t an e x h a u s t i v e l i s t , but a r e p r e s e n t a t i v e sampling. SAIC's SNOOPE (22) used b a c k p r o p a g a t i o n t o l e a r n t o detect



64


p l a s t i c e x p l o s i v e s i n luggage and c a r g o . I t was s u c c e s s f u l l y t e s t e d on 40,000 p i e c e s o f luggage i n June 1988 a t the Los A n g e l e s and San F r a n c i s c o I n t e r n a t i o n a l A i r p o r t s . I t can c o n t i n u o u s l y p r o c e s s 10 bags a minute, and i t d e c i d e s whether o r n o t a bag c o n t a i n s a t h r e a t by the time the bag l e a v e s the system. N e s t o r has developed the Mortgage O r i g i n a t i o n U n d e r w r i t e r , w h i c h assesses p o t e n t i a l borrowers. The i n p u t t o the system c o n s i s t s o f d a t a on the borrower (e. g., c r e d i t r a t i n g , number o f dependents, number o f y e a r s employed, c u r r e n t income), the mortgage ( l o a n - t o - v a l u e r a t i o , type o f mortgage, income-to-mortgage r a t i o ) , and the p r o p e r t y (age, number o f u n i t s , a p p r a i s e d v a l u e ) . The network can be c o n f i g u r e d i n one o f t h r e e r i s k c l a s s i f i c a t i o n modes, depending on the a c c e p t a b i l i t y o f an e r r o r . I t was t r a i n e d on p o o l o f mortgage a p p l i c a t i o n s , and i t shows a degree o f agreement w i t h a human underwriter. Sejnowski and Rosenberg's (23) NETtalk, a three-layer b a c k p r o p a g a t i o n network, l e a r n e d t o s y n t h e s i z e speech from E n g l i s h t e x t . A f t e r t r a i n i n g , i t c o u l d t u r n t e x t i n p u t i n t o phonemic r e p r e s e n t a t i o n s w h i c h a computer c o n v e r t e d t o sound. N E T t a l k was t r a i n e d on a f i r s t grade r e a d i n g t e x t , and on randomly-ordered words from a d i c t i o n a r y . The i n p u t t o the network was a "window" o f seven c h a r a c t e r s ; N E T t a l k ' s o u t p u t was a p h o n e t i c symbol f o r the c e n t e r c h a r a c t e r , the c o n t e x t f o r w h i c h was p r o v i d e d by the o t h e r s i x l e t t e r s . A f t e r each t r a i n i n g t r i a l , the window advanced one c h a r a c t e r p o s i t i o n , and N E T t a l k p r o v i d e d a p h o n e t i c symbol f o r the new c e n t e r c h a r a c t e r . N E T t a l k a t t a i n e d o v e r 90Z a c c u r a c y a f t e r 5 passes t h r o u g h a t r a i n i n g s e t o f 1000 words. A f t e r the f i r s t few t r a i n i n g s e s s i o n s , N E T t a l k ' s o u t p u t sounded l i k e b a b b l e , p r o g r e s s e d t h r o u g h pseudowords, and began t o be u n d e r s t a n d a b l e by about the t e n t h pass t h r o u g h the training set. Speech r e c o g n i t i o n has p r o v e n t o be a p a r t i c u l a r l y f r u i t f u l f i e l d f o r n e u r a l network a p p l i c a t i o n s . W h i l e the p r e v i o u s l y - c i t e d study by Elman and Ζ i p s e r and the s t u d y by R e n a i s and Rowhrer showed t h a t networks c o u l d l e a r n t o c l a s s i f y i s o l a t e d speech sounds, a number o f o t h e r i n v e s t i g a t o r s have d e v e l o p e d networks w h i c h l e a r n t o r e c o g n i z e whole words. Krause and H a c k b a r t h (2Λ), showed t h a t a network c o u l d a c c u r a t e l y r e c o g n i z e whole words from a l i m i t e d German v o c a b u l a r y . D e m i c h e l i s , F i s s o r e , L a f a c e , M i c c a , and P i c c o l o (25) c o n s t r u c t e d a network t h a t r e c o g n i z e d I t a l i a n d i g i t - w o r d s , and Sakoe, I s o t a n i , Y o s h i d a , I s o , and Watanabe (26) d i d the same f o r Japanese d i g i t - w o r d s . I n a l l t h r e e s t u d i e s each h i d d e n u n i t was c o n n e c t e d t o a s u b s e t o f i n p u t u n i t s , r a t h e r t o t h a n the e n t i r e i n p u t l a y e r , as i n the phoneme work. T h i s v a r i a t i o n shows p r o m i s e , and may one day l e a d t o networks w h i c h r e c o g n i z e l a r g e v o c a b u l a r i e s and w h i c h c o u l d be e n g i n e e r e d i n t o commercial a p p l i c a t i o n s . A r e c e n t survey (_27) touched on s e v e r a l o t h e r a p p l i c a t i o n s : AIWARE has b u i l t a system w h i c h t r o u b l e s h o o t s g r i n d i n g o p e r a t i o n s i n a f a c t o r y ; G l o b a l H o l o n e t i c s ' LIGHTWARE performs q u a l i t y c o n t r o l on an assembly l i n e ; Widrow has d e v e l o p e d a n e u r a l network w h i c h e l i m i n a t e s echoes i n t e l e p h o n e l i n e s , and i s used i n modems and o t h e r signaling devices; C a r l e t o n U n i v e r s i t y ' s Neuroplanner i s a set of networks w h i c h e n a b l e s a r o b o t t o n a v i g a t e i t s workspace.


4.

SCHMULLER

Neural Networks and Environmental Applications


Some P o s s i b l e E n v i r o n m e n t a l A p p l i c a t i o n s Decision-makers who d e a l w i t h problems o f the environment t y p i c a l l y use l a r g e amounts o f d a t a from d i v e r s e f i e l d s i n o r d e r t o make c o n c l u s i o n s . N e u r a l networks c o u l d h e l p by d i s c o v e r i n g p a t t e r n s i n the d a t a and making recommendations. One p o t e n t i a l a p p l i c a t i o n i s t h e use o f n e u r a l networks t o f a c i l i t a t e d e c i s i o n s about hazardous waste s i t e s . These s i t e s generate a g r e a t d e a l o f d a t a , i n w h i c h p a t t e r n s a r e i n h e r e n t . S i t e s t h a t once produced b a t t e r i e s , f o r example, w i l l t y p i c a l l y show a g r e a t d e a l o f cadmium i n the s o i l ; t h i s f i n d i n g u s u a l l y l e a d s t o a d e c i s i o n about a p a r t i c u l a r form o f r e m e d i a t i o n . A network's i n p u t l a y e r c o u l d r e p r e s e n t c h a r a c t e r i s t i c s o f hazardous s i t e s (such as type o f s i t e , volume o f c o n t a m i n a t i o n , type o f c o n t a m i n a n t s , c o n t a m i n a t e d media, e t c . ) , and i t s output u n i t s c o u l d c o r r e s p o n d t o p o s s i b l e d e c i s i o n s r e g a r d i n g methods o f c l e a n u p . Such a network c o u l d be t r a i n e d and t e s t e d on RODs (Records o f D e c i s i o n ) t o e s t a b l i s h the a p p r o p r i a t e r e l a t i o n s h i p s and a s s e s s the network's a c c u r a c y . I n the same v e i n , a network c o u l d be used f o r e s t i m a t e s o f l e v e l of-effort and financing f o r the RI/FS (Remedial I n v e s t i g a t i o n / F e a s i b i l i t y S t u d y ) , the i n i t i a l stage o f hazardous waste c l e a n u p . A g a i n , u s i n g s i t e c h a r a c t e r i s t i c s as i n p u t and h i s t o r i c a l d a t a f o r t r a i n i n g and t e s t i n g , a network c o u l d l e a r n t o a r r i v e a t r e l i a b l e , c o n s i s t e n t e s t i m a t e s , thus f a c i l i t a t i n g what has u s u a l l y been a c o s t l y and time-consuming b u d g e t i n g p r o c e s s . Two other p a t t e r n - r e c o g n i t i o n based applications suggest themselves: (a) a n a l y s i s o f s o i l and l i q u i d samples, and (b) g e o p h y s i c a l e x p l o r a t i o n f o r groundwater. S o i l and L i q u i d A n a l y s i s . One t e c h n i q u e f o r a n a l y z i n g s o i l s and l i q u i d s combines gas chromatography w i t h mass s p e c t r o m e t r y . I n t h i s p r o c e d u r e , gas chromatography i s used t o s e p a r a t e and i o n i z e t h e components o f a s o i l o r l i q u i d m i x t u r e w h i c h has been c o n v e r t e d t o a gas. The i o n i z e d components a r e t h e n passed t o and t h r o u g h a mass s p e c t r o m e t e r , whose mass a n a l y z e r s o r t s the i o n s i n t o beams o f the same mass-to-charge r a t i o . The s p e c t r o m e t e r ' s d e t e c t i o n system d e t e c t s these mass-analyzed i o n s e i t h e r p h o t o g r a p h i c a l l y o r e l e c t r o n i c a l l y , and t h e s p e c t r o m e t e r ' s r e c o r d e r u l t i m a t e l y produces a f r e q u e n c y d i s t r i b u t i o n o f mass-to-charge r a t i o s f o r each component. This d i s t r i b u t i o n , c a l l e d a " f i n g e r p r i n t " , i s o f t e n i d e n t i f i e d by matching i t a g a i n s t a computerized l i b r a r y o f t y p i c a l f i n g e r p r i n t s f o r substances. N e u r a l networks r e p r e s e n t an a l t e r n a t i v e way o f i d e n t i f y i n g t h e s e f i n g e r p r i n t s . The i n p u t l a y e r w o u l d r e p r e s e n t s e v e r a l ranges o f f r e q u e n c i e s f o r each o f a wide range o f mass-tocharge r a t i o s . The o u t p u t l a y e r w o u l d have one u n i t f o r each p o s s i b l e substance t o be i d e n t i f i e d . E x p l o r a t i o n f o r Groundwater. S u r f a c e and s u b s u r f a c e methods f o r g e o p h y s i c a l e x p l o r a t i o n f o r groundwater r e s u l t i n p a t t e r n s o f d a t a whose i n t e r p r e t a t i o n r e q u i r e s t r a i n i n g and e x p e r i e n c e . One s u r f a c e method, s e i s m i c r e f r a c t i o n , t a k e s advantage o f t h e f a c t t h a t an e l a s t i c wave's v e l o c i t y t h r o u g h e a r t h m a t e r i a l v a r i e s w i t h the d e n s i t y of the m a t e r i a l . When an e l a s t i c wave c r o s s e s o v e r a g e o l o g i c boundary between two f o r m a t i o n s w i t h d i f f e r e n t e l a s t i c p r o p e r t i e s , the wave's p a t h i s r e f r a c t e d . I n t h i s type o f g e o p h y s i c a l e x p l o r a t i o n ,


65


66


e l a s t i c waves are g e n e r a t e d by a s m a l l e x p l o s i o n (or sometimes by the blow o f a hammer) a t the s u r f a c e . A s e t o f r e c e i v e r s c a l l e d geophones i s s i t u a t e d i n a l i n e r a d i a t i n g outward from the o r i g i n a t i o n p o i n t o f the waves. The waves f o l l o w t h r e e types o f paths t o the geophones - d i r e c t ( a l o n g the s u r f a c e ) , r e f r a c t e d , and r e f l e c t e d . The time f o r a wave t o reach a geophone w i l l depend on the p a t h i t t a k e s and the d e n s i t y of the m a t e r i a l . The t i m e - d i s t a n c e r e l a t i o n s h i p s w h i c h form the b a s i s of the d a t a a n a l y s i s are o f t e n c o m p l i c a t e d by the presence of s e v e r a l d i s t i n c t l a y e r s o f sediment. To f a c i l i t a t e the a n a l y s i s , a n e u r a l network c o u l d be t a u g h t t o c l a s s i f y t y p i c a l p a t t e r n s o f wave a r r i v a l s . The i n p u t l a y e r w o u l d r e p r e s e n t a s e t o f wave a m p l i t u d e s over time and d i s t a n c e ; an i n p u t u n i t would f i r e o n l y i f i t s a m p l i t u d e - t i m e - d i s t a n c e c o m b i n a t i o n was r e p r e s e n t e d i n the d a t a . A n o t h e r s u r f a c e e x p l o r a t i o n method i s based on electrical r e s i s t i v i t y of a g e o l o g i c a l f o r m a t i o n . I n a s o i l o r rock t h a t has been s a t u r a t e d w i t h f l u i d , the r e s i s t i v i t y depends m a i n l y on the p o r o s i t y and the d e n s i t y o f the m a t e r i a l and on the s a l i n i t y o f the saturating f l u i d . I n an e l e c t r i c a l r e s i s t i v i t y s u r v e y , two c u r r e n t e l e c t r o d e s pass an e l e c t r i c c u r r e n t i n t o the ground, and the p o t e n t i a l drop i s measured a c r o s s a p a i r o f p o t e n t i a l e l e c t r o d e s ; the s p a c i n g between the c u r r e n t e l e c t r o d e s d e t e r m i n e s the depth o f p e n e t r a t i o n . The r e s i s t i v i t y i s c a l c u l a t e d from the measured p o t e n t i a l drop, the a p p l i e d c u r r e n t , and the e l e c t r o d e s p a c i n g ; as r e s i s t i v i t y v a l u e s change ( e i t h e r w i t h i n c r e a s i n g depth i n one l o c a t i o n , o r a t one depth over many l o c a t i o n s ) , they i n d i c a t e change i n s u b s u r f a c e c o n d i t i o n s . R e s i s t i v i t y i s p l o t t e d a g a i n s t e l e c t r o d e s p a c i n g , and the p l o t i s compared against published plots to provide stratigraphie i n t e r p r e t a t i o n . A n e u r a l network c o u l d be t r a i n e d on the p u b l i s h e d r e s i s t i v i t y p l o t s , and then be used t o i n t e r p r e t the o b t a i n e d p l o t s from r e s i s t i v i t y s u r v e y s . The i n p u t l a y e r would r e p r e s e n t a s e t o f r e s i s t i v i t y ranges f o r d i f f e r e n t e l e c t r o d e s p a c i n g s , and the o u t p u t u n i t s would c o r r e s p o n d t o the p o s s i b l e s t r a t i g r a p h i e i n t e r p r e t a t i o n s . Many s u b s u r f a c e methods are based on b o r e h o l e g e o p h y s i c s -- a s e t of t e c h n i q u e s i n w h i c h a s e n s i n g d e v i c e i s lowered i n t o a h o l e t o g a t h e r d a t a w h i c h are then i n t e r p r e t e d i n terms of the c h a r a c t e r i s t i c s of the g e o l o g i c f o r m a t i o n s and the f l u i d s they c o n t a i n . One f r e q u e n t l y - u s e d s e n s i n g d e v i c e u t i l i z e s an e l e c t r o d e dropped i n t o a b o r e h o l e , one a t the s u r f a c e , and a source o f c u r r e n t . Two d a t a r e c o r d s are g a t h e r e d -- (a) the p o t e n t i a l d i f f e r e n c e ( v s . depth) between the b o r e h o l e e l e c t r o d e and the s u r f a c e e l e c t r o d e w i t h the c u r r e n t source t u r n e d o f f , and (b) r e s i s t i v i t y v s . depth f o r a g i v e n c u r r e n t s t r e n g t h . The s p i k e s c o n t a i n e d i n these r e c o r d s c o n s t i t u t e an i d e a l type o f d a t a f o r i n t e r p r e t a t i o n by a n e u r a l network. I n e i t h e r case, the i n p u t l a y e r would r e p r e s e n t ranges of s p i k e amplitudes over a range of d e p t h s . The network c o u l d be t r a i n e d and t e s t e d on p u b l i s h e d d a t a , and used t o i n t e r p r e t the s p i k e p a t t e r n s i n the o b t a i n e d r e c o r d s . Environmental A p p l i c a t i o n s ; C o n c l u s i o n . N e u r a l networks are not without t h e i r d e f i c i e n c i e s . To a t t a i n s u i t a b l y h i g h l e v e l s of a c c u r a c y , they r e q u i r e a g r e a t d e a l o f t r a i n i n g and computational r e s o u r c e s . A l s o , because c u r r e n t i n t e r e s t i n n e u r a l networks i s r e l a t i v e l y new, the optimum n e u r a l net a r c h i t e c t u r e i s not y e t known f o r every type of problem; i n d e e d , f i n d i n g the i d e a l a r c h i t e c t u r e s f o r various problem-classes is a continuing research area (28) .


4.

SCHMULLER


N e v e r t h e l e s s , as o r g a n i z a t i o n s concerned w i t h the environment come t o r e l y on i n c r e a s i n g l y l a r g e d a t a s e t s (and on the a u t o m a t i o n o f t h e management o f these data s e t s ) , they are l i k e l y t o t u r n t o n e u r a l networks t o h e l p use these d a t a t o r e a c h d e c i s i o n s .


Literature Cited 1.

Anderson, J. A. In Basic Processes in Reading, Perception and Comprehension; LaBerge, D. and Samuels, S. J., Eds.; Erlbaum: Hillsdale, NJ, 1977, pp 27-90.

2.

Rumelhart, D. E.; Hinton, G. E.; McClelland, J. L. In Parallel Distributed Processing; Rumelhart, D. E. and McClelland, J. L., Eds.; MIT Press: Cambridge, MA, 1987; Vol. 1, pp 45-76.

3.

Widrow, G.; Hoff, M. E.; Western Electronic Show and Convention, Convention Record, 1960, Pt. 4, pp 96-104.

4.

Jones, W. P.; Hoskins, J. Byte 1987, 12 (10), pp 155-162.

5.

Rumelhart, D. E.; Zipser, D. In Parallel Distributed Processing; Rumelhart, D. E. and McClelland, J. L., Eds.; MIT Press: Cambridge, MA, 1987; Vol. 1, pp 151-193.

6.

Elman, J. L.; Zipser, D. J. Acoust. Soc. Am. 1988, 83 (4), pp 1615-1626.

7.

Renals, S.; Rohwer, R. Proc Int. Conf. on Acoustics, Speech, and Signal Processing, 1989, pp 413-416.

8.

Touretsky, D.; Pomerleau, D. Byte 1989, 14(8), pp 227-233.

9.

Kruskal, J. B. Psychometrika 1964 29, pp

10.

Johnson, S. C. Psychometrika 1967 32, pp 241-254.

11.

Schmuller, J. Brain and Language 1979 8, pp 263-274.

12.

Stentiford, F.; Hemmings, R. Proc Int. Conf. on Acoustics, Speech, and Signal Processing, 1989, pp 310-313.

13.

Sawai, H.; Waibel, Α.; Miyatake, M.; Shikano, K. Proc Int Conf on Acoustics, Speech, and Signal Processing, 1989, pp 25-28.

14.

Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, IEEE: Glasgow, Scotland, 1989.

1-27.

15. Rumelhart D. E.; McClelland, J . L. Parallel Distributed Processing; MIT Press: Cambridge, MA, 1986; Vols. 1 and 2. 16. Kohonen, T. Self Organization and Associative Memory; SpringerVerlag: New York, 1988.


68


17. Grossberg, S. Studies of Mind and Brain; Reidel: Holland, 1982.

Dodrecht,

18. Hopfield, J. J. Proc. Natl. Acad. Sci., USA, 1982, pp. 2554-2558. 19. Caudill, M. AI Expert 1987, 2 (12), pp. 46-52. 20. Industry Report: A r t i f i c i a l Intelligence Research; New Science Associates: South Norwalk, CT, 1988(Spring). 21. Research Highlights: A r t i f i c i a l Intelligence Research; New Science Associates: South Norwalk, CT, 1989(July).


22. Obermeier, K. K.; Barron, J. J. Byte 1989, 14 (8), pp. 217-224. 23. Sejnowski, T. J.; Rosenberg, C. R. JHU/EECS-86/01, School of Electrical Engineering and Computer Science, Johns Hopkins University, 1986. 24. Krause, Α.; Hackbarth, H. Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, 1989, pp 21-24. 25. Dimichelis, E.; Fissore, L.; LaFace, P.; Micca, G.; Piccolo, E. Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, 1989, pp 314-317. 26. Sakoe, H.; Isotani, R.; Yoshida, K.; Iso, Κ. I.; Watanabe, T. Proc. Int. Conf. on Acoustics, Speech, and Signal Processing, 1989, pp 29-33. 27. Thompson, D.; Bailey, D., Feinstein, J. PC AI 1989, 3(2), pp. 5658. 28. Hinton, G. E. CMU-CS-87-115, Computer Science Department, Carnegie-Mellon University, 1987. RECEIVED

April 23, 1990


Expert Systems for Environmental Applications - ACS Publications

Recommend Documents