Verification and Validation of Environmental Expert Systems

Chapter 3

Verification and Validation of Environmental Expert Systems Downloaded by UNIV OF CALIFORNIA SANTA BARBARA on April 3, 2016 | http://pubs.acs.org Publication Date: July 5, 1990 | doi: 10.1021/bk-1990-0431.ch003

Mark Stunder GEOMET Technologies, Inc., 20251 Century Boulevard, Germantown, MD 20874

This chapter provides an overview of expert-system verification and validation (V&V) techniques. Several methods are presented. First, many of the conventional software V&V techniques such as requirements analysis and unit testing can be applied to expert-system development. Second, an expert-system developer can use automated tools to test rule consistency and structure. A more viable alternative, however, is for the developer to create his own set of consistency and completeness tests. Finally, a developer should rely on qualitative judgment to determine the validity of a knowledge base. This judgment could include expert opinion as well as specialized tests designed to determine knowledge-base certification. The chapter suggests that methods should be combined into an optimal mix in order to best undertake V&V. The g o a l o f e v e r y s o f t w a r e procurement e f f o r t , whether f o r t h e government o r p r i v a t e i n d u s t r y , i s t o o b t a i n a v a l i d a t e d , w o r k i n g , u s a b l e , and c o s t - e f f e c t i v e system. When e x p e r t systems (ES) a r e i n c l u d e d i n such a s o f t w a r e e f f o r t , t h e o v e r a l l c o m p l e x i t y o f systems i n c r e a s e s w i t h a c o r r e s p o n d i n g i n c r e a s e i n t h e importance, d i f f i c u l t y , and e f f o r t r e q u i r e d f o r v e r i f i c a t i o n and v a l i d a t i o n o f the d e l i v e r e d system. E x p e r t systems p r e s e n t problems because they o f t e n f a i l t o e x p l i c i t l y r e p r e s e n t t h e r e a s o n i n g , knowledge, and thought t h a t went i n t o t h e i r d e s i g n . O f t e n t h e code i t s e l f does n o t reveal i t s relationship t o higher-level linkages. T h i s hampers a programmer's o r knowledge e n g i n e e r ' s d i a g n o s i s o f t h e causes o f an e r r o r and t h e l o c a t i n g and m o d i f y i n g o f a l l r e l e v a n t code. V e r i f i c a t i o n i s t h e r e v i e w o f system r e q u i r e m e n t s t o ensure t h a t the system has been b u i l t t o s p e c i f i c a t i o n s . Thus, v e r i f i c a t i o n i n v o l v e s communicating a c t i v i t i e s and r e s u l t s o f p r o j e c t a c t i v i t i e s . T h i s i s done t h r o u g h documentation and v e r i f y i n g t h a t c e r t a i n s t e p s have been t a k e n . 0097-6156/90/0431-0039$06.00/0 © 1990 American Chemical Society

Hushon; Expert Systems for Environmental Applications ACS Symposium Series; American Chemical Society: Washington, DC, 1990.

Downloaded by UNIV OF CALIFORNIA SANTA BARBARA on April 3, 2016 | http://pubs.acs.org Publication Date: July 5, 1990 | doi: 10.1021/bk-1990-0431.ch003

40

EXPERT SYSTEMS FOR ENVIRONMENTAL APPLICATIONS

V a l i d a t i o n i n v o l v e s d e t e r m i n i n g t h a t the system performs w i t h a r e a s o n a b l e l e v e l of a c c u r a c y . V a l i d a t i o n i s accomplished through test and e v a l u a t i o n of ES s o f t w a r e and integrated hardware. V a l i d a t i o n thus ensures t h a t the c a p a b i l i t i e s t h a t have been speci f i e d i n the ES r e q u i r e m e n t s have been e x e r c i s e d and meet l e v e l s a c c e p t a b l e t o the u s e r . Thus, w i t h o u t a t r u e V&V methodology, much time i s l o s t i n the e v o l u t i o n a r y e x p e r t system development p r o c e s s . I t may seem s t r a n g e t o the r e a d e r t h a t a c h a p t e r on V&V appears e a r l y i n a book on e n v i r o n m e n t a l e x p e r t systems; however, t h i s i s done p u r p o s e l y . To a c h i e v e s u c c e s s i n the f i e l d i n g of e n v i r o n m e n t a l e x p e r t systems, we must u n d e r s t a n d t h a t V&V i s p a r t of the o v e r a l l e x p e r t system l i f e c y c l e . I n f a c t , how V&V w i l l be a c c o m p l i s h e d s h o u l d be p a r t of the e x p e r t - s y s t e m d e s i g n s p e c i f i c a t i o n , statement of work ( i n the case of government c o n t r a c t i n g ) , o r s p e l l e d out i n a t a s k assignment. T h i s means t h a t V&V needs t o be o p e n l y d i s c u s s e d by the e x p e r t system development team and the c l i e n t on day one. Too o f t e n have we seen e x p e r t systems b e i n g produced w i t h o u t f o r m a l v e r i f i c a t i o n (knowledge-base c e r t i f i c a t i o n ) o r w i t h o u t v a l i d a t i o n . T h i s i s p a r t i c u l a r l y common i n the e n v i r o n m e n t a l a r e a where v e r y few e x p e r t systems have gone t h r o u g h r i g o r o u s e v a l u a t i o n . The purpose of t h i s c h a p t e r i s t o d e s c r i b e t h r e e methods t h a t are a v a i l a b l e f o r e n v i r o n m e n t a l e x p e r t system V&V. These methods a r e : (1) (2) (3)

A d a p t a t i o n of c o n v e n t i o n a l V&V methods t o e x p e r t system V&V F u l l y automated p r o c e d u r e s and ES d e v e l o p e r t e s t i n g tools Specific knowledge-base verification procedures ( q u a l i t a t i v e review).

I t i s hoped t h a t e n v i r o n m e n t a l e x p e r t - s y s t e m d e v e l o p e r s can u t i l i z e these t e c h n i q u e s t o p r o v i d e r e a s s u r a n c e t o c l i e n t s on the q u a l i t y of t h e i r expert-system e f f o r t s . Method I : C o n v e n t i o n a l V&V

A p p l i e d t o E x p e r t Systems

The f i r s t method draws on the guidance a v a i l a b l e from the convent i o n a l s o f t w a r e V&V l i t e r a t u r e and attempts t o t r e a t e x p e r t system V&V l i k e c o n v e n t i o n a l V&V whenever p o s s i b l e . T h i s approach has been suggested by several authors (see for example Jacob and F r o s c h e r 1986). I n the c o n v e n t i o n a l approach, s o f t w a r e V&V i s l i k e the q u a l i t y a s s u r a n c e of any o t h e r p r o d u c t : i t r e f e r s t o the p r o c e d u r e s used t o ensure t h a t the p r o d u c t meets i t s s p e c i a l i z e d development r e q u i r e m e n t . F o r example, the Department of Defense (DOD) r e q u i r e m e n t s f o r s o f t w a r e development found i n DOD-STD-2167A c a l l f o r s o f t w a r e development a c t i v i t i e s , p r o d u c t s , r e v i e w s / a u d i t s and b a s e l i n e / d e v e l o p m e n t a l c o n f i g u r a t i o n s . That document s p e c i f i e s a d e f a u l t development c y c l e w h i c h has become known throughout the government as the " w a t e r f a l l c h a r t . " T h i s c h a r t i s shown i n F i g u r e 1. The Roman numbers on the c h a r t i n d i c a t e the f i v e phases of the c o n v e n t i o n a l l i f e c y c l e w h i c h a r e : (I) (II)

Requirements Design

Development and A n a l y s i s



3.

STUNDER

I System Requirements Analysis

Verification and Validation ofEnvironmental Expert Systems

I IV System Integration and Test

System Design

I Software Requirements Analysis

V Operation and Maintenance

II Preliminary Design II Detailed Design III Coding & Unit Testing III Component Integration III Item Testing F i g u r e 1. W a t e r f a l l C h a r t Showing C o n v e n t i o n a l Development C y c l e

Software


41

42



(III) (IV) (V)

E n c o d i n g and T e s t i n g I n t e g r a t i o n and I n s t a l l a t i o n O p e r a t i o n s and Maintenance.

The r e q u i r e m e n t s phase i n v o l v e s d e f i n i t i o n s and l i m i t a t i o n s o f the problem and c r e a t i o n o f p l a n s ( p r o j e c t p l a n s , V&V p l a n e t c . ) . The d e s i g n phase i n v o l v e s development o f d e s i g n s p e c i f i c a t i o n s and l o g i c a l processing requirements. The encoding and t e s t i n g phase i n v o l v e s a c t u a l s o f t w a r e code g e n e r a t i o n and s o f t w a r e u n i t t e s t i n g as w e l l as any e x t e r n a l i n t e r f a c e t e s t i n g . The i n t e g r a t i o n and i n s t a l l a t i o n phase i n v o l v e s p l a c i n g t h e system i n o p e r a t i o n and e n s u r i n g t h a t a l l components and t h e system as a whole have been tested properly. F i n a l l y , the operations and maintenance phase i n v o l v e s c h e c k i n g t h e system f o r any problems a f t e r i n s t a l l a t i o n and modifying the software accordingly. Many non-DoD government and p r i v a t e groups have a l s o adopted t h e g e n e r a l " w a t e r f a l l approach" as t h e i r o v e r a l l c o n v e n t i o n a l s o f t w a r e development c y c l e o r methodology. F o r example, W i l b u r n (1983) f o r Westinghouse, P o w e l l (1982) f o r t h e N a t i o n a l Bureau o f S t a n d a r d s and B r y a n t and W i l b u r n (1987) i n NUREG/CR-4640, show how v e r i f i c a t i o n i s imbedded t h r o u g h o u t t h e w a t e r f a l l c y c l e o f s o f t w a r e development. Requirements specification analysis, functional specification/ d e t a i l e d s o f t w a r e d e s i g n , c o d i n g / s o f t w a r e g e n e r a t i o n and o f course the i n t e g r a t i o n and t e s t i n g s t a g e s a l l c o n t a i n V&V elements. T a b l e I shows how t h e " w a t e r f a l l c h a r t " phases can u t i l i z e various conventional s o f t w a r e V&V methods. Unfortunately many p e o p l e (Weaver 1989; M i k s e l l 1989) b e l i e v e t h a t v e r i f i c a t i o n s p e c i f i c a l l y b e l o n g s i n t h e t r u e t e s t i n g phase o n l y . I n r e a l i t y , t h e most e f f i c i e n t use o f v e r i f i c a t i o n i s throughout t h e s o f t w a r e c y c l e and p a r t i c u l a r l y during the requirements a n a l y s i s . V e r i f i c a t i o n w i t h i n t e s t i n g i t s e l f , however, can be d i v i d e d i n t o three parts c a l l e d : • • •

Unit testing I n t e g r a t i o n t e s t i n g and Formal q u a l i f i c a t i o n t e s t i n g .

U n i t t e s t i n g i s the t e s t i n g of the smaller i d e n t i f i a b l e software components ( u n i t s ) . I n t e g r a t i o n t e s t i n g i s performed t o demonstrate t h a t u n i t s and h i g h e r - l e v e l components o f t h e system work t o g e t h e r . Formal q u a l i f i c a t i o n t e s t i n g i s a n o t h e r term f o r v a l i d a t i o n and i s performed t o f o r m a l l y demonstrate t h a t t h e s o f t w a r e meets i t s e s t a b l i s h e d requirements. Most e x p e r t - s y s t e m development e f f o r t s w i l l a l s o f o l l o w t h e f i v e phases o f t h e c o n v e n t i o n a l life c y c l e w i t h heavy emphasis on knowledge a c q u i s i t i o n and r e p r e s e n t a t i o n i n t h e d e s i g n phase. S e v e r a l o f t h e V&V p r o c e d u r e s o u t l i n e d i n T a b l e I f o r c o n v e n t i o n a l s o f t w a r e l i f e c y c l e can be a p p l i e d i n ES V&V. Table I I provides a l i s t o f some a p p l i c a b l e p r o c e d u r e s . Many p r o c e d u r e s a r e s i m p l y a part o f any s o f t w a r e development e f f o r t , but a r e many t i m e s o v e r l o o k e d i n ES e f f o r t s . F o r example, ES d e v e l o p e r s o f t e n t i m e s do not develop adequate requirements documentation. They have difficulty i n t r a c i n g knowledge r e p r e s e n t a t i o n structures (rule p a t h s , frames, e t c . ) back t o d e s i g n documents w h i c h i n t u r n makes


3.

STUNDER

Verification and Validation of Environmental Expert Systems


T a b l e I . C o n v e n t i o n a l Software V&V Methods R e l a t e d t o C o n v e n t i o n a l Software Development C y c l e

W a t e r f a l l C y c l e Phase

P o s s i b l e V&V Method

(I)

Requirements

Tracking o f requirements Develop t e s t a b l e r e q u i r e ments Use o f s t r u c t u r e d methods (break b i g p i e c e s t o smaller pieces) C o n c e p t u a l model d e v e l o p ment/verification

(II)

Software D e s i g n

Use o f assumptions as check p o i n t s ; v i o l a t i o n s cause known items t o o c c u r S t r u c t u r e d programming t e c h n i q u e s ( a i d s i n V&V) Complete d e s i g n documentation T r a c i n g o f d e s i g n back t o requirements

(III)

Encoding and T e s t i n g

(IV)

I n t e g r a t i o n and

Peer r e v i e w ( o v e r a l l check) Team development (team check a g a i n s t i t s e l f ) Use o f approved s t a n d a r d s (format l i k e modules a l i k e ) V a r i o u s code and d a t a f l o w analyses D e s i g n p a r a l l e l modules ( e r r o r i n one c o u l d be an e r r o r i n another) Unit testing Integration testing Formal q u a l i f i c a t i o n testing Systems t e s t i n g / s i m u l a t e d o p e r a t i n g environments/ documentation Acceptance t e s t i n g ( a f t e r installation) Summary o f v e r i f i c a t i o n performed A p p r o p r i a t e v a l i d a t i o n and acceptance

(V)

O p e r a t i o n s and

Analysis

Installation

Maintenance

Checks o f performance requirements V e r i f y any m o d i f i c a t i o n s made


4

44

EXPERT SYSTEMS FOR ENVIRONMENTAL APPLICATIONS T a b l e I I . Use o f C o n v e n t i o n a l V&V P r o c e d u r e s


Development ES Phase

V&V

i n ES Development

Example Procedures

(D

Requirements A n a l y s i s

• •

Conceptual modeling/flowcharting Tracking requirements ( p a r t i c u l a r l y i f ES w i l l be large)

(II)

Design

•

Knowledge-base d e s i g n t o r e q u i r e ments check Modular programming s t r a t e g y ( i f ES d e s i g n p e r m i t s )

•

(III) Testing

• •

U n i t t e s t chunks o f ES code Test i n t e g r a t i o n o f o u t s i d e ( i . e . , LOTUS) programs

(IV)

• •

ES r e p r o d u c i b i l i t y check ( m i r r o r expert?) Acceptance t e s t i n g

•

V e r i f y any m o d i f i c a t i o n s mode

(V)

I n t e g r a t i o n and Installation

O p e r a t i o n and Maintenance


3.

STUNDER


knowledge base v e r i f i c a t i o n d i f f i c u l t . Emphasis on t r a c e a b i l i t y could, therefore, a l l e v i a t e a p o t e n t i a l v e r i f i c a t i o n stumbling block. Stunder (1986) has shown t h a t two i s s u e s a r i s e from t h e d i r e c t a p p l i c a t i o n o f c o n v e n t i o n a l V&V t e c h n i q u e s t o e x p e r t system V&V. These i s s u e s a r e :


• •

Establishment o f c l e a r acceptance c r i t e r i a A r c h i t e c t u r a l items.

Acceptance c r i t e r i a f o r e x p e r t systems depends l a r g e l y on t h e f u n c t i o n o f t h e system t o be d e v e l o p e d . T h i s r e l a t e s t o the need f o r c l e a r l y d e f i n i n g requirements. F o r example, t h e Thermal Performance A d v i s o r System developed f o r the E l e c t r i c Power R e s e a r c h I n s t i t u t e and used i n n u c l e a r p o w e r p l a n t s r e l i e s on c r i t e r i a centered around efficient plant performance. The Zeus m e t e o r o l o g i c a l system, devel-oped by GEOMET, I n c . , f o r t h e U n i t e d S t a t e s A i r F o r c e ( S l e t t e n e t a l . 1988), b u i l t a c c e p t a n c e c r i t e r i a around site flight profiles and a c c e p t a b l e weather forecast parameters as a means o f v e r i f i c a t i o n and v a l i d a t i o n . GEOMET's Underground S t o r a g e Tank r e g u l a t o r y system on the o t h e r hand used c l e a r l y e s t a b l i s h e d EPA r e g u l a t i o n s and p r o c e d u r e s i n e s t a b l i s h i n g acceptance c r i t e r i a . I n a l l c a s e s , a c c e p t a n c e c r i t e r i a were documented i n the r e q u i r e m e n t s phase. Many t i m e s , the a c c e p t a n c e c r i t e r i a a l s o depend on t h e type o f knowledge t o be c a p t u r e d . Sometimes an e x p e r t - s y s t e m approach i s chosen as a convenience t o i n c o r p o r a t e " h a r d " knowledge, such as i n f l e x i b l e r u l e s i n v o l v i n g r e g u l a t o r y r e q u i r e m e n t s ( S t u n d e r and H l i n k a 1989). I n such c a s e s , t h e e x p e r t system approach has no d i r e c t a f f e c t on the acceptance c r i t e r i a ; t h e y a r e no d i f f e r e n t t h a n f o r c o n v e n t i o n a l s o f t w a r e . F r e q u e n t l y , however, an e x p e r t system i s chosen t o encode knowledge t h a t has a " s o f t " component. T h i s means t h a t knowledge needs t o be g a t h e r e d f o r the system from e x p e r t s o r from r e f e r e n c e m a t e r i a l . A second a r e a a f f e c t i n g d i r e c t a p p l i c a b i l i t y o f c o n v e n t i o n a l V&V t e c h n i q u e s t o ES V&V i s t h e a r c h i t e c t u r a l s t r u c t u r e o f e x p e r t systems. F i r s t , guidelines for assigning architectural levels i n c o n v e n t i o n a l s o f t w a r e do n o t n e c e s s a r i l y a p p l y i n e x p e r t system design. F o r example, i n c o n v e n t i o n a l s o f t w a r e development, u n i t s are d e f i n e d f o r t e s t i n g purposes; i n ES development, u n i t s may be h a r d e r t o d e f i n e because o f the dependency o f r u l e s , o b j e c t s , e t c . S e c o n d l y , i n r u l e - b a s e d systems, r u l e i n t e r a c t i o n s a r e f r e q u e n t l y d i f f i c u l t t o predict. O f t e n t i m e s , the ES code does not r e v e a l i t s relationship t o higher level linkages. A programmer may have a d i f f i c u l t time f i n d i n g t h e cause o f an e r r o r and m o d i f y i n g a l l codes. C o n s i s t e n c y and completeness c h e c k i n g a l g o r i t h m s a r e a v a i l a b l e (Nguyen e t a l . 1985) w h i c h c a n p o i n t o u t s i m p l e examples o f c o n f l i c t i n g r u l e s and m i s s i n g r u l e s . Some o f t h e s e r u l e problems are shown i n T a b l e s I I I and IV, but t h e r e can be a d d i t i o n a l s u b t l e i n t e r a c t i o n s a c r o s s a r u l e base. There a r e f i v e p o s s i b l e ways t h a t r u l e s can be i n c o n s i s t e n t . These i n c l u d e : • •

Redundancy Conflict


4


Table I I I .

Common Consistency Problems


Redundancy - two r u l e s have the same antecedent ( I F p a r t ) and t h e i r conclusions (THEN p a r t ) contain i d e n t i c a l a c t i o n s / c l a u s e s , e.g. Rule 1:

IF (SUBSYSTEM 1 (STATUS) - ABNORMAL) THEN NOTIFY(OPERATOR)

Rule 2:

IF (SUB_SYSTEM1(STATUS) - ABNORMAL) THEN NOTIFY(OPERATOR) AND SHUT_DOWN(SUB_SYSTEM1)

Only Rule 2 i s necessary.

C o n f l i c t - two r u l e s have the same antecedent, but t h e i r c o n c l u sions are c o n t r a d i c t o r y , e.g. Rule 1:

IF (SUB_SYSTEM1(TEMP) > 140) THEN SUB_SYSTEM1(STATUS):- ABNORMAL

Rule 2:

IF (SUB_SYSTEM1(TEMP) > 140) THEN SUB_SYSTEM1(STATUS):- NORMAL

Subsumption - two r u l e s have the same c o n c l u s i o n , but the antecedent(s) of one i s contained w i t h i n the other, e.g. Rule 1:

IF (SUB_SYSTEM1(TEMP) > 140) THEN SUB_SYSTEM1(STATUS):- ABNORMAL

Rule 2:

IF (SUBSYSTEM 1 (TEMP) > 140) AND (SUB_SYSTEMi(VOLTAGE) - 0) THEN (SUBSYSTEM 1(STATUS):- ABNORMAL

Rule 2 i s unnecessary.

Unnecessary IF r u l e s - two r u l e s have c o n t r a d i c t o r y clauses i n otherwise i d e n t i c a l antecedents, and i d e n t i c a l conclusions, e.g. Rule 1:

IF (SUBSYSTEM 1 (STATUS) - ABNORMAL) AND (SUB_SYSTEM2(STATUS) - ABNORMAL) THEN NOTIFY(OPERATOR)

Rule 2:

IF (SUB_SYSTEM1(STATUS) = NORMAL) AND (SUB_SYSTEM2(STATUS) - ABNORMAL) THEN NOTIFY(OPERATOR)

The c o n c l u s i o n , NOTIFY(OPERATOR), i s based s t r i c t l y on the " i d e n t i c a l " p o r t i o n of the antecedent, (SUB_SYSTEM2(STATUS) ABNORMAL). These two rules should be combined into one r u l e , e l i m i n a t i n g the (SUB_SYSTEM1(STATUS) - ...) clause.

C i r c u l a r i t y - a set of r u l e s forms a c y c l e , r e s u l t i n g i n the Expert System equivalent of an i n f i n i t e loop.


3.

STUNDER

Verification and Validation of Environmental

T a b l e IV.

47

Common Completeness Problems

U n r e f e r e n c e d a t t r i b u t e v a l u e s - l e g a l v a l u e s w h i c h may be a s s i g n e d but r e s u l t i n no f u r t h e r p r o c e s s i n g , i . e . , no o t h e r r u l e s r e f e r t o t h a t v a l u e , e.g.


I F . . . THEN SUBSYSTEM 1 (STATUS) := QUESTIONABLE but no o t h e r r u l e s r e f e r t o QUESTIONABLE s t a t u s .

I l l e g a l a t t r i b u t e v a l u e s - a r u l e r e f e r s t o an a t t r i b u t e ' s where t h e v a l u e i s not a l e g a l v a l u e , e.g.

value

IF (SUBSYSTEM 1(STATUS) - DESCENDING) THEN where DESCENDING i s not a l e g a l v a l u e f o r subsystem s t a t u s , f o r the s a t e l l i t e .


only


48


Subsumption Unnecessary i f s Circularity. Examples o f each problem a r e g i v e n i n t h e t a b l e ; however, c i r c u l a r i t y and c o n f l i c t appear t o be t h e most p r e v a l e n t o f t h e problems. C i r c u l a r i t y i n v o l v e s a s e t o f r u l e s f i r i n g and r e s u l t i n g i n a repet i t i o u s answer. The key t o s t o p p i n g c i r c u l a r i t y i s t o f i n d t h e proper p o i n t i n the r u l e path t o branch o f f (without d e s t r o y i n g another p o r t i o n o f t h e code). The p r o c e s s can be t e d i o u s . Conflict u s u a l l y r e s u l t s from poor d e s i g n o r knowledge r e p r e s e n t a t i o n where rules provide contradictory conclusions. Completeness problems t y p i c a l l y i n v o l v e r u l e s t h a t a r e ends i n and o f themselves. F o r example, a r u l e may r e f e r e n c e an a t t r i b u t e v a l u e w h i c h does n o t r e s u l t i n f u t u r e p r o c e s s i n g . S i m i l a r l y , a r u l e may r e f e r e n c e an i l l e g a l v a l u e , whereby t h e r u l e f a i l s . Rule o r d e r may i n f l u e n c e b o t h t h e o p e r a t i o n (and t h e r e f o r e t h e r e s u l t s ) and t h e e x e c u t i o n time o f a r u l e base. Therefore i t i s n e c e s s a r y t o t e s t t h e e x p e r t system under a demanding v a r i e t y o f scenarios. Many o f t h e c o n v e n t i o n a l V&V t e c h n i q u e s w i t h i n t h e c o n v e n t i o n a l s o f t w a r e l i f e c y c l e can e i t h e r be d i r e c t l y a p p l i e d t o BS development o r can be m o d i f i e d t o handle t h e somewhat i n s t r u c t u r e d n a t u r e o f an ES. Formal u n i t t e s t i n g used i n c o n v e n t i o n a l s o f t w a r e development can a l s o be a p p l i e d by l o o k i n g f o r items such as c o n s i s t e n t and complete r u l e s . Method I I ;

F u l l y Automated and ES D e v e l o p e r T e s t i n g T o o l s

F u l l y Automated Methods. The second approach t o e x p e r t system V&V has been t h e use o f v e r i f i c a t i o n t o o l s . These t o o l s g e n e r a l l y a n a l y z e t h e source code f o r t h e e x p e r t system. T h i s a n a l y s i s does not i n v o l v e e x e c u t i n g o r e x e r c i s i n g t h e system. V e r i f i c a t i o n t o o l s f o r e x p e r t systems a r e s i m i l a r t o those o f c o n v e n t i o n a l s o f t w a r e . These t o o l s p r o c e s s t h e source code and i n some cases d e s c r i p t i o n s of t h e source code l o o k i n g f o r problems t h a t can be d e t e r m i n e d m e c h a n i s t i c a l l y by a program. C o n v e n t i o n a l t o o l s can d e t e c t : • • • •

Uninitialized variables Type mismatches Number o f l i n e s p e r module Redundant code

• • •

Unreachable s e c t i o n s of code QA s t a n d a r d violations P r o p e r commenting

T o o l s f o r r u l e - b a s e d e x p e r t systems ( a s w e l l as manual methods) s h o u l d e v a l u a t e t h e c o n s i s t e n c y and completeness o f t h e r u l e s . The TEIRESIAS program ( D a v i s 1976) l i n k e d t o t h e MYCIN i n f e c t i o u s d i s e a s e system was one o f t h e f i r s t attempts t o d e v e l o p an automated verification tool. L a t e r work by Suwa e t a l . (1982) f o r t h e ONCOCIN ( c l i n i c a l o n c o l o g y ) system examined a r u l e s e t as i t was read i n t o the system. T h i s r u l e checker assumes t h a t f o r each c o m b i n a t i o n o f a t t r i b u t e v a l u e s a p p e a r i n g i n t h e antecedent a c o r r e s p o n d i n g r u l e exists. The LES system d e s c r i b e d by Nguyen e t a l . (1987) i s a g e n e r i c r u l e - b a s e d e x p e r t system b u i l d i n g t o o l w h i c h has an e x t e n s i v e


3.

STUNDER



c h e c k e r c a l l e d CHECK. Redundant and c o n f l i c t i n g r u l e s a l o n g w i t h many o t h e r t y p e s o f problems can be uncovered w i t h CHECK. The problem w i t h automated t o o l s i s t h a t many t i m e s t h e y a r e s o f t w a r e / s y s t e m dependent. F o r example, CHECK has been p o r t e d t o the Automated Reasoning T o o l (ART) framework b u t t o l i t t l e e l s e ( s h o r t o f a MYCIN type s t r u c t u r e ) . Thus, automated t o o l s a r e n o t necessarily available f o r off-the-shelf applications i n shell, language o r t o o l e x p e r t system development. Cost-to-benefit-type t r a d e - o f f s t u d i e s need t o be undertaken b e f o r e f u l l y automated t o o l s are f u l l y r e c o g n i z e d as a ( p a r t i a l ) means o f e x p e r t system V&V. E x p e r t System Developer T e s t i n g P r o c e d u r e s . An e x p e r t system d e v e l oper c a n g e n e r a t e h i s / h e r own code f o r t e s t i n g v a r i o u s elements o f the ES code w i t h o u t going to fully automated s e l f - c o n t a i n e d procedures. Borrowing from t h e c o n v e n t i o n a l s o f t w a r e s i d e , t h r e e methods can be used f o r code t e s t i n g . They a r e : • • •

stubs drivers simulators.

A s t u b i n v o l v e s a s e t o f code which produces known r e s u l t s . F o r example, a s t u b imbedded i n ES code c o u l d c o n t a i n "canned" f a c t s r a t h e r than e x e c u t a b l e r u l e s . A s t u b c o u l d be w r i t t e n t o determine whether t h e o u t p u t from a module i s c o r r e c t . The s t u b may be as s i m p l e as a "canned" temperature v a l u e as p a r t o f a r u l e . I f the r u l e p a t h i s e x e c u t i n g p r o p e r l y , t h a t "canned" temperature v a l u e would be r e t u r n e d t o t h e module. Stubs a r e u s u a l l y u t i l i z e d e a r l y i n t h e ES s o f t w a r e development c y c l e where l i n k i n g r u l e s , o b j e c t s o r frames a r e n o t a v a i l a b l e , but p i e c e s o r u n i t s o f ES code a r e available. Stubs a r e a l s o o c c a s i o n a l l y i n s e r t e d i n t h e i n t e g r a t i o n phase t o f o r c e a known r e s u l t o r t o v e r i f y a p o r t i o n o f t h e knowledge-base. D r i v e r s p r o v i d e another means o f t e s t i n g ES code. They a r e much l a r g e r i n scope and e s s e n t i a l l y c a n d r i v e t h e s o f t w a r e u n i t under t e s t . A d r i v e r can e x e r c i s e a p o r t i o n o f t h e o v e r a l l program over a f u l l range o f p o s s i b l e o u t p u t s , thus a l l o w i n g f o r d e t e r m i n a t i o n o f e r r o r - b o u n d c r o s s i n g s ( e . g . , u n r e a l i s t i c v a l u e s ) o r l a c k o f code execution. A t h i r d t e c h n i q u e t h a t a d e v e l o p e r c a n u s e t o t e s t ES code i n v o l v e s s i m u l a t i o n . S i m u l a t o r s can t e s t e n t i r e systems o r p a r t s o f systems and a r e s i m i l a r t o d r i v e r s . A s i m u l a t o r approach i s more f u n c t i o n a l l y o r i e n t e d than a d r i v e r i n t h a t a s i m u l a t o r u s u a l l y w i l l be u t i l i z e d t o t e s t a c a l l f o r a c o n v e n t i o n a l language o r package ( e . g . , LOTUS), whereas a d r i v e r i s i n t e r e s t e d i n v a l u e s o r a s p e c i f i c ES element. Simulators can a l s o generate data input f o r r e a l - t i m e t e s t i n g o f ES code. Use o f a s i m u l a t o r t e s t approach, p a r t i c u l a r l y j u s t p r i o r t o i n t e g r a t i o n and i n s t a l l a t i o n , a l l o w s f o r i n i t i a l d a t a and i n f o r m a t i o n t o be g e n e r a t e d f o r Method I I I which i s the q u a l i t a t i v e knowledge-base r e v i e w approach. Method I I I ; Q u a l i t a t i v e Knowledge-Base Review The q u a l i t a t i v e r e v i e w o f knowledge bases i s a major i s s u e f a c i n g e x p e r t - s y s t e m d e v e l o p e r s . I t i s sometimes r e f e r r e d t o as knowledge-


4


50


base c e r t i f i c a t i o n . T h i s i s because o f e x p e r t disagreement on knowledge-base c o n t e n t . Many a u t h o r s suggest t h a t knowledge-base e x p e r t disagreement c o u l d be downplayed by t h e p r o p e r s e l e c t i o n o f expert-system a p p l i c a t i o n s . Many e x p e r t systems a r e u n f o r t u n a t e l y developed i n a r e a s w h i c h a r e s t i l l i n t h e b a s i c R&D stage and a r e not ready y e t f o r widespread use. In d e t e r m i n i n g e x p e r t o p i n i o n on ES o u t p u t , a v a r i a t i o n o f t h e c l a s s i c T u r i n g T e s t (Stunder e t a l . 1988) can work w i t h a group o f experts. Test d e s i g n i n v o l v e s p r e s e n t i n g a s e r i e s o f s i m u l a t e d cases t o a group o f e x p e r t s and t o t h e e x p e r t system. T h i s i s done to v e r i f y not o n l y t h a t t h e e n t i r e e x p e r t system responds i n a way c o n s i s t e n t w i t h human performance, but t h a t each p i e c e o f t h e knowledge base e x e c u t e s c o r r e c t l y . [ I d e a l l y i t s h o u l d be i m p o s s i b l e f o r an o b s e r v e r r e v i e w i n g t e s t r e s u l t s t o determine w h i c h s e t o f r e s u l t s comes from i n t e r a c t i o n w i t h t h e e x p e r t system (Jacobs and Chee, 1988).] Independent r e v i e w o f t h e knowledge base i s p o s s i b l e by u s i n g v a r i o u s acceptance l e v e l s , human performance checks and s e l e c t e d groups (Stunder e t a l . 1988). The use o f a s i m u l a t o r t e c h n i q u e d e s c r i b e d under Method I I can generate s c r i p t s w h i c h when reviewed a g a i n s t e x p e r t o p i n i o n can p r o v i d e u s e f u l knowledge-base review. In q u a l i t a t i v e l y r e v i e w i n g t h e knowledge-base, i t i s sometimes d i f f i c u l t t o determine t h e minimum competency l e v e l o f t h e system i n o r d e r t o even undertake a T u r i n g T e s t approach. Thus, d e f i n i n g minimum e n v i r o n m e n t a l e x p e r t system competency i s a d i f f i c u l t t a s k w h i c h s h o u l d be done i n t h e s o f t w a r e r e q u i r e m e n t s phase. This i s where many e x p e r t systems f a i l . C r i t e r i a a r e u s u a l l y n o t developed to assess expert system performance i n the e a r l y stages. Q u a n t i f i a b l e c r i t e r i a such as "average a c c u r a c y " w i l l a i d i n any knowledge-base review process, but j u d g i n g which r u l e s meet competency c r i t e r i a i s d i f f i c u l t . The r e c e n t e x p e r i e n c e o f L a i and P e a r t (1989) w i t h t h e FARMSYS a g r i c u l t u r a l e x p e r t system, f o r example, i n d i c a t e s t h a t t h e methodo l o g y o f a s m a l l group o f e x p e r t s meeting t o g e t h e r t o e v a l u a t e and c r i t i q u e a system o f f e r s a good p r a c t i c a l a l t e r n a t i v e f o r v e r i f y i n g and v a l i d a t i n g a system. S i m i l a r l y , Stunder e t a l . (1986) used a g r o u p - o r i e n t e d methodology t o n o t o n l y v a l i d a t e a c t u a l e n v i r o n m e n t a l system performance but t o v e r i f y t h e p h y s i c a l c o n c e p t s imbedded w i t h i n t h e knowledge base. The degree o f q u a l i t a t i v e r e v i e w w i l l v a r y f o r each ES development. Developers s h o u l d c a r e f u l l y c o n s i d e r , however, how e x p e r t s can r e v i e w the e x p e r t system i n an u n b i a s e d method. The T u r i n g Method p r o v i d e s a means o f u n d e r t a k i n g such an evaluation. Summary The p r o l i f e r a t i o n o f e n v i r o n m e n t a l e x p e r t systems means t h a t more a t t e n t i o n must be spent on how these systems a r e f i e l d e d . Proper V&V o f e x p e r t systems ensures t h a t a c l i e n t ' s c o n t r a c t d o l l a r s a r e w e l l spent on t h e o v e r a l l system e f f o r t . I t i s e s p e c i a l l y important to i n c l u d e V&V i n a l l d e s i g n s p e c i f i c a t i o n s . This chapter has o u t l i n e d three general approaches f o r u n d e r t a k i n g V&V. There i s no one s i n g l e approach. Instead, a l l approaches and t e c h n i q u e s s h o u l d be used i n some form d u r i n g an e n t i r e e x p e r t system l i f e c y c l e . As more e n v i r o n m e n t a l expert systems a r e f i e l d e d , i t i s hoped t h a t t h e w r i t i n g o f V&V r e q u i r e m e n t s becomes mandatory throughout t h e e x p e r t system l i f e cycle process. Hushon; Expert Systems for Environmental Applications ACS Symposium Series; American Chemical Society: Washington, DC, 1990.

3.

STUNDER


51

LITERATURE CITED 1. 2.


3. 4. 5. 6. 7.

8. 9. 10. 11.

12. 13. 14.

15. 16. 17.

Bryant, J.L.; Wilburn, N.B. Handbook of software and quality assurance techniques applicable to the nuclear industry; NUREG/CR-4640, 1987. Davis, R. Applications of meta-level knowledge to the construction, maintenance, and use of large knowledge bases, Stanford University, Dept. of Computer Science, Ph.D. dissertation, 1976. Jacob, R.J.; Froscher, J.N. Developing a software engineering methodology for knowledge-based systems, TR9019, Naval Research Laboratory, 1986. Jacobs, J.M.; Chee, C.W. Specification of expert systemstesting, TR 88-303, HQ Space Division USAF, 1988. Lal, H.; Peart, R.M. Engineering farm knowledge for a seamless decision support system, 1989. Miksell, S.G. Operational evaluation of an expert system: the FIESTA approach. Ann. Conf. Int. Assoc. of Knowledge Eng., University of Maryland, 1989. Nguyen, T.A.; Perkins, W.A.; Laffey, T.J.; Pecora, D. Checking an expert system's knowledge base for consistency and completeness. In Proceedings: Ninth Joint Int. Conf. on A r t i f i c i a l Intelligence, Menlo Park, California, 1985. Nguyen, T.A.; Perkins, W.A.; Laffey, T.J.; Pecora, D. Knowledge-Base Verification. AI Magazine 1987, 8(2), 69-75, 1987. Powell, P.B. Software validation, verification and testing: technique and tool reference guide, NBS SPI Pub NBS-SP-500-93, National Bureau of Standards, Washington, DC, 1982. Sletten, T.N.; Stunder, M.J.; Lee, S.M. Use of environmental expert systems. Army Information Systems and AI Conference, American Defense Preparedness Association, El Paso, Texas, 1988. Stunder, M.J. Managing ΑΙ-environmental applications. First A r t i f i c i a l Intelligence Research in Environmental Sciences Conference, Environmental Research Laboratory, NOAA, Boulder, Colorado, 1986. Stunder, M.J; Hlinka, D.J. Applications of hypertext systems in the environment, Third A r t i f i c i a l Intelligence Research in Environmental Sciences Conference, Washington, DC, 1989. Stunder, M.J.; Pelton, D.J.; Lee, S.M. Zeus I I : a multipurpose environmental expert system, 55th Operations Research Society Meeting, Montgomery, Alabama, 1988. Stunder, M.J.; Pelton, D.J.; Sletten, T.N. A meteorological knowledge-based expert system, First A r t i f i c i a l Intelligence Research in Environmental Sciences Conference, Environmental Research Laboratory, NOAA, Boulder, Colorado, 1986. Suwa, M.; Scott, A.C.; Shortliffe, E.M. An approach to verifying completeness and consistency in a rule-based expert system, AI Magazine 1982, 3(4), 16-21. Weaver, S.J. Conventional testing and knowledge-based systems, Ann. Conf. Int. Assoc. of Knowledge Eng., University of Maryland, 1989. Wilburn, N.P. Guidelines-software verification. HEDL-TC-2425, Westinghouse Hanford Company, Richland, Washington, 1983.

RECEIVED February 12, 1990


Verification and Validation of Environmental Expert Systems

Recommend Documents