Computer Assembled Testing in a Large Network: The SOCRATES

9

Computer Assembled Testing in a Large Network: Downloaded by UNIV OF CALIFORNIA SANTA BARBARA on March 22, 2018 | https://pubs.acs.org Publication Date: June 1, 1975 | doi: 10.1021/bk-1975-0019.ch009

T h e S O C R A T E S System WILLIAM V. WILLIS Chemistry Department, California State University, Fullerton, Calif. 92634 OLIVER J. SEELY, JR. Chemistry Department, California State College, Dominguez Hills, Calif. 90747 The use of computers in the formal educational process is increasing in diversity and extent. As scientists, chemists have often used computer techniques to solve a wide variety of problems. As educators, we are beginning to use computers as a resource and tool to develop more effectively and efficiently the knowledge and s k i l l s of our students. Much of this use has been by individual instructors, and therefore has been developed with highly specialized goals in mind for a small number of users. For example, on-campus CAI type instruction in the analysis of NMR spectra, or solving stoichiometry problems etc. This paper describes the formulation and implementation of a large general purpose program called SOCRATES (Student Oriented Classroom Analysis and Test Evaluation System) designed for operation in a 19 campus computer network serving 290,000 students and 16,000 faculty in the California State University and Colleges. SOCRATES is a data management system designed to produce and process exercises (tests, tutorials, homework, etc.) as directed from collections of questions, or data banks, in thirteen fields: chemistry, physics, mathematics, psychology, economics, US history, accounting, FCC review, biology, testing and measurement, counseling and guidance, p o l i t i c a l science and data processing. The concept of generating tests by selecting prepared questions from a collection is certainly not novel. Instructors have traditionally maintained in personal collections f i l e cards which are searched and given to a typist for suitable duplication. On a larger scale, commerical testing services have maintained computerized files for approximately twenty years, and presently more than one hundred educational institutions in the USA and Canada have 129 Lykos; Computer Networking and Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 1975.

Downloaded by UNIV OF CALIFORNIA SANTA BARBARA on March 22, 2018 | https://pubs.acs.org Publication Date: June 1, 1975 | doi: 10.1021/bk-1975-0019.ch009

130

COMPUTER

NETWORKING AND

CHEMISTRY

c o l l e c t i o n s s t o r e d on magnetic t a p e . Usage o f t h e l a t t e r has not been g r e a t f o r a v a r i e t y o f r e a s o n s , i n c l u d i n g poor communications, l i m i t e d a c c e s s i b i l i t y , and t e a c h e r a t t i t u d e s , One SOCRATES c r i t i c summed up t h e s e a t t i t u d e s i n the remark "You a r e j u s t automati n g what we a l l do." We w i l l attempt i n t h e f o l l o w i n g paragraphs t o d e s c r i b e t h i s v e r y automation, and the r e a c t i o n s o f e d u c a t o r s , s t u d e n t s , and o t h e r s which l e a d us t o b e l i e v e t h a t i t w i l l produce a p r o f o u n d change i n c e r t a i n a s p e c t s of the e d u c a t i o n a l p r o c e s s ; i n d e e d , t h e l a r g e s c a l e a p p l i c a t i o n o f computer t e c h nology t o e d u c a t i o n t h r u d a t a banks and networking i s f e a s i b l e , e c o n o m i c a l , and p e d a g o g i c a l l y sound. The

SOCRATES R e t r i e v a l System

Our d a t a banks were d e v e l o p e d i n i t i a l l y f o r use i n a r e t r i e v a l s y s t e m d e s c r i b e d by L i p p e y , e t a l . (_1) . I t was i m p o s s i b l e f o r a l l campuses i n t h e network t o a c c e s s the d a t a banks d i r e c t l y due t o hardware and management c o n s t r a i n t s . The system was r e d e s i g n e d and r e w r i t t e n i n ANSI COBOL t o o p e r a t i o n on a CDC 3300 c e n t r a l computer. Formal i n s t i t u t i o n a l s u p p o r t f o r t h e system has been a v a i l a b l e f o r the p a s t two y e a r s , d u r i n g which time t h e c h e m i s t r y d a t a bank has grown from 2,000 t o 10,000 i t e m s . The r e t r i e v a l system a l l o w s t h e u s e r t o s e l e c t items from the bank employing a v a r i e t y o f parameters, e i t h e r s i n g l y on i n c o n c e r t , t o execute the s e a r c h . These parameters i n c l u d e a c a t e g o r y number (see below), l e v e l o f d i f f i c u l t y , a b e h a v i o r l e v e l ( r e q u i r ing e i t h e r the d e m o n s t r a t i o n of knowledge o r t h e a p p l i c a t i o n of knowledge), key words, i t e m s o u r c e i d e n t i f i c a t i o n , and a s p e c i a l i z e d s e a r c h parameter whose f u n c t i o n v a r i e s from bank t o bank. Two o t h e r g e n e r a l s e a r c h parameters a r e a v a i l a b l e which a l l o w b l o c k s o f i n t e r r e l a t e d q u e s t i o n s (macro items) and items which r e q u i r e m a t e r i a l s not i n t h e d a t a bank, such as s l i d e s , a u d i o o r TV c a s s e t t e s and f i g u r e s , (enhanced items) t o be s e l e c t e d o r s u p p r e s s e d from selection. Each e x e r c i s e can c o n t a i n up t o 150 i t e m s , and may be e d i t e d by the u s e r t o d e l e t e o r add i t e m s . Up t o n i n e v e r s i o n s o f the same e x e r c i s e may be r e q u e s t e d w i t h d i f f e r e n t o r d e r s of p r e s e n t a t i o n of the c o n t e n t s , and t h e e x e r c i s e can be produced by l i n e p r i n t e r on e i t h e r c o n s o l e paper o r c o n t i n u o u s form r e p r o d u c t i o n m a s t e r s a t the u s e r s r e q u e s t . Obj e c t i v e items i n the e x e r c i s e s ( m u l t i p l e c h o i c e , t r u e f a l s e ) can be computer s c o r e d ; i n d i v i d u a l s t u d e n t performance may be then compared w i t h r e c o r d s o f

Lykos; Computer Networking and Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 1975.


9.

WILLIS

AND

SEELY

Computer

Assembled

Testing

131

a v e r a g e s f o r p a s t r e s p o n s e s m a i n t a i n e d by SOCRATES. The SOCRATES system c a n be used e i t h e r i n a b a t c h mode o r a t i m e s h a r i n g mode. No knowledge o f computer programming i s n e c e s s a r y t o use t h e system's e d u c a t i o n a l f e a t u r e s . The c o n t e n t , g e n e r a t i o n , admini s t r a t i o n , and g r a d i n g o f SOCRATES e x e r c i s e s i s cont r o l l e d by t h e u s e r . T y p i c a l l y , a request submitted by t e l e p h o n e w i l l r e s u l t i n t h e o v e r n i g h t p r o d u c t i o n o f t h e e x e r c i s e , and d e l i v e r y t h e f o l l o w i n g day. The most i m p o r t a n t f e a t u r e i n t h e d e s i g n f o r u s e r a c c e s s t o each d a t a bank i s t h e s u b j e c t m a t t e r c l a s s i f i c a t i o n system. The SOCRATES system u s e s t e n major c a t e g o r i e s which a r e a r r a y e d i n a h e i a r c h y f o r s e a r c h ing. T a b l e I shows t h e main s u b j e c t c l a s s i f i c a t i o n s f o r t h e c h e m i s t r y bank. Table I Major C a t e g o r i e s i n t h e SOCRATES C h e m i s t r y Bank 00000 I. Reserved (non-standard items) 10000 I I . I n t r o d u c t o r y Chemistry 20000 I I I . Atomic Theory and S t r u c t u r e s 30000 IV. Chemical P e r i o d i c i t y 40000 V. S t o i c h i o m e t r y and The Mole Concept 50000 V I . K i n e t i c and E q u i l i b r i u m 60000 V I I . S o l u t i o n Chemistry 70000 VIII.Bonding 8 0000 IX. Thermodynamics 90000 X. Topics W i t h i n each major c a t e g o r y , sub c a t e g o r i e s a r e e s t a b l i s h e d , as shown i n T a b l e I I which shows a p o r t i o n o f the d e t a i l e d c l a s s i f i c a t i o n system f o r s o l u t i o n chemistry. Table I I P a r t i a l SOCRATES C a t e g o r y L i s t i n g f o r S o l u t i o n Chemistry 60000 V I I . S o l u t i o n C h e m i s t r y 62000 62100 62140 62141 62142

B. A c i d s , Bases, and S a l t s 1. G e n e r a l P r o p e r t i e s (D) A c i d s and Bases (1) S t r o n g A c i d s ( i n c l u d i n g oxidizing properties) (2) Weak A c i d s

The e x e r c i s e g e n e r a t i o n program would, upon r e q u e s t f o r s o l u t i o n c h e m i s t r y i t e m s , s e a r c h a l l items between t h e l i m i t s 60000 and 69999; i f i n s t r u c t e d t o s e a r c h f o r items d e a l i n g w i t h a c i d s , bases, and s a l t s , c a t e g o r i e s 62000 t h r u 62999 would be s e a r c h e d , and so on. I n t h i s manner, a 10,000 i t e m bank c a n be s e a r c h -



132

COMPUTER

NETWORKING AND

CHEMISTRY

ed t o produce a 100 i t e m t e s t i n 1.07 minutes t o t a l time, i n c l u d i n g I/O, a t a c o s t o f $3.46. The u s e r i s p r o v i d e d a copy o f the t e s t w i t h a l l q u e s t i o n s s e q u e n t i a l l y numbered, a c o v e r i d e n t i f i c a t i o n page which a l s o has a t e l l t a l e t o i n s u r e p r i v a c y of t h e e x e r c i s e , a t e s t r e q u e s t page showing the r e q u e s t s u b m i t t e d , and an i t e m s t a t i s t i c s page which l i s t s c o r r e c t answers, i t e m s o u r c e s and o t h e r i n f o r m a t i o n . The c o n t e n t and c l a s s i f i c a t i o n system o f the d a t a banks i s c o n t r o l l e d by the d a t a bank c o o r d i n a t o r s and the SOCRATES d i r e c t o r . Their personal viewpoints w i l l be r e f l e c t e d t o some degree i n the c l a s s i f i c a t i o n system. The C h e m i s t r y bank f o l l o w s a g e n e r a l i z e d p a t t e r n d e r i v e d from the format of i n t r o d u c t o r y t e x t s , s i n c e the p r i n c i p a l use o f the bank i s e n v i s i o n e d to be f o r i n t r o d u c t o r y c o u r s e s w i t h l a r g e e n r o l l m e n t s . As an example o f another approach the U.S. h i s t o r y bank i s a r r a n g e d c h r o n o l o g i c a l l y . I r r e s p e c t i v e of t h e c l a s s i f i c a t i o n used w i t h i n the r e q u i r e d h e i r a r c h y , a l l banks a r e b e i n g d e v e l o p e d so as t o p e r m i t the g r e a t e s t f l e x i b i l i t y and promote u s e r a c c e p t a n c e . Thus items which f a v o r p a r t i c u l a r p e d g o g i c a l t e c h n i q u e s , such as programmed l e a r n i n g , a r e d i s c o u r a g e d . T e s t g e n e r a t i o n i s but one f u n c t i o n o f SOCRATES F i g u r e 1 shows a b l o c k diagram o f the c o n t r o l p o i n t s and s o f t w a r e modules. The b a t c h t e s t g e n e r a t o r has been d e s c r i b e d above. The i n t e r a c t i v e t e s t g e n e r a t o r (ITG) and s c o r i n g modules can o p e r a t e i n two modes. The f i r s t mode a l l o w s the t e a c h e r t o d e v i s e an e x e r c i s e i n t e r a c t i v e l y from a remote c o n s o l e u s i n g the same s e a r c h parameters as p r o v i d e d i n the b a t c h mode.



9.

WILLIS

AND SEELY

U.S. HIST

φ BATCH TEST GENE RATOR SCORER

Computer

BUS

Assembled

PSYCH

φ INTER ACTIVE TEST GENE RATOR

133

Testing

FCC EXAM]

GUID

TESTS! MEAS

φ

φ

ITG SCORER

INTE GRATED MEDIA *

ADMINISTERED BY TEACHER

A = Content and s c o r i n g determined by teacher. Β = Content and s c o r i n g determined by student. * = Module under development. Figure 1.

Principal SOCRATES modules and control points



134

COMPUTER

NETWORKING AND

CHEMISTRY

A f t e r e d i t i n g o u t p u t can be p r o v i d e d a t the j o b i n i t i a t i o n s i t e (hard c o p y ) , o r by l i n e p r i n t e r from the Los A n g e l e s d a t a p r o c e s s i n g c e n t e r . The second mode a l l o w s the s t u d e n t t o choose items from a group s e l e c t e d by t h e t e a c h e r , and t o answer them f o r c r e d i t . The s t u d e n t has o p t i o n s as t o t h e d i f f i c u l t y (hence p o i n t v a l u e ) and the c o n t e n t (the q u e s t i o n p r e s e n t e d may be r e f u s e d ) o f the e x e r c i s e . The ITG module does not p r o v i d e CAI o r programmed l e a r n i n g i n the sense t h a t s o l u t i o n h i n t s a r e not p r o v i d e d , and the c o n t e n t of subsequent q u e s t i o n s i s not n e c e s s a r i l y c o n t i n g e n t on s t u d e n t r e s p o n s e . ITG s c o r i n g i s p r o v i d e d by a s e p a r a t e module. The i n t e g r a t e d media module (IM) w i l l be developed t o use t y p e s of enhanced items i n the d a t a bank which r e q u i r e the use of i n f o r m a t i o n o r d e v i c e s m a i n t a i n e d o u t s i d e the SOCRATES d a t a bank, and t h e r e f o r e not d i r e c t l y under c o n t r o l o f the s o f t w a r e . P r e s e n t l y a l l enhanced items i n each bank a r e keyed t o r e s o u r c e f i g u r e f i l e s , c o p i e s o f which a r e p r o v i d e d f o r each SOCRATES u s e r on r e q u e s t . These a r e t y p i c a l l y l i n e drawings s p e c i f i c a l l y p r e p a r e d f o r ease o f d u p l i c a t i o n by Xerox o r s p i r i t p r o c e s s masters. The IM module would a l l o w u s e r s t o i n c l u d e q u e s t i o n s based on l o c a l r e s o u r c e s such as TV t a p e s , s l i d e s , f i l m l o o p s , e t c , t h i s i s , m a t e r i a l s which would not n o r m a l l y be d u p l i c a t e d and g i v e n t o s t u d e n t s i n d i v i d u a l l y i n o r d e r f o r them t o complete t h e e x e r c i s e (e.g., p e r i o d i c t a b l e s ) . Computing Hardware i n the C a l i f o r n i a S t a t e U n i v e r s i t y and C o l l e g e s CSUC The hardware c u r r e n t l y i n use i s shown i n T a b l e I I I . Networking i s l i m i t e d f o r data processing to computers and the S t a t e U n i v e r s i t y i n Los A n g e l e s .

i n t h e CSUC system i n t h e u s u a l y sense l i n t e between l o c a l Data C e n t e r (SUDC)



9.

WILLIS

AND SEELY

Computer

Assembled

135

Testing

T a b l e I I I . Computer Hardware i n t h e C a l i f o r n i a U n i v e r s i t y and C o l l e g e s System Time Enrollment Sharing Other CDC Campus (Fall, 1974) Ports* Long Beach 3150 11 31, 228 IBM 360/40 30, 564 12 San Diego 3300 26, 829 7 San Jose 3170 11 Northridge 25, 377 3150 23, 679 Los Angeles 7 12 3150 San Francisco 20, 855 Fullerton 7 3150 20, 053 8 3150 Sacramento 19, 662 8 3150 Fresno 15, 331 IBM 360/40 7 San Luis Obispo 14, 434 3150 Chico 6 12, 680 3150 Hayward 8 11, 711 3150 Pomona 6 11, 099 3150 3 Humbolt 7,290 NCR 200 Sonoma 2 5,798 Honeywell Dcminguez H i l l s 2 5,747 Honeywell 2 San Bernardino 3,501 Honeywell Bakersfield 1 2,897 Honeywell 2 Stanislaus 2,800 3300, State University 3300 Data Center Totals

291, 916

112

14

State

20/20 20/20 20/20 20/20

7

*A11 t i m e s h a r i n g o p e r a t i o n s use t h e CDC 3170 memory and c e n t r a l p r o c e s s o r a t t h e N o r t h r i d g e Campus; p o r t s may have t e l e t y p e , CRT, o r b o t h t y p e s o f t e r m i n a l s ; a c c e s s i s c o n t r o l l e d by t h e SUDC CDC 3300 executive.



136

COMPUTER

NETWORKING AND

CHEMISTRY

Data f i l e s i n o t h e r computers may be a c c e s s e d i n d i r e c t l y t h r o u g h the e x e c u t i v e CDC 3300 a t SUDC by p r e arrangement. The demand f o r memory o r c o m p u t a t i o n a l speed t o d a t e has been met by t h i s c o m b i n a t i o n , so t h a t d i r e c t l i n k between campus computers has not been needed. However, t h e demand f o r t i m e s h a r i n g o p e r a t i o n s has f a r exceeded c a p a c i t y , p a r t i a l l y because o f the low t r a n s m i s s i o n r a t e s (110 baud) d i c t a t e d by t h e e x i s t i n g equipment which must span some 1200 km (maximum) between t e r m i n a l s . To meet t h i s need, H e w l e t t P a c k a r d 2000 computers w i l l be i n s t a l l ed l o c a l l y f o r hard w i r e d t i m e s h a r i n g a t h i g h e r d a t a t r a n s m i s s i o n r a t e s . The impact o f t h e s e f a c t o r s on SOCRATES use and networking w i l l be d i s c u s s e d i n a following section. The

Chemistry

Data Bank

The c h e m i s t r y d a t a bank was f i r s t c o m p i l e d i n 1972-73 u s i n g items donated by the E d u c a t i o n a l T e s t i n g S e r v i c e and the Monmouth, Oregon C i t y S c h o o l s . The m o d i f i c a t i o n of t h e s e e x i s t i n g q u e s t i o n s a l l o w e d us t o q u i c k l y d e v e l o p t h e bank t o a u s e f u l s i z e . The p r o j e c t was supported by an I n n o v a t i v e Funds G r a n t from the O f f i c e o f t h e C h a n c e l l o r , C a l i f o r n i a S t a t e U n i v e r s i t y and C o l l e g e s . The i n i t i a l i n t e n t was t o d e v e l o p items c o v e r i n g the f i r s t y e a r o f c o l l e g e l e v e l chemistry. A t t h e p r o j e c t t e r m i n a t i o n d a t e , the bank c o n t a i n e d 4000 s e p a r a t e q u e s t i o n s , p r i m a r i l y of the m a c h i n e - g r a d a b l e m u l t i p l e c h o i c e t y p e . The manpower r e q u i r e d f o r p r o d u c t i o n was as f o l l o w s : 1/2 manyear f o r c l a s s i f y i n g and p r o o f r e a d i n g keypunched items; 1/2 manyear o f keypunch time; 1/2 man y e a r f o r c o o r d i n a t i o n o f t h e s e e f f o r t s and d a t a p r o c e s s i n g . (The SOCRATES s o f t w a r e system i t s e l f has r e q u i r e d some 5 manyears f o r development t o d a t e ) . In t h e p a s t y e a r the bank has been expanded t o 10,000 i t e m s . Itemë have been donated by E a s t e r n M i c h i g a n U n i v e r s i t y , M i c h i g a n S t a t e U n i v e r s i t y , and t h e U n i v e r s i t y o f P i t t s b u r g . T h i s e x p a n s i o n has r e q u i r e d approxi m a t e l y 3 manyears o f e f f o r t . The bank i s now b e i n g used by f i v e p e r c e n t of the r e g i s t e r e d u s e r s each day t o produce q u i z z e s , homework a s s i g n m e n t s , t u t o r i a l s , d i a g n o s t i c t e s t s , c h a l l e n g e e x a m i n a t i o n s , and f i n a l e x a m i n a t i o n s . Other bank uses (over twenty have been r e p o r t e d ) i n c l u d e p r o d u c t i o n o f c o p i e s o f the e n t i r e bank on paper on m i c r o f i c h e f o r s t u d e n t r e v i e w i n l i b r a r i e s o r l e a r n i n g c e n t e r , and p r o d u c t i o n o f b l o c k s o f " u n s u i t a b l e " q u e s t i o n s which a r e g i v e n t o s t u d e n t s t o



9.

WILLIS

AND

SEELY

Computer

Assembled

Testing

137

c r i t i q u e and improve. A p o p u l a r f e a t u r e i s the p r o d u c t i o n o f a g e n e r a l c h e m i s t r y placement examin a t i o n which i s r e g u l a r l y a d m i n i s t e r e d t o a l l f i r s t time c h e m i s t r y s t u d e n t s a t f o u r campuses. T h i s has g r e a t l y f a c i l i t a t e d comparisons o f s t u d e n t p o p u l a t i o n s , and has l e d t o a study o f the p o t e n t i a l o f the placement t e s t as a c o u n s e l i n g t o o l f o r incoming freshmen. One c h r o n i c problem c r e a t e d by the d i v e r s i t y of hardware w i t h i n the system and the language r e q u i r e ments of c h e m i s t r y , p h y s i c s , and mathematics has r e v o l v e d around use o f s t a n d a r d b u s i n e s s p r i n t c h a i n s on l i n e p r i n t e r s . Many commonly used s c i e n i t i c symbols a r e not a v a i l a b l e , most i m p o r t a n t l y sub and s u p e r s c r i p t s and lower c a s e l e t t e r s . We have d e v e l o p e d a s e t o f c o n v e r s i o n s which i s s e r v i c e a b l e , and meets the f o l l o w i n g c r i t e r i a : (1) easy t o l e a r n ; (2) uses s t a n d a r d b u s i n e s s p r i n t c h a i n s ; (3) r e a d i l y i n t e r c o n v e r t i b l e t o y i e l d s i m i l a r o u t p u t by l o c a l computer s o f t w a r e ; (4) w i l l r e q u i r e m i n i m a l r e f o r mating as a p p r o p r i a t e p r i n t c h a i n s a r e a c q u i r e d . The c o n v e n t i o n s which have been used a r e perhaps d i s c o n c e r t i n g a t f i r s t g l a n c e t o t h e u s e r who has a l o n g - s t a n d i n g f a m i l i a r i t y w i t h commonly a c c e p t e d c h e m i c a l r e p r e s e n t a t i o n s , but students r e a d i l y l e a r n these conventions. A few h i g h l i g h t s f o l l o w as examples. E l e m e n t a l symbols ( t w o - l e t t e r ) a r e under l i n e d : thus c o b a l t i s w r i t t e n as CO which d i s t i n g u i s h e s i t from c a r b o n monoxide. S u b s c r i p t s a r e s h i f t e d a l i n e down, and s u p e r s c r i p t s a l i n e up: 238 H 50 ; ΤΗ . E q u i l i b r i u m i s r e p r e s e n t e d by , 2 4 resonance by < >, and r e a c t i o n by > . Questions u s i n g e l e c t r o n d o t f o r m u l a s o r Kekule s t r u c t u r e s a r e b e t e n o i r e o f the s t a n d a r d p r i n t c h a i n c h a r a c t e r s e t , and have g e n e r a l y been s u p p l i e d s e p a r a t e l y as enhanced items w i t h p r e p a r e d f i g u r e s . 1

Usage o f the C h e m i s t r y

Data Bank

Most t e s t r e q u e s t s a r e made v i a v o i c e t e l e p h o n e d i r e c t l y t o SUDC, w i t h a s m a l l e r p e r c e n t a g e i n i t i a t e d by remote b a t c h . S i n c e a t y p i c a l e x e r c i s e may c o n t a i n 1000 l i n e s , most u s e r s e l e c t t h e s e o p t i o n s r a t h e r t h a n i n t e r a c t i v e t e s t g e n e r a t i o n which must p r o c e e d a t 110 baud. The a c t u a l amount of t e a c h e r time i n v o l v e d i s perhaps s l i g h t l y l e s s than t h a t r e q u i r e d by t r a d i t i o n a l methods, but the f o l l o w i n g advantages ar«crueby SOCRATES usage: (1) t h e t e s t may be



138

COMPUTER

NETWORKING AND CHEMISTRY

c o n v e n i e n t l y e d i t e d any number o f t i m e s ; (2) t h e t e s t i s produced neat and e r r o r - f r e e on r e p r o d u c t i o n m a s t e r s ready f o r p r o c e s s i n g (thus r e d u c i n g t h e amount o f p r o o f r e a d i n g and t y p i n g n e c e s s a r y ) ; (3) t h e t e s t i s machine graded; (4) s t a t i s t i c s a r e p r o v i d e d on s t u d e n t performance. The n e t r e s u l t i s t h a t t e a c h e r s spend t h e i r time i n two o f t h e t a s k s most d e s e r v i n g o f t h e i r a t t e n t i o n and e x p e r t i s e , namely s e l e c t i n g the c o n t e n t o f t h e e x e r c i s e and e v a l u a t i n g s t u d e n t p e r f o r m a n c e . T h i s s h o u l d p r o v i d e more time f o r t h e t e a c h e r t o spend i n s c h o l a r l y and academic endeavor, and o t h e r w i s e b e n e f i t the student. However, a c c e p t a n c e o f SOCRATES, i f demonstrated by i n c r e a s i n g usage, i s slow t o come. O b v i o u s l y p r o f e s s o r s do n o t have t h e same degree o f f a m i l i a r i t y w i t h e d u c a t i o n a l computer usage as w i t h r e s e a r c h o r i e n t e d usage, o r w i t h computer produced o u t p u t as w i t h m a t e r i a l s produced by s e c r e t a r i a l s t a f f . Complete, hands-on, immediate c o n t r o l o f a l l phases o f e x e r c i s e g e n e r a t i o n by t h e i n d i v i d u a l i s a s t r o n g h a b i t . How d e e p l y t h i s p r o c e s s i s i n g r a i n e d i n us i s amply demonstrated by some f a c u l t y who r e q u e s t t e s t s , perhaps e d i t once t o add and/or d e l e t e q u e s t i o n s , o b t a i n t h e f i n a l o u t p u t on r e p r o d u c t i o n m a s t e r s , then have t h e s e c r e t a r y r e t y p e t h e e n t i r e e x e r c i s e w i t h minor wording changes. Others p r e f e r t o examine a l i s t i n g o f t h e e n t i r e bank cont e n t s , s e l e c t i t e m s , and have a s e c r e t a r y type t h e e x e r c i s e d i r e c t l y from t h e l i s t i n g . Our e x p e r i e n c e has shown t h a t s c i e n c e t e a c h e r s , as opposed t o l i b e r a l a r t s t e a c h e r s , a r e more r e c e p t i v e t o t h e i d e a o f "computerized" t e s t i n g . This i s due t o two f a c t o r s : t h e s c i e n t i s t o f times has a c l o s e r a c q u a i n t e n c e w i t h t h e c a p a b i l i t i e s , modes, and l i m i t s o f computer usage; t h e n a t u r e o f s c i e n t i f i c study which l e n d s i t s e l f more r e a d i l y t o a q u a n t i t a t i v e approach t o t h e s u b j e c t , which f a c i l i t a t e s development and use o f i t e m banks s u i t a b l e f o r computer a d m i n i s t r a t i o n . The c h a l l e n g e t h e r e f o r e l i e s i n d e v e l o p i n g q u e s t i o n s which a r e b o t h p e d a g o g i c a l l y e f f e c t i v e and computer managable (e.g., machine g r a d a b l e ) . We have encountered a wide spectrum o f r e a c t i o n s t o SOCRATES machine g r a d a b l e q u e s t i o n format among e d u c a t o r s . Many t e s t i n g and c o u n s e l l i n g o f f i c e r s and community c o l l e g e t e a c h e r s a r e e x t r e m e l y enthusiastic. P r a c t i c a l l y a l l o f t h e c h e m i s t s we have t a l k e d t o f e e l t h a t our i t e m bank, i f n o t t h e e n t i r e SOCRATES system, i s a v a l u a b l e r e s o u r c e f o r t r a i n i n g t h e i r s t u d e n t s . O f t e n times t h e most



9.

WILLIS

A N D SEELY

Computer

Assembled

139

Testing

e n t h u s i a s t i c s u p p o r t e r s a r e those t e a c h e r s who have a l r e a d y d e v e l o p e d t h e i r own p e r s o n a l i t e m banks. Although o c c a s i o n a l o b j e c t i o n s are r a i s e d t o question phrasing or p r i n t e r conventions, people a c t i v e l y usi n g t h e banks items havfe found t h e i r s t u d e n t s have l i t t l e d i f f i c u l t y i n a d a p t i n g t o computer produced materials. A p o p u l a r f e a t u r e o f f e r e d by SOCRATES i s t h e m u l t i p l e v e r s i o n o p t i o n . Up t o n i n e v e r s i o n s o f t h e same e x e r c i s e c a n be g e n e r a t e d , e d i t e d , and s c o r e d as e a s i l y as one. The l a r g e s i z e o f t h e c h e m i s t r y bank p e r m i t s m u l t i p l e exams t o be d e v e l o p e d on t h e same s u b j e c t m a t t e r f o r use by c o u r s e s e c t i o n s meeting at d i f f e r e n t times. One c r i t i c i s m o f SOCRATES i s r a i s e d a t u s e r workshops and i n f o r m a t i o n a l meetings which d e s e r v e s some comment, as i t a l s o r e f l e c t s h a b i t u a t i o n t o t h e t r a d i t i o n a l t e s t i n g p r o c e s s as w e l l as a p h i l o s o p h i c a l viewpoint. Concern i s e x p r e s s e d about t h e e f f e c t i v e ness o f o b j e c t i v e m u l t i p l e c h o i c e q u e s t i o n s as t e a c h i n g and e v a l u a t i o n t o o l s i n s u b j e c t a r e a s such as c h e m i s t r y where g r e a t emphasis i s p l a c e d on d e v e l o p i n g t h e s t u d e n t s a b s t r a c t r e a s o i n g and conceptualizing s k i l l s . Many o f us have v e r y d e f i n i t e g o a l s f o r t h e c o n t e n t , p h i l o s o p h y , and a p p r o p r i a t e language f o r q u e s t i o n s . These f r e q u e n t l y i n v o l v e a d e m o n s t r a t i o n by t h e s t u d e n t o f some l o g i c a l s e t of o p e r a t i o n s which r e l a t e t o t h e s c i e n t i f i c p r i n c i p l e we a r e a s k i n g about. The a b i l i t y o f m u l t i p l e c h o i c e q u e s t i o n s t o meet t h e s e g o a l s depends on t h e i r d e s i g n , r a t h e r than t h e i r i n h e r e n t approach. By c a r e f u l c h o i c e o f o p t i o n s , i n c o r r e c t r e s p o n s e s c a n be made t o p i n p o i n t s p e c i f i c e r r o r s i n s t u d e n t s reasoni n g o r approach t o a problem, a t any l e v e l . Thus an e x a m i n a t i o n o f t h e i t e m s t a t i s t i c s ( s t u d e n t responses) to an e x e r c i s e c a n t e l l n o t o n l y g e n e r a l s t u d e n t performance, b u t i d e n t i f y i n d i v i d u a l déficiences a s well. T h i s i s not t o say t h a t m u l t i p l e c h o i c e quest i o n s w i l l s u p p l a n t t r a d i t i o n a l s u b j e c t i v e items such as e s s a y s , e t c . Indeed, t h e r e s e r v e d c a t e g o r y i n t h e c l a s s i f i c a t i o n system ( 0 0 0 0 0 ) i n c l u d e s t h e s e . However, o u r e x p e r i e n c e i s t h a t they a r e an e f f e c t i v e t e a c h i n g t o o l , as demonstrated by t h e f a c t t h a t s t u d e n t s from SOCRATES a i d e d p r e r e q u i s t e c l a s s e s p e r f o r m as w e l l i n subsequent c l a s s e s as o t h e r students. We a r e n o t d i s c o u r a g e d by t h e slow a c c e p t a n c e o f the u t i l i t y o f i t e m banks. They h o l d g r e a t promise f o r a l l o w i n g us t o do what we a l r e a d y do by a u t o mating c e r t a i n non c r u c i a l s t e p s . As t e a c h e r s we 1


140

COMPUTER

NETWORKING AND

CHEMISTRY

can thus r e a c h more s t u d e n t s , more r a p i d l y , more i n e x p e n s i v e l y , w i t h fewer e r r o r s , and a t many v a r i o u s levels. We view SOCRATES as a t o o l t o a l l o w us t o r e t u r n t o t h e i n d i v i d u a l s t u d e n t and h i s problems i n t h i s day of mass e d u c a t i o n .


The

F u t u r e o f SOCRATES-Type Programs i n Networks

S i n c e SOCRATES i s b a s i c a l l y a d a t a management system u s i n g a m o d e r a t e l y s i z e d d a t a base, the main advantages of networking l i e i n p r o v i d i n g r a p i d s e r v i c e to a large geographical area. T h i s c o u l d be extended g l o b a l l y v i a s a t e l l i t e u s i n g the EDUCOM, CYBERNET, TYMSHARE, o r s i m i l a r f a c i l i t i e s . Phone l i n k s have a l r e a d y been used t o demonstrate SOCRATES i n P e n n s y l v a n i a , I l l i n o i s , and Texas; s a t e l l i t e r e l a y d e m o n s t r a t i o n s a r e planned i n F r a n c e and Mexico. The q u e s t i o n t h e r e f o r e i s not one o f t e c h n o l o g y , but o f u s e r demand. The c u r r e n t n a t i o n a l t r e n d i s f o r p a r a l l e l independent development o f i t e m banks. We have honored a number o f r e q u e s t s f o r c o n s o l e paper o r magnetic tape l i s t i n g s o f t h e c h e m i s t r y bank, and encourage the f r e e exchange o f i t e m s . However, such exchanges a r e a t times troublesome due t o i n t e r s c h o o l hardware and/or s o f t w a r e d i s p a r i t i e s . The problem o f s t a n d a r d i z a t i o n t o f a c i l i t a t e t r a n s f e r a b i l i t y has been r e c o g n i z e d and i s b e i n g a d d r e s s e d by groups such as CONDUIT. S u f f i c e i t t o say here t h a t good p l a n n i n g and communication i s needed t o p r e p a r e independent banks and s o f t w a r e f o r an easy merge as the need a r i s e s . In the immediate f u t u r e t o t a l d u p l i c a t i o n and t r a n s f e r of items banks w i l l be the r u l e , s i n c e most e d u c a t i o n a l i n s t i t u t i o n s p r e f e r t o s u p p o r t t h e i r own independent o p e r a t i o n s . The need f o r networking t o f a c i l i t a t e t i m e s h a r i n g o p e r a t i o n s such as t h e SOCRATES ITG module, o r g e n e r a l computer a s s i s t e d i n s t r u c t i o n (CAI) i s quest i o n a b l e f o r the near f u t u r e , p r i m a r i l y due t o hardware c o s t s o f p r o v i d i n g p o r t s and t e r m i n a l s f o r even modest c l a s s s i z e s . Typical installations in t h e CSUC system have one t o t h r e e t e r m i n a l s a t each port. P o r t s a r e l o c a t e d i n a c c e s s i b l e p l a c e s so t h a t s t u d e n t s may use t i m e s h a r i n g d u r i n g t h e i r f r e e p e r i o d s , and i n s t r u c t i o n a l use i s planned on t h a t b a s i s . Since system t i m e s h a r i n g use i s heavy (15,000 hr/month average connect time f o r the F a l l 1974 semester, 22,000 hr/month peak) minicomputers a r e b e i n g i n s t a l l ed on each campus f o r l o c a l i z e d t i m e s h a r i n g . Language and a c t i v e s t o r a g e r e s o u r c e s w i l l be l i m i t e d , so SOCRATES cannot be l o c a l l y implemented. It is felt



9. Willis and seely

Computer Assembled Testing

141

that a l l SOCRATES banks could be implemented using the disc space released after transfer of certain operations to the local minicomputers. We envision that SOCRATES operations w i l l evolve primarily using RJE (remote job entry via local computer to the SUDC CDC 3300 executive) and either local line printers for remote areas, or a central line printer at SUDC and twice daily courier service for the Los Angeles service area. As with any evolving system, new uses w i l l be added as the demand for them arises. Management features such as preparing class grade l i s t s and some format options have already been discussed. Users may elect to write their own programs using the item banks as a resource, but the lessons learned from PLATO experience should be obvious: from 50 to 200 hours of programming effort are required to produce one hour of interactive program. Perhaps an observation by Marshall McLuhan w i l l also appropriately reflect the coming state of both the SOCRATES effort, and educational computer networking for information retrieval: "In the age of Xerox, every man is a publisher". With the advent of large accessible data bases, teachers can readily publish a wide variety of truly educational aids according to the dictates of their personal standards. Literature Cited 1. Lippey, G . , Toggenburger, F., and Brown, C.D., Assoc. for Educ. Data Syst. Journal (1971), March, p. 75.


Computer Assembled Testing in a Large Network: The SOCRATES

Recommend Documents