Research: Science and Education
Focus Groups and Exit Interviews Are Components of Chemistry Department Program Assessment Joseph H. Dreisbach Department of Chemistry, University of Scranton, Scranton, PA 18510-4626 Thomas P. Hogan and Anne Marie Stamford Assessment and Institutional Research Office, University of Scranton, Scranton, PA 18510-4629 John W. Greggo Department of Counseling and Human Services, University of Scranton, Scranton, PA 18510-4523
Assessment of academic programs and curricula continues to receive expanding emphasis in higher education. For many programs, especially those with modest or low enrollments, it is often difficult to acquire meaningful, systematic assessment data. In this paper we describe an effective, inexpensive assessment procedure that involves direct contact with students through focus groups and exit interviews. Although structures of assessment plans and support services are quite diverse in different institutions, our approach can easily be modified to fit the needs of any academic setting. The procedures described here were applied within the chemistry department at the University of Scranton. The university is a private institution with total enrollment of about 5000, primarily undergraduate students. Its Carnegie classification is Comprehensive MAI. The chemistry department offers five undergraduate tracks: chemistry, biochemistry, chemistry–computers, chemistry–business, and medical technology. The first three programs conform to the guidelines of the ACS Committee on Professional Training (ACSCPT) and the program in chemistry is certified by ACSCPT. Students in the biochemistry and chemistry–computers program can receive ACS certification in chemistry by taking a few extra courses. All three programs incorporate research and a thesis defense requirement. For the past three years, the programs have graduated an average of 23 students each year. Like many chemistry departments, this one has a substantial service responsibility to other departments. In fall 1996, 314 students took the freshman and sophomore chemistry courses; 86% of these students were from majors outside the department. The chemistry department also offers three master’s degree programs: chemistry, biochemistry, and clinical chemistry. However, the procedures described in this article were applied only to undergraduates; hence no further information is given about these graduate programs. The Assessment Context The procedures described here were undertaken within the context of the university’s overall assessment plan. Part of that plan calls for a periodic review of all academic programs— a requirement that is nearly universal in institutions of higher education within the United States. Miller (1) provides a
1330
standard description of such reviews and Nichols (2) gives a contemporary treatment of the same topics, with useful case studies. The university’s plan also incorporates several other university-wide types of assessment such as surveys of seniors and alumni, analysis of retention, and follow-up of graduates regarding job placement or graduate school entrance. The university’s assessment plan includes explicit encouragement for departments to conceptualize and conduct their own assessment activities; see Hogan and Stamford (3) for a general description of these efforts. The remainder of this article describes the chemistry department’s attempts to customize portions of its own academic program review process within the context of encouraging departmental innovation in assessment. Focus Group Approach The chemistry department had gathered some assessment information over the years but this was in anecdotal form, obtained in a nonsystematic way, and difficult to use with confidence. The department was interested in a systematic and broad assessment of the programs offered to department majors, especially in the junior and senior years, as well as assessment of programs for nonmajors. A qualitative approach was chosen in order to explore undergraduate students’ perceptions of the chemistry program at the University of Scranton. Use of focus groups seemed to be a good approach for gathering the information, to enable undergraduate students to communicate their perceptions in their own words. Focus groups have been useful both for research data collection and for program evaluation (4–7 ). Focus groups are carefully planned discussion groups that explore topics selected and focused by the moderator; they are generally audiotaped or videotaped for ease of data collection and analysis. They utilize small group settings to explore the perspectives of participants in an open and nonthreatening environment. Group members are encouraged to share ideas and to interact freely, responding to mutually generated comments. While guided by the moderator and a few key questions, the group interactions provide spontaneous responses that are particularly appropriate for investigating questions about why individuals act and believe as they do (8). This approach concerns itself with the feelings, opinions, and meaning that participants assign to a particular problem, experience, or other phenomenon (9). Focus group leaders
Journal of Chemical Education • Vol. 75 No. 10 October 1998 • JChemEd.chem.wisc.edu
Research: Science & Education
use their position in the group to facilitate participants’ sharing divergent views, to keep the group focused, and to encourage elaboration of relevant points. The focus group is one of several inductive approaches to elicit information (10). The design enables the trained moderator to identify what group members feel is important and to gain insights into the extent to which their experiences are shared. Understanding the participants’ perspective, as a crucial element in successful programming, increases a program’s sensitivity, acceptability, successful implementation, and maintenance. When used as part of a “formative” evaluation process, focus groups can reveal valuable information, important concerns, and potential barriers to successful implementation. The chemistry department established the goal of obtaining broad assessment information on the effectiveness of its major programs as well as its service components. These assessment objectives dictated separation of students into two sets, with seven focus groups in all and three to six students in each focus group. One set, consisting of four groups, comprised students in the first- and second-year courses. No attempt was made to separate chemistry department majors from nonmajors in this set. A second set, with three groups, comprised junior and senior department majors. The moderation of each focus group was cofacilitated by a pair of practicum students from the university’s graduate counseling programs who were completing a preinternship in the university’s graduate counselor training center. Volunteer undergraduate participants were assigned randomly (by academic year) to a focus group. Group member identity was anonymous, and no references were made to member identity in the final focus group reports written by the practicum students. A “consent statement” describing anonymity of responses was read into the audiotape at the beginning of each focus group. Each counseling practicum student had previously completed a graduate Group Process and Practice course and received focus group training as a component of the spring semester practicum training orientation as well as in the weekly group practicum supervision class. Focus group questions were developed during consultation between the chair of the chemistry program, the director of the counselor training center, and faculty in the department of counseling and human services. Questions were provided to practicum students two weeks in advance of the focus groups in their weekly group practicum supervision class. Separate sets of questions were developed for each set to generate comment and conversation. The questions were written to initiate discussion in relevant areas, but they were sufficiently broad to allow a variety of responses from different student perspectives. Questions for the first- and second-year students focused on their experience with the courses relative to their impressions and expectations from previous exposure to science and chemistry. A question was added to elicit responses on the lecture–laboratory relationship and a final question was added to encourage students to make remarks on chemistry as a potential major. For upper-level department majors, questions were somewhat broader and dealt with students’ perceptions, based on their experience in the department. A third question requesting specific suggestions on ways to “improve the chemistry major” was designed to have students move beyond generalizations and provide concrete
examples of problems and solutions. Final versions of the questions are provided in the box. The focus group project was performed in spring 1995. Summary reports, with names of faculty and students removed, from each focus group facilitating team were submitted to the chemistry department chairman a few weeks after the meetings. Utilizing graduate counseling practicum students to facilitate the chemistry program review focus groups was fortuitous in that counseling students benefited from the training opportunity and group experience while the chemistry program received valuable information on what students perceive as strengths and weaknesses of the program. Results An immediate, and somewhat unexpected, result was the overwhelming appreciation of student participants for the opportunity to play a direct role in the assessment project. We also learned about some important fundamental strengths of the department, and these strengths would serve as the foundation for subsequent development plans. Two of these identified strengths were faculty availability and the quality of the undergraduate research program. For example, students commented that “The chemistry faculty are very approachable” and “the one-to-one interaction with our research mentors helps us understand real chemistry.” Of course we communicated these strengths to the administration to maintain their support of independent study in the department. In addition, such comments helped to reinforce faculty efforts in these areas. We expected considerable criticism and we were not disappointed. It is important to reemphasize that the project was designed, from the outset, to assess programs and not personnel. Thus any citations of names of personnel, even though many of the citations were positive, were not included in the facilitators’ reports. In preparing the reports, facilitators attempted to indicate the broadly affirmed criticisms and reduce emphasis on single comments. The department chair-
Focus Group Questions Questions for the First- and Second-Year Student Focus Groups 1. After two semesters of chemistry, what is your impression of the chemistry program? 2. What was your impression prior to taking these courses? 3. Did your chemistry/laboratory course work change or strengthen this impression? 4. Knowing what you know now, why would you pursue a chemistry major at the University? Questions for the Third- and Fourth-Year Chemistry Department Majors Focus Groups 1. What are the strengths of your chemistry program? 2. How have the laboratory courses complemented the lecture courses in your major? 3. What specific suggestions do you have for improving the chemistry major?
JChemEd.chem.wisc.edu • Vol. 75 No. 10 October 1998 • Journal of Chemical Education
1331
Research: Science and Education
person identified recurring themes by comparing reports from the various focus groups. A number of common criticisms were evident. For example, “Our laboratory courses do not relate to what we are doing in the lecture courses,” or “Our laboratory is tedious and repetitious.” Although some of these difficulties had already been identified by the department, the focus group project provided a direct articulation of the problems from student perspectives. This direct feedback gave us a better understanding of these situations and allowed us to better prioritize these needs and develop approaches to address them. As a result, the department is pursuing a number of immediate and long-range plans to develop undergraduate structured-laboratory courses. Exit Interviews Are Used as Part of the Annual Internal Department Assessment The success of the focus group project resulted in the chemistry department using exit interviews as part of its annual internal department assessment activities. We follow the same basic procedures used for the focus group design when developing the annual exit interview plan. The exit interviews are done internally using only department resources. The required research course is used as the vehicle to inform students of the plan; chemistry–business and medical technology majors who do not take research are notified individually. Preliminary steps require establishing the assessment objective. In 1995 and 1996 our approach was to have broad assessment goals and students were asked about their perceptions of departmental strengths and weaknesses. In 1997 we focused on departmental advising. Students are informed, in writing, of the exit interview process including general goals. They are told that the process is voluntary and confidential. In 1996 and again in 1997, the faculty moderator of the
1332
research course assumed the role of interviewer. During the meeting students are again informed of program goals. The interviewer presents the question and facilitates discussion. Results are reported to the chairperson in summary form without including names of students. The summary report is used to identify trends and recurring comments. Results from the exit interview process are consistent with those from the focus group project. We obtain what we believe to be good, reliable information on the department and the programs. The information is used as part of the basis for short-term as well as longer-range development plans. Literature Cited 1. Miller, R. I. The Assessment of College Performance: A Handbook of Techniques and Measures for Institutional Self-Evaluation; JosseyBass: San Francisco, 1979. 2. Nichols, J. O. Assessment Case Studies: Common Issues in Implementation with Various Campus Approaches to Resolution; Agathon: New York, 1995. 3. Hogan, T. P.; Stamford, A. M. Assessment Update 1997, September/October, p 4. 4. Basch, C. E. Health Educ. Q. 1987, 14, 411. 5. Jacobi, M. NASPA J. 1991, 28, 195. 6. Kuh, G. D.; Andreas, R. E. J. Coll. Student Devel. 1991, 32, 397. 7. Morgan, D. L. Focus Groups as Qualitative Research; Sage: Newbury Park, CA, 1988. 8. Calder, B. J. J. Market. Res. 1977, 14, 353. 9. Mullen, P. D.; Reynolds, R. Health Educ. Monogr. 1978, 6, 280. 10. Collins, B.G.; Collins, T. M.’ Botyrius-Maier, G.; MacIntoshDaeschler, S. J. Mental Health Counseling, 1995, 16, 261.
Journal of Chemical Education • Vol. 75 No. 10 October 1998 • JChemEd.chem.wisc.edu
