Exploration of a Method To Assess Children's Understandings of a

Article pubs.acs.org/jchemeduc

Exploration of a Method To Assess Children’s Understandings of a Phenomenon after Viewing a Demonstration Show Brittland K. DeKorver,*,† Mark Choi, and Marcy Towns Department of Chemistry, Purdue University, West Lafayette, Indiana 47907, United States S Supporting Information *

ABSTRACT: Chemical demonstration shows are a popular form of informal science education (ISE), employed by schools, museums, and other institutions in order to improve the public’s understanding of science. Just as teachers employ formative and summative assessments in the science classroom to evaluate the impacts of their efforts, it is important to assess learning outcomes in ISE settings. However, assessment in ISE, which necessarily takes place outside of a classroom, presents unique challenges. Designing assessment forces the researcher to find a balance between a rigorous assessment that will yield insightful results while avoiding any measures that might violate the free-choice, leisure atmosphere of ISE activities and risk alienation of participants. This research explored a potential method for evaluating the effects of a demonstration show on audience understanding of a chemical concept. Two formats for presenting demonstration shows, the traditional format and the Fusion Science Theater format, were compared via brief interviews with child audience members about their understanding of a molecular-level explanation for bouncing. KEYWORDS: General Public, Elementary/Middle School Science, Inquiry-Based/Discovery Learning, Demonstrations, Public Understanding/Outreach, Misconceptions/Discrepant Events, Interdisciplinary/Multidisciplinary, Analogies/Transfer, Chemical Education Research, Student-Centered Learning FEATURE: Chemical Education Research

C

such as attempting to sequence or control the educational experience, collecting pretest and post-test data, or traditional assessments such as tests, are poorly aligned with the structure of informal environments and have even been described as a threat to the experience.11 In addition to the restrictions on assessment techniques, there are additional barriers to recruiting participants in an informal science education (ISE) setting. Learners in informal environments have an expectation of leisure and enjoyment, and may resist situations in which they may be made to feel uncomfortable due to lack of knowledge.10 Furthermore, researchers have reported difficulty in recruiting participants at the end of an ISE experience, often because there are other exhibits or activities that the audience wishes to pursue.11 To overcome these difficulties, researchers of ISE experiences have worked with or within formal school settings to recruit participants.12,13 Peleg interviewed the children audience members of a theatrical science play to assess the children’s conceptual understandings about the nature of matter.13 They interviewed 32 children over the course of 17 interviews lasting 20−40 min each. Approximately one-quarter of the interview

hemical demonstration shows are popular among audiences and presenters alike. Many creative variations of demonstration shows have been reported, aiming to achieve learning outcomes such as generating interest and excitement about chemistry and teaching about chemistry concepts.1−8 In order to evaluate whether programs such as demonstration shows are meeting their objectives, some assessment, or measurement of the learning outcomes, must be conducted.9 Assessments are important not only because they provide insights into the outcomes of an educational program, but because they can be used to improve future implementations of the program. In school settings, assessments can take the form of scores and grades. However, demonstration shows are frequently carried out in informal learning environments such as museums, libraries, university campuses, and other venues open to the public. These settings do not accommodate the educational assessment methods that are routinely used in schools and pose unique challenges to measuring their learning outcomes.10 This study was carried out to explore how short interviews can be used to assess the learning outcomes of chemical demonstration shows.

■

DIFFICULTIES OF ASSESSMENT IN INFORMAL SETTINGS Conducting assessments in informal settings presents a unique set of challenges. Standard techniques in educational research, © XXXX American Chemical Society and Division of Chemical Education, Inc.

Received: July 12, 2016 Revised: October 25, 2016

A

DOI: 10.1021/acs.jchemed.6b00506 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

of the wide range of differences in learners’ experiences and abilities and capture the great demographic variability of an ISE audience.19,20 One avenue for this type of data is an openended or semistructured interview, especially for children audience members who may not be able to fully express their ideas via written responses. The goal of this study was to explore the use of interviews as a data source for investigating children’s understanding of a chemistry concept after viewing a demonstration show, focusing on these particular research questions: • What insights into children audience members’ understanding of a chemistry concept can be offered by brief interviews conducted after a demonstration show? • What are the advantages and disadvantages to this method?

prompts were related to the child’s conceptual understanding of the target concept, with the remaining questions focusing on recall of the show and affective measures. The play was carried out at a school during instructional time, with the children’s teachers selecting a mixed-ability group of “good informants” to participate in the interviews.13 The data gained from these interviews, along with a pre- and postshow questionnaire, allowed evaluation of the play’s effect on learners’ knowledge of the chemistry concepts and their attitudes toward science. Despite its convenience, data collection via collaboration with schoolteachers imparts serious risks to the validity of the results. Attitudes, behaviors, and practices carried out in the formal educational setting of a school are greatly different than those in ISE settings (Table 1). Unlike students in a school

■

Table 1. Comparative Features of Instructional and Research Activities in Formal and Informal Learning Enviornments

METHODS

Presentation of Shows Element of Instruction or Research Amount of time spent on activity Return visits Assessment of conceptual knowledge Minimum expected outcomes Recruitment of participants Assessment of conceptual knowledge

Formal (School) Science Education

Two formats of demonstration shows were presented as part of a larger science festival event held on a university campus attended by children and their guardians from the local communities. Dozens of other learning activities, such as handson activities, informational booths, competitions, and other exhibitions, were being held concurrently. The two formats of demonstration shows were presented simultaneously in adjacent rooms with groups of audience members directed to each show in an attempt to achieve a roughly equal distribution of age and number of audience members. The selection of two formats increased the potential for a wide range of learning outcomes to be observed during the interviews. Approximately 40 children audience members of ages 4−12 attended each type of show, along with their respective adult guardians. The demonstration shows that were used in this study targeted the cross-cutting concept that the molecular interactions of an object give rise to its observable characteristics.21 To do this, the two formats of demonstration shows each included the Happy and Sad Balls demonstration.22 Although they look identical to the naked eye, the two polymer balls have different properties. The demonstration consists of dropping both balls from a height; one ball bounces, and the other does not. This phenomenon provides a context for the students to explore submicroscopic causes for their macroscopic observations. One show was of the traditional format, employing the following devices sequentially: (1) introduce the setup of the demonstration, (2) carry out the demonstration, and (3) summarize and explain the results of the demonstration.23 The other show was in the Fusion Science Theater format.18 It began by (1) introducing the setup, (2) posing a question and soliciting an audience prediction, (3) explaining the concept of the demonstration, (4) acting out a kinesthetic model of the phenomenon, (5) soliciting a second audience prediction, and, finally, (6) carrying out the demonstration (see Figure 1). Due to the additional components, the modified format requires more time to present the Happy and Sad Balls demonstration than the traditional format. Because the research goal was to evaluate this method as an assessment tool for ISE programs, it was important to remain faithful to the formats as they are typically performed in ISE settings. Therefore, to create two shows that were the same length (approximately 30 min) and preserve the shows’ format, the traditional demonstration show included several other well-known demonstrations to extend its duration.

Informal Science Education

Decided by instructor

Chosen by participant

Mandated by social norms Regularly incorporated into instructional practices Increased knowledge

Chosen by participant Not usually present

Teacher serves as gatekeeper Corresponds to typical instructional practices

No familiar figure to act as gatekeeper Requires additional investment from participant

Enjoyment

setting, the audience of ISE experiences is free to come and go at their leisure, which may change the duration of the interaction with the learning materials, thus impacting the learning outcomes. Furthermore, the acquisition of data itself would be changed by the transition from a formal to informal learning environment. In a school, the teacher serves as a gatekeeper, facilitating the recruitment and selection of interviewees.14 In an ISE setting, the recruitment of participants is carried out by a relative stranger, i.e., the researcher, rather than a familiar authority figure, i.e., the child’s teacher, and no prior information about the ability level of the participant is available. In addition to influencing the composition of the participant pool, using the participants’ teacher as a gatekeeper may change the way participants relate and respond to the interviewers. Finally, the student population of a particular school or schools cannot represent the diversity of learners that attend the learning activities in informal environments. Although these differences tend to make data acquisition easier in school settings,15 it would be dangerous to assume the results obtained there would be consistent with the same study carried out in a free-choice ISE setting.16 Assessment of Demonstration Shows

Although there is little research on the cognitive impacts of chemical demonstration shows, recent work has explored the use of a single multiple-choice concept question to evaluate the audience members’ understanding of the target chemical concept or concepts.17,18 There have been no reports of assessment that allows audience members of a demonstration show an opportunity for open-ended responses. The richer detail available in open-ended responses would allow for documentation B



Article

Figure 1. Comparison of demonstration show formats.

The well-known Happy and Sad Balls demonstration22 was selected to support the target concepts due to its availability in both formats. This demonstration consists of showing the audience two identical-looking polymer balls. After the audience observes their similarities, the presenter simultaneously drops the two balls from a height onto a flat surface. Due to the different composition of the balls, one bounces and the other does not. The explanation of the phenomenon consisted of three components: that one of the polymers contained extra, or cross-linked, bonds and the other did not (see Figure 2); that

the monomers, and temperature of the balls, none of these factors were described to participants. Demonstration show presenters were recruited from a student organization that consisted of chemistry graduate students. Training of the presenters and other volunteers involved in the production of the demonstration shows was carried out by one of the authors who had previously worked with the Fusion Science Theater25 team and had experience training demonstration performers in both the traditional and FST methods. All presenters were provided with a script and training materials with tips for presenting demonstrations in their respective formats. They met with the researcher for a brief training session (10−30 min) to introduce them to the script and presentation method. After approximately 2 weeks, the presenters each attended a 40−60 min rehearsal with the researcher to practice presenting the show. Two graduate students performed the FST show once each, and one student performed the traditional show twice. Recruitment of Participants

Following each show, the researcher invited audience members to participate in follow-up interviews. The participants and their guardians were offered to schedule the interview immediately after the show or at another mutually agreeable time. A set of Happy and Sad Balls (approximate value $4.35, available from Arbor Scientific) were offered as incentives for each interview participant. In this manner, 14 participants were recruited, 11 of whom were interviewed immediately following their viewing of the show with the remaining 3 interviews conducted within a week (Table 2). The Institutional Review Board granted approval for this study.

Figure 2. Ball-and-stick representations of the balls’ polymer structures, with different spheres representing different monomer units. The top image represents the structure of the polymer that does not have cross-linking, and the bottom represents the structure of the polymer that does have cross-linking.

Data Collection

Data collection was based on the work of Dohn, who evaluated the impact of a field trip to an aquarium on high school students’ situational interest.26 One source of data was short, individual interviews, each lasting only 2−4 min and occurring during participants’ field experiences. These brief, open-ended interviews allowed meaningful data collection with minimal interruption to the participants’ engagement in the activities. This model of brief interviews was adopted for this study in order to minimize the interruption in participants’ enjoyment of the science festival. The semistructured interviews ranged from about 3 to 7 min long.

the cross-linking made the ball less “smushable”; and that because one ball did not “smush” as much on impact, the kinetic energy of the fall was transformed into a bounce whereas the other ball was “smushed” upon impact, with little to no kinetic energy remaining (it is dissipated by heat). This simplified explanation of the complex phenomenon allowed participants to focus on a single target concept, increasing the likelihood of cognitive impact.24 Although other variables also contribute to the balls’ macroscopic properties, e.g., identity of C



Article

explanation, e.g., “used the diagram and the concept of energy to explain the phenomenon”. These descriptions were used to categorize different ways that the students understood the phenomenon.

Table 2. Participant Demographics Participant Namea

Participant Age

Show Typeb

Greg Evan Amy Ellie Abby Robert Tom Allie Carld Matthew Rachel Whitneyd Chris Alexander

6 7 9 9 11 11 11 6 7 8 8 9 10 12

Fusion Science Theater Fusion Science Theaterc Fusion Science Theater Fusion Science Theater Fusion Science Theater Fusion Science Theater Fusion Science Theater Traditional Demonstration Traditional Demonstration Traditional Demonstration Traditional Demonstration Traditional Demonstratione Traditional Demonstratione Traditional Demonstration

■

RESULTS The goal of this study was to investigate the potential for interviews as a method for data collection to measure audience learning after viewing demonstration shows. Thus, this section will present a reflection on the implementation of the method, in addition to the results of the interviews as they pertain to the participants’ understandings of the concepts. Implementation

The interviewer was sensitive to the mood of the participants during the interviews; if they seemed disinterested or consistently answered that they did not know, the interview was ended to avoid causing the participant any anxiety or detracting from their enjoyment of the event. The interviewer offered each participant the option of conducting the interview in the presence of the child’s adult guardian. This seemed to be appreciated by the younger participants, but the older participants did not object to private interviews. One interview was conducted with two participants (Carl and Whitney), a pair of siblings who requested to be interviewed together. Although group interviews have been shown to aid in generating dialogue among the participants and interviewer in informal science education settings,24 it was difficult to differentiate between the two participants’ individual explanations, because they tended to repeat or agree with each others’ statements. In spite of this, requests to be interviewed in the presence of a sibling or parent should be honored in order to minimize any undue anxiety for the participants.10 On the first day of data collection, audience members were invited to sign up for interview times immediately following the show or to provide contact information so that the researcher could schedule an interview to be held at a future time. By scheduling interviews in advance, the time that audience members spent waiting for participation in the research study would be minimized, allowing participants to visit additional exhibits at the science festival. However, no participants volunteered to be interviewed at a later time during the day. Two participants, siblings Carl and Whitney, were interviewed immediately following the show. Three participants, Evan, Chris, and Rachel, provided their information to be contacted for scheduling an interview for a future date. Due to the weak response to the invitation to schedule interviews, the researcher changed the emphasis for the shows on the following day. Instead of scheduling specific interview times, the researcher invited any willing participants to wait in the room of the demonstration show and then interviewed them immediately in a semiprivate alcove near the exit of the room. This was much more enthusiastically received by the audience members, and the remaining participants were recruited in this manner. To aid the reader in determining which participants were recruited by each method and which type of show the participants viewed, this information will be displayed after the participant’s pseudonym in the following format: Name (age in years, type of show with T for traditional and F for Fusion Science Theater, and time elapsed between show and interview in days), e.g., Robert (11, F, 0). The interviewer was able to complete the interview protocol for each of the participants within 3−7 min. Nine such

a Pseudonyms were assigned. bInterviews were held on the same day as the show. cDays elapsed between show and interview: 3. dParticipants were siblings who requested to be interviewed together. eDays elapsed between show and interview: 7.

Participants were asked to summarize their observations of the balls during the demonstration show and explain any differences between the balls’ physical and chemical properties. They were also asked to use diagrams of the balls’ respective polymer structures (see Figure 2) to identify which ball would bounce and which would not (see Supporting Information for the interview protocol). Follow-up questions were asked to probe the participants’ responses. If the participant indicated that they did not know a response, the interviewer attempted to scaffold the answer by offering more guided probes. The interviewer also presented a ball-and-stick diagram of the two polymers (Figure 2) to further probe after the participants’ initial responses. Data Analysis and Theoretical Framework

The goal of this study was to investigate the advantages and disadvantages of this method for describing children audience members’ conceptual understandings. Phenomenography, which seeks to categorize and describe the various ways people perceive and experience a phenomenon,27 was used as a lens for interpreting the interviews to find commonalities and differences among the participants’ responses. A productive assessment method would allow detection of commonalities and trends among the data, while also allowing for differentiation of participants or groups of participants. Phenomenography further informed the recruitment of participants; rather than recruiting a specific type of participants in an effort to control confounding variables, a wide variety of participants were chosen for the study so that we could maximize the range of experiences we measured. Interviews were transcribed within 1 day of being conducted. The transcripts were inductively coded by one researcher. This generated a set of codes: use of the diagram, use of the concept of energy, use of the concept of “smushability”, and alternative explanation. The first three corresponded to the elements employed by the demonstration shows to explain the phenomenon. The fourth encompassed any other ways the student explained the phenomenon. A second researcher independently read the transcripts and applied those codes. The two researchers were in perfect agreement regarding the presence or absence of each of the codes within each transcript. These codes were used to generate descriptions of each participant’s D



Article

Interviewer: The ball. Ok, what if I tell you the circles aren’t balls, but they represent the molecules that make up the ball. Would that make you change your mind? Or think about it differently, that all of these circles represent the molecules? Rachel: [no response] Interviewer: Could you tell me which one of these balls will bounce, and which one will, um be still? Rachel: [shakes head] Interviewer: Don’t know? Rachel: They look a lot alike. Interviewer: They do look a lot alike. Is there any way you could tell without bouncing them? Rachel: [shakes head] Rachel’s (8, T, 7) assignment of macroscopic properties to symbols depicting submicroscopic characteristics is a common confusion for chemistry learners.28 Despite this, she was correct in her indication of which ball was bouncy and which was not. Allie (6, T, 0) also interpreted the diagram incorrectly. When asked to select which diagram represented the bouncy ball, she stated that the cross-linked diagram looked “bigger”, even though the monomer units and diagrams as a whole were the same size. Allie: (points to cross-linked diagram) Interviewer: This one? Why does that one look bigger? Allie: Because I think it’s the bouncy. Interviewer: Why do you think this one is the bouncy one? Allie: Because um, I think bigger makes stuff bounce better. This is indicative of attribute substitution, an associative processing bias.29 The attribute of the number of bonds was more easily accessible to Allie, so she used it as a proxy for an evaluation of the molecular structure. This phenomenon of generating heuristics using surface features of the compound has been reported among undergraduate chemistry students, as well.30 Two siblings who requested to be interviewed together, Carl (7, T, 0) and Whitney (9, T, 0), correctly interpreted the diagrams as molecular representations, although they also provided some incorrect ideas, providing anthropomorphic attributes to the molecules. Interviewer: [Which ball will bounce] do you think Carl? Carl: I think you’d need a microscope that holds it still so you can see if it has the bouncing things in it.

interviews were completed back-to-back at the conclusion of the show by one interviewer. Participants and their families did not seem deterred by the short time they were required to wait in turn. The participants were excited to receive the Happy and Sad Ball set at the end of the interview, and quickly conducted their own spontaneous presentations of the demonstration for the interviewer. Neither the participants nor their guardians gave any indication of frustration at waiting. None of the participants who joined the queue to be interviewed at the end of the show left the line prior to being interviewed. Differentiation of Participants’ Conceptual Understanding

The codes generated during analysis corresponded to the three components of the explanation offered by the presenters: the diagram that depicted cross-linking, the “smushability” of the ball, and the transfer of energy from the drop to a smush or a bounce. When provided with the diagram, 13 of the 14 participants were able to correctly identify which structure belonged to the ball that bounced, yet participants exhibited a wide range in their abilities to correctly explain the phenomenon; however, there were commonalities within that variability. Phenomenographic categories emerged on the basis of the number of components present in participants’ explanations (Table 3). Some participants offered alternative explanations that were incorrect. For example, Chris (10, T, 7) suggested that the difference in the balls was related to density. Other participants offered no explanation and were not able to decode the diagrams to indicate the presence of cross-linking in one of the balls. Rachel: [pointing to one of the diagrams] It looks like, it looks like it’s bouncy. Interviewer: It looks like it’s bouncy. Ok, can you tell me what about it makes it look like it’s bouncy? Rachel: Here, because it has this, it’s like a trick in your eyes, it goes “woop”. Interviewer: What do you think these circles, you said it looks like a trick in your eyes and you pointed to the circle and you went “woop” to the next circle. What do you think those circles are? Rachel: The ball.

Table 3. Participants’ Explanations for the Happy and Sad Balls Demonstration Elements Corresponding to Demonstration Show in Participant’s Explanation

Participant (Gender: boy/girl)

Description

None

The participant did not refer to any of the explanatory elements that were provided by the demonstration shows.

Rachel (g) Greg (b) Allie (g)

Diagram

The participant grounded his or her particulate-level explanation for the demonstration in the diagram, but included alternate explanations or did not provide any other justification.

Carl (b) Whitney (g) Chris (b)

Diagram + 1

The participant used the diagram and one other concept (“smushability” or energy transfer) to explain the outcome of the demonstration.

Tom (b) Alexander (b) Amy (g) Evan (b) Robert (b)

Diagram + 2

The participant used the diagram and both concepts (“smushability” and energy transfer) to explain the outcome of the demonstration.

Ellie (g) Abby (g) Matthew (b)

E



Article

Evan: I forgot. Interviewer: You forgot? That’s ok. Do you know what the circle parts represent? What do those signify? Evan: I forgot.

Interviewer: What if I have pictures diagrams, of what the molecules look like. What if I tell you that these are what the molecules look like? Whitney: [pointing to the cross-linked diagram] this one will bounce. Because it’s connected. Carl: Probably that one. I have to agree with Whitney. Interviewer: You have to agree with Whitney. So what do you think these things are? [pointing to circles] Together: Molecules! Interviewer: What do you think these mean, these lines right here? Carl: Hmmm. Whitney: It’s where the molecules make the, get like a chain, so that they meet? Interviewer: Ok. So the molecule gets a chain so they meet? Whitney: Well they meet, that’s the line that, they meet. Carl: They have eyes on each side so they can meet everybody. Half of the participants supported their explanations with additional concepts, such as the “smushability” or considerations of the ball’s energy. Alexander (12, T, 0) used crosslinking and energy to explain the bounce, but was not able to explain how the two were associated. Interviewer: Ok. Why do you think that this one is the one that did bounce and this one did not? Alexander: Because it has the more energy that it needed, when it bounces, it reflected the energy back up. Interviewer: Ok. What reflects the energy back up? Alexander: The um... I think the air? Interviewer: The air? Ok. Can you tell me what’s similar or different in these pictures? Alexander: This one’s smaller, this one’s bigger, this one has lines in it. Interviewer: What do the lines represent? Alexander: The chains. Interviewer: So um, what do the circles represent? Alexander: The... geez, what’s it called? I think the point of the chain, of where it ends. I think. Interviewer: Ok. Um, so if I tell you that this ball goes with this picture and this ball goes with this picture, does that change your mind at all about which one you think will bounce? Alexander: I know exactly which one is going to bounce. Interviewer: Ok. And can you tell me again why you think you know which one will bounce? Alexander: Because um, that one has chain, the cross-link, and that one doesn’t. Although he called on energy to justify the ball’s rebound, he was unable to describe why the ball’s structure would result in a difference in energy. Robert (11, F, 0) and Evan (7, F, 3) discussed the balls’ “smushability” using the diagram, but did not reference energy. Interviewer: Ok, can you tell me about those pictures? Evan: This one will bounce less and this one will bounce more. Interviewer: Ok. Why do you think this one will bounce less and this one will bounce more? Evan: Because that is the one that will squish because it doesn’t have the two lines. Interviewer: It doesn’t have the two lines so you think it will squish. Evan: Yep. Interviewer: Ok. What do those lines represent?

Robert: This one has two other lines. Interviewer: Two other lines? What do the lines mean? Robert: The structure is, like this one is stronger? Interviewer: Stronger? Robert: Doesn’t smush as much. Interviewer: Doesn’t smush as much. Ok. So if I told you that this ball matches this diagram and this ball matches this diagram, would that help you tell me which one bounces? Robert: [nods] Interviewer: Which one do you think [will bounce]? Robert: [points to correct ball] Alexander, Evan, and Robert considered the diagrams to identify the ball that bounced and used additional information about the energy or “smushability”, but none of the three were able to offer a description in terms of bonding or molecules. Their explanations were primarily relational, where they were able to describe how one variable would affect another variable, but were unable to articulate why the relationship existed.31 In contrast, the three participants who did interpret the diagrams as representing bonds and molecules displayed mechanistic reasoning. Interviewer: So I have some diagrams here. Can you tell me about these diagrams? Abby: This one will not smush together. Because there’s two bonds going across from the two lines. Interviewer: Why do the two bonds make it so that it does not smush together? Abby: They make it stronger. Interviewer: Ok. What does smushing have to do with it? Abby: It won’t have enough kinetic energy to make it go bounce up higher. Matthew: They are made of molecules and one, one had cross, cross-links and the other didn’t, so one had the energy of the fall to go back up and the other didn’t because it had no, the molecules smushed together. Abby, Matthew, and Ellie each discussed the “smushability” and the relative energies, and used the diagram in their explanation of the phenomenon. Their interpretation of the diagram included references to bonds or molecules. Comparisons across Variables

Although the purpose of this study was not to evaluate the relative quality of the two types of shows, an exploration across the variables of age, type of show viewed, and time elapsed between show and interview was made to determine the utility of these interviews for differentiating among groups (Figure 3). In general, younger participants offered less complete explanations, while older participants were more thorough. In addition, none of the participants who participated in the delayed interviews generated a complete explanation. Viewers who saw the demonstration embedded within a traditional demonstration show also had less complete explanations than the viewers of the modified (FST) version of the same demonstration. Limitations

This study was limited by the small data set. There were not enough participants to fully explore the relationships among these variables. However, the goal of this study was not to draw F



Article

Figure 3. Comparison of participants’ explanations of the phenomenon between types of show viewed.

impacts of particular components (e.g., explanations, diagrams, questions) of demonstration shows. ISE experiences provide a new area of chemistry education research in which the public’s understanding of chemistry can be studied.12 The popularity of demonstration shows among chemists indicates this is an especially fertile context for ISE research, that is, if appropriate and valid assessments can be conducted. Prior use of interviews to assess the cognitive impacts of science shows has employed formal school settings as a conduit for data collection. This diminishes the validity of the assessment and is problematic for evaluating these programs’ impacts. Carrying out assessment within a more authentic ISE setting improves the validity of the results. Furthermore, openended interviews provide researchers with the flexibility to capture unanticipated learning outcomes, a key feature of informal learning.10 Beyond serving as a tool for researchers, this type of interview could be used to help determine the allocation of ISE resources. Educators who engage in ISE may be interested to measure their programs’ impact on audience understanding, so that they can choose programs that are aligned with their educational objectives.

comparisons, but to explore how to collect interview data with the audience of demonstration shows in informal settings and to determine the effectiveness of this technique for differentiating among participants. In the future, a larger participant pool or more deliberate selection of interviewees could allow a quasiexperimental study to evaluate the outcomes of demonstration shows.

■

CONCLUSIONS This method of brief interviews immediately following the demonstration show provided enough depth to differentiate the children’s explanations of the concept beyond identifying their ability to make a correct prediction. These interviews were also able to detect reasoning patterns similar to those reported for older students in classroom settings, despite being conducted with children in an ISE setting. This indicated that the interviews were fruitful in providing insights into the children’s understanding of the phenomenon and could be used to evaluate demonstration shows. In addition to the complexity and depth of data that these interviews provided, another benefit of this method was that these data were able to be collected over a relatively short time, less than 10 min per interview. The most significant and novel benefit was the ability to collect data about demonstration shows in an authentic setting, at an ISE event, rather than utilizing school groups or performing within a school setting. One disadvantage of this method for data collection was the need to collect the majority of the interviews with little to no prearrangement in back-to-back sessions, since the children audience members and their guardians were most willing to participate in the interview immediately following the demonstration show. This required stamina and flexibility for a single interviewer to accomplish, although additional interviewers could have relieved some of the burden. The interviews in quick succession did not allow for extended reflection or a preliminary analysis on the initial results. Another disadvantage is the expense of providing a small incentive to each participant, but the incentive seemed to be highly motivating to the child participants. Although the design of this study suggested a comparison between two different types of shows, the variability in age and time elapsed after seeing the show limited the validity of drawing such conclusions. Future studies could employ a more targeted recruitment to investigate the effects on conceptual understanding based on the type of show. Other studies might indicate whether there is an optimal age for this type of learning experience and examine the durability of ISE learning and the

■

ASSOCIATED CONTENT

S Supporting Information *

The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.6b00506. Interview protocol (PDF, DOCX)

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. Present Address †

Lyman Briggs College, Michigan State University, East Lansing, Michigan 48825, United States.

Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS We are grateful to the Nu chapter of Phi Lamda Upsilon for providing funding for the demonstration shows and to the members who volunteered their time to perform and produce the demonstration shows. Funding for the participation incentives was provided by Purdue’s Student Activities and Organizations. We would also like to thank Holly Walter Kerby and Fusion Science Theater for allowing us to use their script and training G



Article

(26) Dohn, N. B. Situational Interest of High School Students Who Visit an Aquarium. Sci. Educ. 2011, 95, 337−357. (27) Marton, F. Phenomenography: A Research Approach to Investigating Different Understandings of Reality. J. Thought 1986, 21, 28−49. (28) Johnstone, A. H. Why Is Science Difficult to Learn? Things Are Seldom What They Seem. J. Comput. Assist. Learn. 1991, 7, 75−83. (29) Talanquer, V. Chemistry Education: Ten Heuristics to Tame. J. Chem. Educ. 2014, 91, 1091−1097. (30) Cooper, M. M.; Corley, L. M.; Underwood, S. M. An Investigation of College Chemistry Students’ Understanding of Structure-Property Relationships. J. Res. Sci. Teach. 2013, 50, 699−721. (31) Sevian, H.; Talanquer, V. Rethinking Chemistry: A Learning Progression on Chemical Thinking. Chem. Educ. Res. Pract. 2014, 15, 10−23.

materials. Finally, we are grateful to the anonymous reviewers who provided thoughtful comments on this manuscript.

■

REFERENCES

(1) Bailey, P. S.; Bailey, C. A.; Andersen, J.; Koski, P. G.; Rechsteiner, C. Producing a Chemistry Magic Show. J. Chem. Educ. 1975, 52, 524− 525. (2) Bergmeier, B. D.; Saunders, S. R. The Chemistry Magic and Safety Show. J. Chem. Educ. 1982, 59, 529. (3) Gammon, S. D. The Twelve Days of Chemistry: A Model for a Chemistry Demonstration Show. J. Chem. Educ. 1994, 71, 1077−1079. (4) Hanson, R. H. Chemistry is Fun, Not Magic. J. Chem. Educ. 1976, 53, 577−578. (5) Harrison, A. M. Show ‘n’ Tell. J. Chem. Educ. 1971, 48, 612−613. (6) Ihde, J. Fun-Audience Participation Polymerization Demonstrations. J. Chem. Educ. 1990, 67, 264. (7) Lasry, N. The Magic of Science Through the Science of Magic. Sci. Educ. Rev. 2012, 11, 40−43. (8) Sherman, M. C-H-E-M Spells Chemistry is Fun: An Outline for a Very Involving Chemistry Demonstration. J. Chem. Educ. 1992, 69, 413−415. (9) Astin, A. W.; Antonio, A. L. Assessment for Excellence, 2nd ed.; Rowman & Littlefield Publishers: Lanham, MD, 2012. (10) Learning Science in Informal Environments: People, Places, and Pursuits; Bell, P., Lewenstein, B., Shouse, A. W., Feder, M. A., Eds.; National Academies Press: Washington, DC, 2009. (11) Falk, J. H.; Moussouri, T.; Coulson, D. The Effect of Visitors’ Agendas on Museum Learning. Curator 1998, 41, 107−120. (12) Christian, B. N.; Yezierski, E. J. A New Chemistry Education Research Frontier. J. Chem. Educ. 2012, 89, 1337−1339. (13) Peleg, R.; Baram-Tsabari, A. Atom Surprise: Using Theatre in Primary Science Education. J. Sci. Educ. Technol. 2011, 20, 508−524. (14) King, N.; Horrocks, C. Interviews in Qualitative Research; Sage: Thousand Oaks, CA, 2010; p 31. (15) Mortensen, M.; Smart, K. Free-Choice Worksheets Increase Students’ Exposure to Curriculum During Museum Visits. J. Res. Sci. Teach. 2007, 44, 1389−1414. (16) Dierking, L. D.; Falk, J. H.; Rennie, L.; Anderson, D.; Ellenbogen, K. Policy statement of the “Informal Science Education” ad hoc committee. J. Res. Sci. Teach. 2003, 40, 108−111. (17) Kerby, H. W.; Cantor, J.; Weiland, M.; Babiarz, C.; Kerby, A. W. Fusion Science Theater presents The Amazing Chemical Circus: A New Model of Outreach that Uses Theater to Engage Children in Learning. J. Chem. Educ. 2010, 87, 1024−1030. (18) Kerby, H. W.; Dekorver, B. K.; Cantor, J.; Weiland, M. J.; Babiarz, C. L. Demonstration Show that Promotes and Assesses Conceptual Understanding Using the Structure of Drama. J. Chem. Educ. 2016, 93, 613−618. (19) Brody, M.; Bangert, A.; Dillon, J. Assessing Learning in Informal Science Contexts. Commissioned Paper; National Research Council: Washington, DC, 2008. (20) Framework for Evaluating Impacts of Informal Science Education Projects; Friedman, A., Ed.; National Science Foundation: Arlington, VA, 2008. (21) NGSS Lead States. Next Generation Science Standards; National Academies Press: Washington, DC, 2013. (22) Kauffman, G. B.; Mason, S. W.; Seymour, R. B. Happy and Unhappy balls: Neoprene and Polynorbornene. J. Chem. Educ. 1990, 67, 198−199. (23) Crouch, C.; Fagen, A. P.; Callan, J. P.; Mazur, E. Classroom Demonstrations: Learning Tools or Entertainment? Am. J. Phys. 2004, 72, 835. (24) Sokoloff, D. R.; Thornton, R. K. Using Interactive Lecture Demonstrations to Create an Active Learning Environment. In The Changing Role of Physics Departments in Modern Universities; AIP Publishing: Melville, NY, 1997; pp 1061−1074. (25) Fusion Science Theater. www.fusionsciencelearning.org (accessed October 2016). H


Exploration of a Method To Assess Children's Understandings of a

Recommend Documents