Article pubs.acs.org/jchemeduc
Choice of Study Resources in General Chemistry by Students Who Have Little Time To Study Diane M. Bunce,*,† Regis Komperda,‡ Debra K. Dillner,§ Shirley Lin,§ Maria J. Schroeder,§ and JudithAnn R. Hartman§ †
Chemistry Department, The Catholic University, Washington, D.C. 20064, United States Department of Chemistry, Portland State University, Portland, Oregon 97207, United States § Chemistry Department, United States Naval Academy, Annapolis, Maryland 21402, United States ‡
S Supporting Information *
ABSTRACT: Students with an insufficient amount of time to study are becoming more prevalent in the general college population as many who enroll in college have competing responsibilities (full-time jobs, childcare, etc.). Such students are likely to choose study resources that they consider to be both effective and efficient. Students at the U.S. Naval Academy (USNA) are constrained in their study time because of their required course load and competing institutional daily requirements. The purpose of this study was to survey which resources students at USNA choose for studying and to look at the difference in choice in relation to different types of assessments and different student achievement levels in chemistry. Students (n = 1015) were surveyed four times during the Fall 2013 semester after both instructor-written assessments and departmental multiple-choice common exams. In these surveys, students reported the main study resource they used to prepare for each assessment. A subset of students (n = 57) was interviewed soon after completing the third survey to better understand how the students used the resources they reported choosing. The results show a difference in study resources chosen depending on the type of assessment (instructor-written or common exam) and final course achievement level of the student. Application of these results to a broader audience of students who also have multiple time commitments may help chemical educators better format both the availability and content of chemistry study resources to help students of different achievement levels succeed in general chemistry. KEYWORDS: High School/Introductory Chemistry, First-Year Undergraduate/General, Chemical Education Research, Constructivism, Learning Theories, Professional Development FEATURE: Chemical Education Research
■
BACKGROUND All of the approximately 1100 freshmen who enter the U.S. Naval Academy (USNA) each year are required to successfully complete or test out of an eight-credit, two-semester chemistry majors’ level general chemistry course. Although students undergo a rigorous admission process to be accepted into USNA, not all students have the same chemistry background or proficiency when they enter general chemistry. As a result, there can be a wide range of experience and expertise in chemistry knowledge within a given class, just as there are at other colleges and universities. Because of competing institutional daily requirements (military training and 3 hours of physical development), students at USNA have 3 hours per day in their schedules to study for the courses in their 18 credit hour semester course load. A typical daily schedule for midshipmen is included in the Supporting Information. Although the specific demands on the time of the freshmen at USNA might be more extensive than that of students at other colleges and universities, the USNA population for this study can serve as an extreme example of study time constraints and offer insight into the resources chosen by students when they have limited time, whether it is due to institutional requirements, job, and/or family obligations. To help students of various backgrounds succeed in freshman chemistry at USNA, a wealth of study resources have been © 2016 American Chemical Society and Division of Chemical Education, Inc.
developed. Students are made aware of these resources through required briefings by advisors and e-mail reminders. These resources include the following: 1. print and eBook version of the textbook1 2. print copy of the study guide from the textbook publisher 3. departmental chapter learning objectives posted on the departmental Web site 4. access to the online homework (HW) for review after completion 5. instructor notes provided in class either orally or written and possibly posted on Blackboard after class 6. previous year’s departmental multiple-choice common exams and answer keys on the departmental Web site 7. an upperclassman-run, after-hours tutoring session known as the Midshipman Group Study Program (MGSP) 8. extra instructor-provided, on-demand help sessions (EI) 9. optional instructor-run review sessions prior to common exams 10. an institutional Academic Center that provides ongoing free help classes, drop-in tutoring sessions, and personal tutors Received: April 18, 2016 Revised: October 28, 2016 Published: November 23, 2016 11
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
Journal of Chemical Education
Article
the same weekly teaching schedule, textbook, departmental chapter learning objectives, laboratory experiments, and online homework assignments. There are no teaching assistants at USNA, so instructors teach students in both lecture and lab and grade all lab reports, quizzes, and tests. Departmental common exams are administered concurrently to all students enrolled in the course during designated 50 minute testing periods at 6 weeks and 12 weeks and in a 3 hour final exam at the end of the semester. The multiple-choice common exam questions are a combination of questions chosen from a database of approximately 2000 questions maintained by the chemistry department and new questions written each semester by a committee of current general chemistry instructors. All questions included on the exam are rated for topic coverage and level of difficulty to ensure that each common exam will measure both the breadth and depth of the topics covered and represent a scale from easy to difficult in terms of question-level difficulty that is consistent from year to year. Prior to the first common exam at 6 weeks, instructors are encouraged to develop an assessment of their own choosing. This early assessment can take the form of quizzes or tests. In addition, all instructors administer an instructor-written assessment in the last 4 weeks of the semester between the week 12 common exam and the final exam. The format of these instructor-written tests is primarily open-ended questions, but instructors have the leeway to write the assessments using their preferred format, including some multiple-choice, short-answer, problem-solving, and/or open-ended questions. In a survey of the instructors in this study (86% response rate), the average for instructor use of multiple-choice questions was 25% of the test.
Many if not all of these study resources appear at other colleges and universities. Many of the resources listed for our study were also available for the students in a 2015 study by Sinapuelas and Stacy.2 Their study, by contrast, dealt with how these resources were used. We chose as a point of entry into our investigation students’ choice of study resource. We investigated how the resources were used during interviews with a subset of students. The resource chosen by students was also examined in terms of how well it matched the type of assessment for which the students were preparing. We view the choice of study resource as an action on the part of the student that can be directly measured and used as a beginning point for a further investigation of the study approaches used by students. Surveys to measure student-chosen study resources are not common in the education literature, but there has been some notable work in this area of chemistry education on the study approaches used by students. Hakstian3 reported no effect of anticipated test type (either an objective test containing multiple-choice, true/false, and matching questions, an essay test, or a combined objective/essay test) on study method variables measured for 36 juniors and seniors in an educational foundations course. At least one of the approaches studied by Hakstian,3 use of notes, overlaps with the study resources in our study. Other research by Entwistle and Wilson4 showed that study methods chosen by students are related to their academic performance in college courses but that this relationship does not provide much predictive capability regarding student achievement. Recently, studies have explored the connection between study methods or strategies and achievement in chemistry.2,5,6 Most of these studies start with study approaches as opposed to our choice of study resources as an antecedent to an analysis of students’ study methods. However, there are overlaps between the findings of those investigations of study methods and this investigation of the choice of study resources. This overlap includes the resources that students describe that are interpreted by the researchers as evidence of specific study approaches5 and the differential study behavior for students of different achievement levels.2,5
■
METHODOLOGY To identify the study resources used by students, four separate surveys were conducted during the Fall 2013 semester, administered after each of the two instructor-written assessments and two departmental common exams. A timetable for survey administration and interviews is provided in Table 1. Students
■
Table 1. Timetable for Survey Administration and Interviews
RESEARCH QUESTIONS The research questions for this study include the following: 1. Does student selection of a main study resource change in relation to the type of assessment (departmental multiplechoice common exams vs instructor-written assessments) and is this choice stable over time? 2. Do students of different achievement levels (A/B, C, and D/F) choose a different main study resource for each type of assessment? 3. How do students of different achievement levels describe using their main study resources?
Survey
Administration
Survey 1
After the first instructor-written assessment in the 4th week of the semester After the week 6 common exam After the week 12 common exam After the week 12 common exam in the 13th week of the semester After the final instructor-written assessment in the 16th week of the semester
Survey 2 Survey 3 Interviews Survey 4
received an e-mail invitation to complete the online survey though Google Forms, and a link to the survey appeared in their online homework account. Interviews with a subset of students were used to explain and validate the researchers’ interpretation of the information obtained from student surveys and to investigate how students used the resources they chose. The invitation to be interviewed was made by one of the four researchers who were also chemistry department faculty. No researcher solicited or interviewed his or her own students. The invitation was made at the beginning of scheduled lab sections in week 13. After the semester grades had been submitted, additional data on students’ SAT scores and course grades were provided to the researchers for students who had both consented to participate in
■
COURSE STRUCTURE AND POPULATION The general chemistry course required of all freshmen at USNA is best described as a chemistry majors’ level general chemistry course, which includes 3 hours of lecture and 2 hours of lab per week for a 16-week semester. All students enrolled in general chemistry in Fall 2013 were invited to participate in this study. Participation was voluntary, and in accordance with USNA Institutional Review Board policy, no academic or monetary incentives were offered for participation. During this semester, 1015 students were enrolled in general chemistry in 58 sections of approximately 20 students with 28 different instructors. All of the instructors used 12
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
Journal of Chemical Education
Article
Analysis of the Survey
the research and release their grades. Course grades were used to categorize students into achievement groups in this analysis.
Study Resources Chosen for Instructor-Written versus Common Exams. Only 46 students answered all four surveys during the semester. A comparison of the responses of these 46 students to those of the larger group of students completing the surveys is included in the Supporting Information. In general, the responses of the students responding to all four surveys mirrored the responses of the larger group. On the basis of these results, the analysis of the larger group of responses was used to interpret general trends in response patterns across surveys even though each set of survey responses did not necessarily include the same students. Since the number of students responding to each survey changed over time, the percentage of respondents choosing each main study method is reported rather than the absolute number. As a result, this analysis focuses on the overall trend in responses. Figure 1 shows that for instructor-written assessments (Figure 1a,b), notes (either personal or instructor-provided) were the most commonly chosen main study resource. For these assessments, 41% of students chose notes for the first instructor-written assessment and 34% chose notes for the second instructor-written assessment. For the two common exams (Figure 1c,d), the reported main study resource differed from that reported for the instructor-written assessments, with 46% of the students reporting prior assessments as the main study resource for the week 6 common exam and 54% reporting using prior assessments for the week 12 common exam. On the basis of these results, we address the first research question regarding the relationship between study resource selection and assessment type. In general, trends in the data suggest that USNA chemistry students choose different study resources for different types of assessments. When studying for the instructor-written assessments, students primarily choose to study using notes, either their personal notes or notes supplied by their instructor. On a survey of the instructors asking them what main resource they suggested students use to study for their assessments, 44% reported recommending using notes to prepare for this exam. When studying for the departmental common exams, students choose to study the prior common exams provided by the chemistry department. This is in keeping with the advice supplied by instructors for preparing for the common exams. The majority study method selected for each type of assessment appeared to remain relatively constant over the semester, with the use of previous assessments increasing from the first to the second common exam. This may indicate that the specific choice of study resource is more influenced by the type of assessment than the individual preference of the student.
Surveys
The survey developed for this study asked students to report the study resources they used to prepare for each of four assessments over the course of the semester. The survey given after the week 6 common exam is included in the Supporting Information. Students were asked to indicate, in an open-ended question, the main study resource used to study for each assessment. The open-ended responses to the main study resource question were coded by three researchers into eight categories that grouped together similar study resources. The eight categories are listed in Table 2. Interviews
Fifteen-minute interviews were conducted by one of the researchers in a private room with a blank copy of the survey available to both the interviewer and the participant to guide the discussion during the interview. The purpose of the interview was twofold: to look for evidence of response process validity7 for the online survey responses and to address the research question regarding how students describe using the study resources they chose to prepare for the recently completed week 12 common exam. Fifty-seven students from 11 different instructors agreed to be interviewed. Of the 57 students participating in the interviews, only 28 had provided responses to the third survey. To address this, the interview included going through the survey with the interviewees and asking them to either provide and/or elaborate on their remembered answers to the survey questions. In this way, all 57 students who were interviewed answered the information on the survey whether or not they had completed it online beforehand.
■
RESULTS AND DATA ANALYSIS
Survey Sample
Of the 1015 students enrolled in the course, responses to the four online surveys ranged from a maximum of 550 student responses on the first survey (54% response rate) to a minimum of 216 on the fourth survey (21% response rate). For each survey, responses to the open-ended question that could not be linked to any known study resource as well as blank responses were not included in the analysis. Between 73% and 85% of the openended responses to each survey were able to be coded as one of the eight study resource categories in Table 2. Table 2. Study Resource Categories and Examples Study Resource Category Prior assessments Textbook and learning objectives Online homework Laboratories Notes Study guide Personal help Web
Examples Past multiple-choice common exams with answers posted on the departmental Web site Using the textbook either by reading, practicing unassigned problems from the textbook, or using chapter learning objectives provided by the department Access to completed online homework for review Copies of lab background information and procedures as well as graded lab reports Student- or instructor-prepared notes generated during class and/or instructor notes posted online after class Printed publisher study guide that accompanied the textbook Extra instruction from the course instructor, other instructors, an instructor assigned to the Resource Room, or other students either individually or in small groups, Academic Center classes, tutors or drop-in tutoring sessions, and Midshipman Group Study Program (MGSP) Web resources such as Google, Khan Academy, Tutor.com, YouTube, or other student-found external help 13
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
Journal of Chemical Education
Article
Figure 1. Main study resource chosen by percentage of responding students for each assessment. Results for the instructor-written assessments are in the top row, and results for the common exams are in the bottom row.
Study Resources Chosen by Students of Differing Achievement Levels. High-achieving, average, and lowachieving students were grouped according to their final course grade as either A/B, C, or D/F students to investigate the second research question regarding the relationship between study resource selection and student achievement. Approximately 30% of the total students in the course (305 students) consented to have their grades released. However, not all of these 305 students answered all of the online surveys. As a result, the number of students with coded survey responses who released their grades ranged from a high of 203 (66%) on the third survey to a low of 67 (37%) on the fourth survey. In line with the prior analysis, the percentage of students within each grade category selecting a particular study method is reported rather than the raw numbers. When the coded responses with grades released were analyzed according to achievement level, some of the eight main study resource categories reported in Table 2 were selected by very few students. Therefore, only the three most commonly reported study resources (prior assessments, notes, and personal help) are presented here. The complete data tables can be found in the Supporting Information. Figure 2 shows how the selection of main study resources differed for each grade category by survey.
The number of students receiving D and F course grades constituted about 10% of the 305 students who released their grades. As a result, the percentages of D/F students choosing each study resource should be interpreted cautiously since they are often the result of a relatively small number of students. In general, trends in the data indicate that the more successful A/B and the average C students are most likely to change their main study resource to match the type of assessment. Between 35% and 49% of these students selected notes as their main study resource for the instructor-written assessments (Figure 2a,b), and between 41% and 59% selected prior assessments as their main study method for the week 6 and week 12 common exams (Figure 2c,d). The D/F students made a less pronounced shift to match their main study resource to the type of assessment and were slower during the semester to make the shift than the A/B and C students regardless of the advice provided by their instructors. From Figure 2, it appears that D/F students are more likely to use resources that involve personal help from another individual as their main study resource. Because of this use of personal help throughout the semester, D/F students do not appear to choose different study resources for different types of assessments as often as the A/B or C students. For example, D/F students did 14
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
Journal of Chemical Education
Article
Figure 2. Percentages of students in each grade category selecting prior assessments, notes, or personal help as their main study resource. Results for the instructor-written assessments are in the top row, and results for the common exams are in the bottom row.
at USNA and thus were specifically recruited for the interviews to ensure that their choices were represented.
not choose prior assessments as their highest-frequency choice for their main study resource on the first common exam (Figure 2c) but as a group did increase the use of this resource for the second common exam (Figure 2d). D/F students may respond better to the advice of their instructors or try to mimic study choices of more successful students the second time a particular assessment type is given. To further investigate these trends, interviews of a subset of students were used to both support the results of the survey and to investigate not only what study resource students chose but how they used the study resource.
Analysis of Interviews
Each member of the research team read a subsample of the 57 transcribed interviews and participated in discussions with the rest of the team to determine commonalities among all of the interviews. The three most commonly used main study resources in the week 12 survey (prior assessments, notes, and personal help) were compared with those mentioned most frequently in the interviews and found to be similar. As a result, it was decided to more fully explore student use of these three main study resources. Prior assessments were the most frequently reported main study resource on the week 12 survey (54%) and the second most frequently mentioned during the week 13 interviews (84%). Notes were the second most frequently chosen main study resource on the survey (17%) and the most frequently mentioned in the interview (86%). Personal help was the third most frequently mentioned study resource in both the survey (12%) and the interviews (81%). The apparent discrepancies in frequency between the survey and the interviews are due to the fact that in the interview students could discuss as many study resources as they wanted, whereas on the survey students were asked to designate their main study resource. A comparison of study resource selection in the interviews and in the survey, broken down by course grade, is provided in Figure 3. In general,
Interview Sample
The 57 students who volunteered to be interviewed were compared to the rest of the students enrolled in the course to see whether the interview sample was representative of the class as a whole. The comparison was based on SAT scores (verbal and math) and common exam grades (week 6 and week 12) and carried out with two one-sided t tests (TOST).8,9 The students interviewed were found to have higher scores on the two common exams (week 6 and week 12 exams), lower SAT verbal scores, and SAT math scores equivalent to the rest of the students in the course. The full results and R code used for the TOST analysis can be found in the Supporting Information. The course grade distribution for students participating in the interviews (A = 23%; B = 30%; C = 42%; D = 5%) contained more C students than the overall course grade distribution because these students did not typically volunteer in past studies 15
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
Journal of Chemical Education
Article
Figure 3. Study resource selection for the second common exam as reported on the week 12 survey and during interviews.
• D Student: “Just go through (the prior assessment) and see if there are any huge concepts that I didn’t cover.” Notes-Use Descriptions. During the interviews, students earning B and C final course grades reported using notes to study for the second common exam more often (94% and 92%, respectively) than the A (69%) and D (67%) students, as can be seen in Figure 3b. Notes were used primarily as a study or review tool by all grade groups, but there were differences in how they were used, with the higher-achieving students using them as a quick review, the C students relying heavily on being able to duplicate the way problems were done, and the lower-achieving students reading them over. Some typical quotes that expand on how notes were used are presented below: • A Student: “The instructor’s notesI just reviewed and made sure I understood all the information on there.” • A Student: “I skim my personal notes. We take notes in class and I ... quick reference if I am not too sure how to do something or I know we did an example in class. I don’t rely on them too heavily but I do utilize them.” • B Student: “I generally just go over them and if there’s a way to do a problem I’ll look at how I did the problem. Occasionally I’ll try to recreate it with a new one but usually I just see how I did it and make sure I understand it.” • C Student: “Typically [the instructor] will go through them (worksheet version of notes) in class and fill them out as we go and usually I’m just trying to keep up. And then before class or before a quiz or test I’ll spend hours going over that and that’s typically my study methods.” • D Student: “I would check the learning objectives and then go back to see where I took notes about them and review my notes.” Personal Help-Use Descriptions. The interview results in Figure 3c reveal that students at all achievement levels reported using some source of personal help to study for the second common exam, with a higher use reported by the lower-achieving students. Follow-up questioning during the interviews was used to probe how students used personal help as a study resource. On the basis of the frequency that the use of personal help was mentioned in the interview transcripts, 73% of the time personal help was used to “get the answer” and 27% of the time it was used to “see the whole picture or understand the concepts”. In a timeconstrained environment, students may be choosing resources
the selection of study resources in the interviews followed the same trend as the selection of a main study resource in the survey. Prior Assessments-Use Descriptions. The interview results illustrated in Figure 3a indicate that A and B students reported similar use of prior assessments (92% and 94%, respectively) as study resources for the second common exam (week 12 exam). The C students (75%) used prior assessments less often than the A/B students and in a percentage closer to that of the D students (67%). No F students reported using prior assessments in the third survey, and there were no F students in the interview group. In using prior assessments, the interviews revealed that A students relied on themselves to answer a question and, if appropriate, understand why they got a question wrong. The B and C students relied on themselves initially and then sought personal help if needed to determine why they missed a question. Only one of the D students who reported using prior assessments to study specified that the prior assessments were used as an overview of important topics for the upcoming test. The following quotes from the interviews are provided as examples of student use of prior assessments by achievement level. • A Student: “That’s (prior assessment) usually the very first thing that I do because then you know what you really need to study and what you don’t really need to study because you don’t want to waste your time with something that you know really well whereas you might want to spend time with something that you like find a trend, like you missed all 3 kind of one kind of problem. So I’ve done that. I used that to direct where my studying will go.” • B Student: “I looked (at the prior assessment) to make sure I had an understanding of how to do the questions and if I didn’t have an understanding I’d go to the resources and try to figure it out.” • C Student: “I printed it (prior assessment) out and I would actually sit down and take the entire exam at once. Anything I had trouble on or was really confused on, I circled. After, I went to the answer key after I finished, (and) checked all the ones I got right. Anything I got wrong, I went back and looked and said ok what did I do wrong and if I still did not understand, I went and got help from a classmate who was either in [the next chemistry course] or in the same chemistry level I was, just to confirm how I either messed up a calculation or just was not getting the concept.″ 16
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
Journal of Chemical Education
Article
Our results, based on the study resource survey administered four times during the semester targeting both instructor-written and common exams, demonstrate a trend that higher-achieving (A/B) and average (C) students select resources that match the type of assessment at hand. That is, students study from prior common exams in preparation for common exams and from instructor or personal notes for instructor-written assessments. This trend appears to be stable over the length of the semester. Lower-achieving students (D/F) do not consistently match their choice of study resource to the type of assessment at hand and are slow to change to a more appropriate match. Average and lower-achieving students also tend to choose study resources that provide face-to-face help, including the course instructor or other knowledgeable sources. The use of such resources is not as common among higher-achieving students. The data suggest that higher-achieving students are more likely to use study resources that can be used independently of others, such as notes and prior assessments, but will ask for help from upper classmen or peers when needed. This apparent difference in selection of study resources according to the achievement level of students could be used by instructors to help students who are struggling in the course. Instructors in face-to-face help sessions with students could provide mentoring or teaching techniques to guide students to become more confident and independent learners by stressing metacognition and understanding rather than only providing solutions to problems. The results of this research suggest that instructors or tutors in face-to-face help sessions should emphasize what the student knows and then work with the student to reach the next step in the logical argument of a solution as a way to build both the student’s understanding and confidence as a way of providing the requested help to “get an answer”. This approach may take longer and initially result in some resistance from the student, but if successful, this approach could help average and lower-ability students learn to use independent resources together with faceto-face resources to determine both meaningful questions about the content and self-generated answers to those questions. Average (C students) can be viewed as being in transition between choosing successful and less successful and/or appropriate study resources. Average students could be helped by the instructor to grow in metacognition (self-awareness of what they know and what kind of help they need to be successful) as they study. It should be noted that this research does not attempt to draw a causal connection between the selection of personal help and low achievement or between the selection of independent study resources and high achievement. It may in fact be that the selection of personal help versus independent resources is a symptom rather than a cause of low versus high achievement. It is also reasonable that students who are struggling are likely to seek help, while those who are succeeding may not feel the need to work with others. Furthermore, the element of time constraint may result in students seeking an efficient and certain answer rather than risking failure or investing in more time-intensive resources by pursuing answers on their own that may or may not be correct. In line with prior research,10−14 we find that students who have high mathematical ability as measured by the math SAT are more likely to succeed in general chemistry. This previously developed math ability explains more variance in achievement scores than in the selection of study resources. Part of the reason that study resource selection was not found to be highly related to assessment score may be that simply knowing what resource a student
that are efficient and convenient rather than those that might foster deeper understanding. Interview Summary. From the data, a pattern emerges suggesting that of the students interviewed, the higher-achieving students begin studying by using resources they can use independently and then seek help from other classmates when they need it. B students tend to choose resources similar in frequency to A students but rely more heavily on the personal help options, including extra instruction with the instructor, who is a bona fide expert. C students trend with the higher-achieving students in the use of prior assessments and with the lower achieving students in the use of personal help and notes, demonstrating that they do not have a consistent pattern of study resource selection. The small number of D students interviewed in this study discussed relying primarily on personal help from three sources, namely, the Academic Center/Tutors, their instructor, and their peers. Although there were too few D students in the interview sample to conduct a definitive analysis, it is interesting to note that lower-achieving students may choose help from someone who might be counted on to demonstrate how to do a problem with little input from the student. Relationship between Study Resource Selection and Assessment Scores
Results from the surveys and interviews indicate that students earning different course grades select different study resources for each type of assessment. An analysis of covariance (ANCOVA) was conducted for both common exams and the last instructor assessment to examine the relationship between study resource selection and assessment scores. Math SAT scores were included as a covariate on the basis of literature demonstrating a relationship between math ability (as measured by math SAT scores) and performance in chemistry.10−14 The effect of math SAT was found to be highly significant for all three assessments (p < 0.001) with an effect size (η2) between 0.23 and 0.28. The effect of study resource selection was only significant for the week 6 common exam (F(5,97) = 2.87, p < 0.05; η2 = 0.10), but the Bonferroni pairwise comparisons did not reach statistical significance (p = 0.05) for any group. More information on this analysis can be found in the Supporting Information. The effect of study resource selection on course grade became less important throughout the semester moving to the week 12 common exam (F(6,195) = 2.06, p > 0.05; η2 = 0.04) and the week 16 instructor-written assessment (F(5,60) = 0.36, p > 0.05; η2 = 0.02). Math SAT scores having a larger effect on achievement than the selection of study resource is not unexpected. Many general chemistry course assessments include a substantial number of items that can be solved successfully using math ability with little or no demonstrated knowledge of the underlying chemistry concept. Therefore, math ability often plays a substantial role in overall general chemistry achievement.
■
CONCLUSIONS In this study, we chose an initial research situation of a large student population and relied on volunteers for each data collection point. Relying on volunteers over an extended period of time can interfere with the ability to track a large number of students through all of the time points across the semester. In the analysis, wherever possible, we used quantitative analyses to show both the trends and limitations of our data. We incorporated a smaller qualitative interview component to validate and expand our understanding of how students use study resources. 17
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
Journal of Chemical Education
Article
(4) Entwistle, N. J.; Wilson, J. Degrees of Excellence: The Academic Achievement Game; Hodder and Stoughton: London, 1977. (5) Ye, L.; Shuniak, C.; Oueini, R.; Robert, J.; Lewis, S. Can They Succeed? Exploring At-Risk Students’ Study Habits in College General Chemistry. Chem. Educ. Res. Pract. 2016, 17 (4), 878−892. (6) Chan, J. Y. K.; Bauer, C. F. Learning and Studying Strategies Used by General Chemistry Students with Different Affective Characteristics. Chem. Educ. Res. Pract. 2016, 17 (4), 675−684. (7) American Educational Research Association; American Psychological Association; National Council on Measurement in Education. Standards for Educational and Psychological Testing; American Educational Research Association: Washington, DC, 2014. (8) Limentani, G. B.; Ringo, M. C.; Ye, F.; Bergquist, M. L.; McSorley, E. O. Beyond the t-Test: Statistical Equivalence Testing. Anal. Chem. 2005, 77 (11), 221 A−226A. (9) Lewis, S. E.; Lewis, J. E. The Same or Not the Same: Equivalence as an Issue in Educational Research. J. Chem. Educ. 2005, 82 (9), 1408− 1412. (10) Lewis, S. E.; Lewis, J. E. Departing from Lectures: An Evaluation of a Peer-Led Guided Inquiry Alternative. J. Chem. Educ. 2005, 82 (1), 135−139. (11) Mitchell, Y. D.; Ippolito, J.; Lewis, S. E. Evaluating Peer-Led Team Learning across the Two Semester General Chemistry Sequence. Chem. Educ. Res. Pract. 2012, 13 (3), 378−383. (12) Tien, L. T.; Roth, V.; Kampmeier, J. A. Implementation of a PeerLed Team Learning Instructional Approach in an Undergraduate Organic Chemistry Course. J. Res. Sci. Teach. 2002, 39 (7), 606−632. (13) Xu, X.; Lewis, J. E. Refinement of a Chemistry Attitude Measure for College Students. J. Chem. Educ. 2011, 88 (5), 561−568. (14) Nordstrom, B. Predicting Performance in Freshman Chemistry. Presented at the 199th National Meeting of the American Chemical Society, Boston, MA, April 22−27, 1990. (15) Marton, F.; Säljö, R. Approaches to Learning. In The Experience of Learning; Marton, F., Hounsell, D., Entwistle, N. J., Eds.; Scottish Academic Press: Edinburgh, 1984; pp 36−55. (16) Entwistle, N. J.; McCune, V. The Conceptual Bases of Study Strategy Inventories. Educ. Psychol. Rev. 2004, 16 (4), 325−345.
selects does not provide sufficient insight into how the student uses that resource. The results of this research provide further support for the idea that what is important is the way students are using the resources, not just the specific selection of resources. However, study resource selection can serve as an approximation of student approach. Instructors providing help for students in private or small-group tutoring sessions might do better to both emphasize choosing appropriate and efficient study resources for the task at hand and supporting the learning of concepts and metacognitive approaches together with providing the immediate help of how to solve chemistry problems. We see overlap between this investigation on study resources chosen and the published research on study approaches used by students. In our case, the strategic matching of the study resource to the assessment type is done more often by higher-ability than lower-ability students. The quotes by A students also demonstrate the use of study resources to better understand the processes or concepts and to go beyond getting the correct answer. This behavior is described in the literature on study approaches as a Deep strategy.15,16 The presence of overlap between our research on selection of study resources and the published research on study approaches suggests that further research is needed to investigate how specific study resources can be used effectively by students.
■
ASSOCIATED CONTENT
* Supporting Information S
The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.6b00285. A typical midshipman’s daily schedule, the survey instrument, additional information regarding the 46 students responding to all four surveys, full data tables for the study resource analysis by grade category, R code used to conduct the two one-sided t tests (TOST) with the TOST results, and detailed ANCOVA results (PDF, DOCX)
■
AUTHOR INFORMATION
Corresponding Author
*E-mail:
[email protected]. ORCID
Regis Komperda: 0000-0003-4837-7141 Notes
The authors declare no competing financial interest.
■
ACKNOWLEDGMENTS The authors thank the students and faculty involved in the general chemistry course at the U.S. Naval Academy (Annapolis, Maryland) and the James Kinnear USNA Faculty Research Support Fund for support in completing this research.
■
REFERENCES
(1) Chang, R.; Goldsby, K. Chemistry, 11th ed.; McGraw-Hill: New York, 2013. (2) Sinapuelas, M. L. S.; Stacy, A. M. The Relationship Between Student Success in Introductory University Chemistry and Approaches to Learning Outside of The Classroom. J. Res. Sci. Teach. 2015, 52 (6), 790−815. (3) Hakstian, A. R. The Effects of Type of Examination Anticipated on Test Preparation and Performance. J. Educ. Res. 1971, 64 (7), 319−324. 18
DOI: 10.1021/acs.jchemed.6b00285 J. Chem. Educ. 2017, 94, 11−18
