Senior Undergraduate Research and Assessment ... - ACS Publications

Downloaded via CORNELL UNIV on July 9, 2018 at 12:18:37 (UTC). See https://pubs.acs.org/sharingguidelines for options on how to legitimately share published articles.

Chapter 19

Senior Undergraduate Research and Assessment at Florida Southern College Deborah Bromfield Lee* and An-Phong Le Department of Chemistry, Biochemistry and Physics, Florida Southern College 111 Lake Hollingsworth Drive, Lakeland, Florida 33801, United States *E-mail: [email protected].

Undergraduate research at Florida Southern College’s (FSC) Department of Chemistry, Biochemistry and Physics is required of all seniors. To support this best practice, the department wants to provide a pathway for professional growth for each student. A two-semester sequence of courses was designed to encourage students to improve communication skills as well as contribute to the scientific community through novel research projects, providing authentic learning experiences. The first semester allows students opportunities to develop a research proposal that aligns with one or more faculty projects and work towards developing skills or methodologies that will be fully implemented in the second semester. Students give regular presentations and have assignments leading to the proposal. The second semester culminates in a final research paper and presentation open to the entire FSC community. The course is led by one primary faculty member, but all faculty members generally participate and are integral to the course. All proposals and final senior papers as well as student presentations are assessed by the entire department. This chapter will highlight course structure, assessments with rubrics, and faculty involvement.

© 2018 American Chemical Society Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Introduction Founded in 1883, Florida Southern College (FSC) is the state’s oldest private college. FSC has 2,879 undergraduate and graduate students from 46 states and 52 countries. The mission of the college is to prepare students through dynamic engaged learning to make a positive and consequential impact on society. The Department of Chemistry, Biochemistry and Physics has six chemistry faculty, with at least one member in each of the major disciplines. Students can conduct research in the following sub-disciplines: Analytical, Biochemistry (Molecular Biology), Inorganic (Metal Organic Frameworks), Organic, Physical Chemistry and Chemical Education. In addition, the faculty often do collaborative research with other chemistry faculty as well as other disciplines, such as Biology or Psychology. The department typically houses about 80 Chemistry and Biochemistry and Molecular Biology (BMB) majors. Students have access to a variety of instruments for research and courses that includes GC-MS, 300MHz NMR, HPLC, powder x-ray diffractometer, polarimeter, ultracentrifuge, FTIR, UV-Vis, TGA, and microwave reactor. The department requires a two-semester research course for all chemistry majors. BMB students are required to take two semesters of research, but may take the sequence in either the biology or chemistry departments. We also offer independent research projects for students prior to their senior year. A unique aspect of our research program is that all faculty members are engaged with students and participate in the senior year courses. The focus of this chapter will focus on the assessments of the undergraduate research (UR) experience at FSC, though many of our courses provide foundation for the skills students need to be successful in research. Some of the qualities of UR are addressed in our model, and a survey of current and former students as well as faculty are used to evaluate the model from their perspective. Relevant Literature and Course Overview As chemistry is a hands-on discipline, it is customary for students to have labs for most of the courses in the major. Many innovations in teaching labs have been developed, and efforts on improving students’ experiences have been the focus of many chemical education researchers (1). Many chemistry programs have incorporated inquiry-based approaches (2), writing in the curriculum (3, 4), research-based laboratories (5), and undergraduate research (6–8). Undergraduate research (UR) has become more widespread as it is an important high impact practice that can prepare students with a variety of skills (9). Osborn & Karukstis suggest there are three categories of benefits of UR for students: Cognitive and intellectual growth, professional growth and advancement, and personal growth and development (10, 11). It has been shown that students participating in research-based laboratories perform better than students in traditional labs. Students tend to develop a deeper understanding of concepts and the nature of science (12). UR can be highly student-centered, where students can be mentored one-on-one in an apprenticeship model. It can therefore be expected that unique UR opportunities built into the curriculum can 312 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

promote an understanding on how science is conducted, the ability to develop research questions and more critical data analysis. There are many studies and articles on specific types of implementation on UR. Outside of being a recruiting tool for institutions, it is important that UR benefit the students and faculty involved (13). Some of these studies include assessment of learning gains and faculty perspectives on UR (14, 15). Graduate school attendance and persistence in college have been used as an assessment of UR. The problem with these forms of assessments is that they do not capture an assessment of the program (16). Students and faculty mentors are asked about their experiences in some studies. Students who participated in UR experiences and faculty mentors were asked about their authentic learning experiences in a study by Kardash (17). The study found positive impacts on the development of UR skills by both the faculty and students. Kardash assessed skills such as use of primary literature, writing, evaluation of data, and reformulation of the original hypothesis. Alumni have reported that UR experiences have been highly beneficial, particularly if conducted for more than one semester and if a thesis was completed in the final year (13). The alumni reported significant enhancements in cognitive and personal skills as compared to alumni who did not partake in research. Alumni can provide a perspective remote from the experience, but has been found to correlate to that described by faculty. It is helpful not just to assess the long-term effects and faculty or student opinions, but to have tools that evaluate the implementation of research activities to provide improvements in coursework as well as the mentorship of the research faculty. Metacognitive skills in students can be promoted in a research course, particularly if students are provided a framework to promote metacognition. Tanner provides sample questions faculty can ask students to promote planning, monitoring and evaluating of learning in courses (18). Assignments in which students must plan their work, monitor their progress and evaluate what they have done or learned can easily be implemented in UR. Assignments and assessments that teach students to think about how they are thinking about chemistry and/or research are useful to achieve self-regulation skills. UR can promote students’ abilities to judge their performance, learn how to set realistic goals, and reflect on what they have learned. At FSC, we promote students’ professional development by actively engaging them in an individualized project. The senior research sequence consists of two 2-credit hour courses spanning the final year. Prior to the first course, students are asked to meet with all faculty members and mutually decide on a research topic with at least one faculty mentor. The first course focuses on the development of a research project in which the students develop a research question or a niche under a faculty mentor’s existing research agenda, review literature, identify methods, gather preliminary data, present literature and proposal updates, and write a final proposal. In the second semester, students work towards completion of the project, give a final presentation open to the college and chemistry community members, and write a final research paper. It is our program’s goal to also have students present outside of the college at local, regional or national conferences. Table 1 shows the organization of the courses with the activities of the courses. 313 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Table 1. Organization and Major Assignments of Research Sequence Introduction to Research

Senior Research

Week 0 – Decide project direction Week 1 – Expectations/ethics/safety Week 2 – Summary of Project Weeks 3-4 – Literature Presentation and Literature summaries Weeks 5-6 Proposal Summary Weeks 7-13 – Updates Week 14 -Peer review of Proposal draft Week 15 - Final Proposal Paper Final Proposal Presentation (open to department and students). Research Advisor Evaluation of progress towards research results and skills

Week 1 – overview and discussion Week 2 –Proposal Revised Weeks 3-4 – Methodology Updates Weeks 5-6 – Literature Presentation and Literature discussion Weeks 7-13 – Updates and discussions Week 14 - Peer Review of Final Proposal paper Week 15 - Final Proposal Presentation (campus symposia) Research Advisor Evaluation of research results and skills

Students give presentations, discuss their research progress, and receive feedback from both their peers and from the faculty during the in-class portion of the course for one hour per week. Presentations are rotated between students each week, and each student typically present every other week. Students are expected to spend at least 6 hours in the lab (or generating data for computational projects) per week. In addition, students are expected to analyze their data and meet with their mentor on a weekly basis (or more frequently). About 60% of students enrolled in senior research in the last 4 years engaged in research collaborations with faculty prior to senior research, and some of these students continue those same projects. These students’ prior work typically consisted of beginning experimental work, gathering data for their project, and possibly disseminating their results in a poster presentation. They generally have not completed an in-depth literature review or written extensively about their work, and many students with prior research experience have not reached a significant level of independence in their projects. The UR projects are most often directly related to a faculty’s research agenda, but sometimes projects designed outside the faculty’s expertise are conducted. It is important that students learn to develop a research question in which they are genuinely interested, but it may be necessary at times to limit how far afield students can explore to adequately mentor them. We require an Applied Synthesis and Characterization lab course for chemistry majors in which students can begin developing a proposal that can be carried out in their senior year. Though this is the best place for students to start working on literature and considering a research agenda, not all students will take the course prior to their senior year, limiting its usefulness in the development of a senior research topic. In addition to the research courses, our foundation and in-depth courses provide some of the foundational skills needed for the senior research sequence. In organic chemistry and analytical chemistry, student have scaffolded lab reports in which there is focus on developing the writing of the report in parts (Learning Lab Reports) (19, 20). Students in organic chemistry have activities in which they develop skills to analyze spectroscopic data, evaluate reaction 314 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

conditions, and monitor reactions. Students in analytical chemistry also have exercises in which they are taught to parse and interpret primary literature articles (21). Inorganic chemistry and biochemistry require students to not just conduct literature searches as they do in most of our courses, but to also evaluate literature and give presentations. All our laboratory courses teach students the basic techniques they will encounter in many research projects. In our Applied Synthesis and Characterization course, students have multi-week labs that incorporate various techniques and instrumentation at a more advanced level than their other courses. Students taking Instrumental Analysis work collaboratively on an extended method development project, where they must search the literature for articles related to an assigned analysis, adapt and test the procedures for implementation in the laboratory, and perform the analyses developed by their classmates to provide feedback. In independent study courses focused on research, students conduct authentic research experiences tied to a faculty member’s research and learn specific techniques or methods related to that research agenda. Our department is always looking at ways to improve the experience for students and hence by no means claims our program is flawless. We do, however, feel that the format (emphasis on skill development and pursuit of a research agenda), assignments, and assessments of the course are a critical part of the successes of our program.

Faculty Involvement One unique aspect of our research sequence is the extensive participation of all the Chemistry and Biochemistry faculty. All faculty participate in the research sequence, even if they are not supervising any current research students. The faculty of record (and the accompanying administrative duties) is rotated, but all permanent faculty participate in the assessment of student presentations and written work. Most importantly, all the faculty make a point to attend and to participate in the weekly and final presentations. This gives a ‘group meeting’ feel to the course, where each faculty member provides a different perspective to the discussions and to issues encountered by students. Broad discipline representation in the audience promote student discussions with faculty who have diverse expertise and insights relevant to students’ projects or problems. While our faculty may not need to have group meetings to save time meeting with many students, it allows all the faculty to experience and monitor the students’ progress at one time. On the first day of Introduction to Research, faculty and students both discuss their expectations, concerns, and what they hope to get out of the course. The faculty describe how their involvement and their goal is to provide students with multiple perspectives and different skills based on their own research discipline. We hope that this helps increase student accountability by motivating students to keep better track of their work and thought processes, encouraging them to make progress in their projects, and fostering a sense of community. Students also develop better communication and presentation skills as they must convey their understanding to both experts and non-experts. 315 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Due to the intimidation factor of having several faculty members watching the presentations, it becomes important for the faculty to provide some positive feedback either during the presentation, in their written comments, or through the research advisor. It is easy to critique students to help them improve, but we have found that such criticism is discouraging and demotivating to students who may be struggling with their projects. At this stage in their academic career, discussions on places for improvement are critical, but acknowledging to students where they excelled, have improved, or have even put in effort are just as important. Finkelstein and Fishbach suggest that novices seek positive feedback as it will promote optimism towards goal attainment (22). Assessments A variety of assessment tools have been developed to assess students research progress, communication skills (oral and written) and progress towards goals (Table 1). These assessment tools have been revised as needed each year, as well as continued iteration on assignments to improve the courses. The assignments align with fundamental skills associated with research as well as skills that may be useful to students post the baccalaureate level such as communication skills, formulation of a questions, problem or research direction, working independently and ethical and safe behavior. Assignments are aligned with the student learning outcomes (SLO) of the courses provided below. The rubrics for written and oral presentations were designed using the American Association of Colleges and Universities VALUE rubrics (23). The rubrics provide students with guidelines of our expectations and mechanisms for regular feedback. SLO 1. Review scientific literature and critically assess the information presented. SLO 2. Develop a research proposal based upon a thorough search of the scientific literature. SLO 3. Collaborate with a faculty advisor in the development and implementation of the proposal. SLO 4. Plan and perform experimentation that may utilize state-of-the-art instrumentation (NMR, IR, GC/MS), software and/or scientific techniques. SLO 5. Analyze and interpret data obtained from the experimentation. SLO 6. Demonstrate skills in problem solving, time management and oral and written communication. SLO 7. Utilize technology in reviewing the scientific literature, obtaining current information, and accessing databases. SLO 8. Communicate the results of the research project in a poster or oral presentation (of at least 7 minutes) at a scientific or scholarly meeting. Though the best assessment would have a controlled group, it is not desirable to limit an educational benefit from a group of students in our program. The focus therefore is the assessment tools and perceptions of faculty and students. In addition to assessment of assignments, evaluation of the courses by the faculty and students were solicited. Like Kardash (17), we have surveyed faculty, current students, and former students about UR skills and faculty involvement. Hinkhouse et al. (6) have interviewed students on their experiences in UR in chemistry. They 316 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

asked questions regarding the nature of science, their experience before and after the project, and their impression on portions of the program. In our program, we do not directly interview students, but similar questions are asked of students directly and indirectly through discussion with faculty advisors. Our survey of students also ascertained what they felt about the program, but also promotes interventions via the faculty feedback and researcher mentor meetings.

Coursework Breakdown The following is how the course is broken down and the common assignments for each criteria of the course. 30% Research Proposal: It will encompass the development of research question, a thorough scientific literature search, and the assignment’s relevant research methods. Reports will follow scientific proposal formatting. This grade may also include peer assessments (other than the final) and other intermediate assignments relevant to your development as a researcher and chemistry senior. Includes: resume and cover letters, literature summaries, summary of proposed research, and initial proposal document 20% Development of Laboratory and Instrumentation Skills: The student will begin to apply their scientific knowledge and laboratory skills to the execution of their project and develop new laboratory techniques and gain experience with state-of-the-art instrumentation. Assessment will be based upon the student’s level of proficiency in the various activities. Your research advisor will evaluate your performance each week (progress reports), your lab notebook, and other relevant areas. Includes: notebook (or relevant data reporting method deemed by research advisor), cleanliness, lab skills, research advisor’s analysis, and instrument video 10% Research Results: All preliminary research and laboratory work will be recorded, and the results of the findings and laboratory work will be summarized. Includes: research advisor’s analysis 20% Presentations: The progress and results of this project will also be presented as oral presentations that include peer evaluation. These presentations will be rotated and will be a detailed description of your literature, current findings, and/or results. Includes: initial proposal presentation, literature presentations (oral and uploaded), and results presentations 20% Final Report: Upon completion of the semester’s work, a written research report will be prepared, peer-assessed, and orally-presented. Includes: Final Proposal, peer assessments, and Final Presentation To help senior research students plan research, evaluate goals, and assess what they have done or learned, they submit a weekly progress report which is developed collaboratively with their research mentor. The faculty mentors provide guidance as well as a grade on their progress in their project. The progress report template is included below (Table 2).

317 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Table 2. Weekly Research Progress Report Assessment Weekly Progress Report Assessment Person Reviewed: ___________________________________ Complete the following sections and review your responses with your faculty mentor. This form is due at the beginning of class each Friday. Copy down the goals you set out in your last meeting (if applicable). 1) Last week’s goals: 2) What have you done this week? Describe your research activities this week in moderate detail: 3) What will you work to accomplish in the next week? 4) Next week’s goals: Faculty mentor weekly grade assessment: Grade_____________/100____

Mentor Signature _______________

SLO 3. Collaborate with a faculty advisor in the development and implementation of the proposal. SLO 6. Demonstrate skills in problem solving, time management and oral and written communication.

The reflective progress report factors into the students’ overall course grade, but it also provides opportunities for dialogue (and early intervention, if necessary) between the mentor and the student. This assessment helps to ensure students are in regular contact with their mentor, and it help to develop students’ abilities in reasonable goal setting, honest reflection on their work, and accountability based on their previously selected goals. Students are required in both the introduction to research course and senior research course to present literature articles, research updates, and their research proposal. In addition, students prepare a final research presentation open to the College campus and invited guests (such as local members of industry). Below are sample rubrics from some of these assignments. One change between the first semester and second semester of the course is that these presentations are expected to be of a higher quality in the second semester. The diction in the rubrics may be more strongly worded or the threshold for excellence is higher. The following is a copy of the research presentation update rubric, where the student would update the class on what they have done in their project and on their next steps (Table 3). In this rubric, students evaluated by the quality of their performance (10 – exceeding expectations to 1- did not meet expectation). Each criteria is worth a percentage of the assignment. To calculate the students grade, the quality score is multiplied by the fraction of their grade associated with the criteria. If the students are rated as a 9 (exceeding expectation), they earn 45pts in the content criteria. This allows students to see the weight of each criteria and hence make a correlation to emphasis on their presentation. In most of the student presentations, the content is the most significant criteria. This form of assessment is the same for the remainder of rubrics in this chapter. 318 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Table 3. Rubric for the Presentation of Research Update Presentation of Research Update Assessment (100 points) Person Reviewed: ___________________________________ Instructions: Prepare a presentation (using slides) to give: 1. Introduce project 1 minute 2. Review work since proposal presentation (lab-based AND literature evaluation) 3. Data, Problems, Review literature to evaluate methods, evaluation/analysis of data 4. Closing remarks – wrap it up 1 minute or less Time Limit: 15-20 minutes. You will have 5-10 minutes for questions. Rubric (for more information Refer to the AAC&U Values rubric on Oral Communication) (24). SLO 5. Analyze and interpret data obtained from the experimentation. Assessment

Exceeding Expectations 9 - 10

Average 6-5

Above Average 8-7

Below Average 4-3

Did not meet expectation 1-2

Content (50 pt.) Reasonable amount of data and data analysis. A reasonable amount of work/effort towards the project seems to be made. Clear understanding of underlying principles and subject matter within the data. Central message is compelling (precisely stated appropriately repeated, memorable, and strongly supported). Includes supporting material that is relevant and credible. Organization/Structure (10 pt.) Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is clearly and consistently observable and is skillful and makes the content of the presentation cohesive. Delivery and Language (20 pt.) Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation compelling, and speaker appears polished and confident. Language choices are imaginative, memorable, and compelling, and/or enhance the effectiveness of the presentation. Language in presentation is appropriate to audience. Visuals (10 pt.) Slides are easy to read; Figures and tables are well labeled; Use of drawing or chemical structure software. Answering questions (10 pt.) Confidence is demonstrated. Knowledge in answering questions.


Table 4. Rubric To Assess Presentations on Relevant Literature Articles Presentation of Literature (100 points) Person Reviewed: ___________________________________ Instructions: Prepare a presentation (using slides) to provide a survey of the paper you have posted. The paper should be specific to your proposed work. The general format should include what was the author(s) exploring, what were their findings and the relevance to your work. You should pick a paper that ids either something that you are using in your research, as a guide for your work or work you will try to reproduce (consult your research advisor). By the start of class each presentation should already be up on the computer. There should be no delays in opening files during class. Submit paper online - at least one week in advance of presentation for full points and -1pt for every day after. Time Limit: 10 minutes. It is recommended you plan for 8-9 minutes so you do not go over. You will have 5 minutes for questions. SLO 1. Review scientific literature and critically assess the information presented. Assessment

Above Exceeding Expectations Average 8-7 9 - 10

Average 6-5

Below Average 4-3


Content (50 pt.) Clear understanding of underlying principles and subject matter within the literature article. Application of the paper to your research should be embedded. Organization/Structure (10 pt.) Logical and systematic presentation; Appropriate length of seminar. Delivery (20 pt.) Clarity of speech; enthusiasm; engagement with audience; appropriate volume and rate of speech; appropriate gestures and body movement. Visuals (10 pt.) Slides are easy to read; Figures and tables are well labeled; Use of drawing or chemical structure software. Answering Questions (10 pt.) Confidence is demonstrated knowledge in answering questions.

In the introduction to research course, students give presentations on literature relevant to their proposed project. The audience, both faculty and students, is requested to read the presented papers prior to the course. The presentations require student to thoroughly evaluate and present the rationale cited, data obtained, and conclusions drawn by the authors. This often requires students to find and digest resources outside of the article being presented to better understand the article’s content and its place in the broader research field. The rubric for this assignment follows (Table 4). 320 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Table 5. Assessment To Evaluate Student Leading a Discussion on a Chosen Literature Article Literature Discussion Assessment (100 points) Person Reviewed: ___________________________________ Instructions: 1. Select a paper that you can connect to your research (primary literature) - Introduce project 1-2 minutes - The paper must be of reasonable length – often 4-10 pages - It is suggested you select 2-3 papers to review and select one to use for the session - It may be helpful to review literature surrounding this paper 2. Critically evaluate the paper including - State the uniqueness of the paper - If there are flaws in the design you should comment on these 3. Lead a discussion by maintaining an engaged environment. - To lead a stimulating conversation, it is important to thoroughly analyze the paper. Annotate the paper or make thorough notes on the paper with points to create discussions. Time Limit: 25-30 minutes. SLO 1. Review scientific literature and critically assess the information presented. Assessment

Exceeding Expectations 9 - 10

Average Above Average 6-5 8-7

Below Average 4-3


Choice of paper (10 pt.) Leader must display the significant of the paper to their research or the general project is a part of a larger project. This must be clearly communicated and supported. Critical Analysis (30 pt.) Supports discussions with material from the paper. Thorough analysis of the work presented (positive or negative). Engagement (40pt) Asks questions when needed to clarify understanding or responds to discussion points. Eye contact with speakers. Encourage conversation by asking questions or pointing out areas for discussion. Must maintain the conversation for 25-30 minutes by facilitating the discussion when necessary with critical points from the paper. Participation (15pt) Student actively participates in the literature discussion led by another student. It is expected that when you are not leading the discussions, that you are also engaged in the material. Disseminated Paper in a Timely Manner (5pts)

Submit via the Portal. At least one week in advance of discussion for full points and -1pt for every day after.

To promote an increase in cognitive and metacognitive skills, a literature discussion was added to the senior research course in addition to the literature presentations. In these discussions, students must consider how to lead the audience in a discussion of the literature, rather than simply providing their 321 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

own evaluation. The assignment requires students to broaden their knowledge beyond the article’s content to prepare questions that lead the audience through an in-depth discussion of the paper. The rubric for this assignment follows (Table 5). In addition to presentations, students have writing assignments. Students prepare an annotated bibliography and research proposal drafts on their way to completing the final research paper. Each faculty evaluate 2-3 student papers providing a range of comments (Table 6).

Table 6. Rubric Used by All Faculty To Evaluate the Final Research Paper as well as Initial Proposal Final Research Paper Evaluation Reviewer Name: ____________________________________ Person Reviewed: ___________________________________ For more information on written communications consult the AAC&U values rubric25. SLO 2. Develop a research proposal based upon a thorough search of the scientific literature. Section

Notes

ABSTRACT: Overall research question; Experimental method; Fundamental results; Main conclusions

Introduce the reader to the project and what will be in the paper concisely. Entice a reader to be interested in reading further.

INTRODUCTION: Significance of project: MUST be clearly stated First outline the project – lay out fully what is the plan for the study. Use detailed literature to support your work. Include a purpose and relevance of the investigation Details of the study Appropriate number and use of citations

Demonstrates a thorough understanding of context, audience, and purpose that is responsive to the assigned task(s) and focuses all elements of the work Uses appropriate, relevant, and compelling content to illustrate mastery of the subject, conveying the writer’s understanding, and shaping the whole work. Demonstrates skillful use of high quality, credible, relevant sources to develop ideas that are appropriate for the discipline and genre of the writing

Score

/5

/15

Continued on next page.


Table 6. (Continued). Rubric Used by All Faculty To Evaluate the Final Research Paper as well as Initial Proposal Final Research Paper Evaluation EXPERIMENTAL PROCEDURE (Methodology): This is your own research procedure. Experimental design details. If you are doing a synthesis include % yield, include melting point, IR or NMR peaks. If unsure check any ACS Journal experimental section and model after it. Data collection and analysis details: equations, theory, brief error analysis. Details on the instrumentation / software used if any.

Methodology is skillfully developed according to the discipline. Clear but concise methods for all experiments relevant to final project (experiments from reproduction of literature just for practice is not necessary)

RESULTS AND DISCUSSION Clear, logical, and scientifically correct presentation of the relevant results; Use of figures and tables if any; Clear & reasonable figures and tables, with appropriate labels, units, markers, etc. Reasonable scientific interpretation of results. Depth of interpretation. Comparison to literature values if any; Discussion of any unusual results

Organizes and synthesizes knowledge in a logical way. Comparison to literature values if any Discussion of any unusual results

/15

/40

In consultation with the research advisor students may choose to split the Results and Discussion as separate components

CONCLUSION: Summarize your experimental results in the context of your original question. Must be clear. Must include future work. No extra data should be included in this section

Includes logical conclusions, limitations and suggestions.

REFERENCES: Appropriate format – See the ACS Guide

Appropriate number of references – 15-20 reasonable references

APPENDICES This section should include supplemental information that is not critical to, understanding the text.

Examples are: copies of IR spectra, NMR spectra, chromatograms, computer methods, etc.

/10

/5

Continued on next page.


Table 6. (Continued). Rubric Used by All Faculty To Evaluate the Final Research Paper as well as Initial Proposal Final Research Paper Evaluation STYLE AND CLARITY Overall organization Use of appropriate terminology Tables are properly formatted Figures are properly labeled

Is the title specific and informative? Does the title complement the abstract? Proofread the document

/Total Score

/10

/100

Research faculty evaluation of students include the evaluation of their students’ research progress and laboratory skills. In the senior research course, the verbiage of the rubric changes from making progress to having made progress (Table 7).

Table 7. Assessment of Research Skill Development and Results Assessment of Laboratory Skills and Results This assessment serves as a judgment of the faculty advisors rating of their student and any relevant comments. Development of Laboratory and Instrumentation skills: The student will begin to apply their scientific knowledge and laboratory skills to the execution of their project and develop new laboratory techniques and gain experience with state-of-the-art instrumentation. Assessment will be based upon the student’s level of proficiency in the various activities. Based on your area of expertise and the student’s progress through the semester, please rank the student in the following criteria. Each criteria will be averaged to give an overall rating out of and up to 10 points. Exceeding the expectations means a student is superior to most students typically for this stage in their research for any of these criteria. Assessment criteria

Exceeding Expectation

Above Average

Average

Below Average

Under Expectation

10

9- 8

7-5

4-3

2-1

Student has attempted to gain knowledge or expertise in any instrument, software and/or protocol necessary for their research (10 points) Student has begun to gather data, set-up experiments, devise their research instrumentation and/or proposed valid methods of gathering and analyzing data (10 points) Student has proficiency in their research, based on your expectation of a typical student at this point in their project. (10 points) Continued on next page.


Table 7. (Continued). Assessment of Research Skill Development and Results Assessment of Laboratory Skills and Results Student has displayed independence in the research project (10 points) Student has displayed an appropriate work ethic needed to be successful (10 points) Student has put into their project the appropriate amount of time for a 2 credit-hour course – physically, preparation for Friday meetings, and/or through research (including literature) (10 points) Average Score

/10 points (all criteria averaged together)

Comments: Research Results: All preliminary research and laboratory work will be recorded, and the results of the findings and laboratory work will be summarized into weekly progress reports. A summary of which is also attached for your records. 10% of this grade is based on your belief on the students’ current results. Exceeding the expectations - a student is superior to most students typically for this stage in their research for any of these criteria. Assessment criteria

Exceeding Expectation

Above Average

Average

Below Average

Under Expectation

10

9-8

7-5

4-3

2-1

Student has shown progress in their research activities (10 points) Student has begun to gather data, which they have demonstrated to you through reports, lab notebook, etc. (10 points) Student keeps a notebook with details of their work or keeps a detailed record of their work according to your preference. (10 points) Average Score

/10 points (all criteria averaged together)

Comments:

Conclusion: Outcomes, Revisions, and Survey Students beginning the research sequence, former students and faculty were surveyed in 2017 on their research skill development. Though our sample size is small, these data give some insight into the skills developed in the course and how faculty and student vary in their view of the courses. Current students just beginning the senior research sequence and the chemistry faculty were asked to assess the extent to which they themselves (or senior research students in general for faculty responses) have mastered different skills prior to the course. The current students were surveyed prior to any discussion of the course or assignments. Responses may be influenced by information and anecdotes relayed by their classmates or through other informal interactions with faculty. Summaries of current student and faculty responses are presented in Table 8. 325 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Table 8. Survey Results Regarding Research Skill Mastery Prior To Starting the Course (5 = Absolutely Agree, 1 = Absolutely Disagree) Criteria Average (Interquartile Range)

Current Students (n = 6)

Faculty (n = 5)

Difference (Student avg. – Faculty avg.)

Communication (oral)

3.5 (2.5 – 4)

2.6 (2 – 3)

+0.9

4.0 (3.25 – 4.75)

2.6 (2 – 3)

+1.4

Develop a research question/line of inquiry

2.8 (2 – 3.75)

2.8 (1 – 5)

0

Data analysis or Interpretation of result

4.3 (4 – 4.75)

2.6 (2 – 3)

+1.7

Evaluate the quality of literature sources

4.3 (4 – 4.75)

2.2 (2 – 2)

+2.1

Working independently

4.2 (4 – 4.75)

3.2 (2 – 4)

+1.0

Tolerance for obstacles or set-backs

3.5 (2.5 – 4)

2.2 (2 – 2)

+1.3

Understanding how knowledge is constructed

3.2 (3 – 4)

2.6 (2 – 4)

+0.6

Providing feedback to others

3.5 (3 – 4)

3.0 (3 – 3)

+0.5

Ethical conduct and behavior

4.7 (4.25 – 5)

4.0 (4 – 4)

+0.7

Communication (written)

Students beginning their senior research evaluated their skills on average 1 point higher than the faculty. These results are not necessarily surprising as the students may not have enough experience to assess the expected quality of research presentations and papers. Each student is assessing their own individual ability, while the faculty are assessing the students as a whole. Additionally, the students in mind for the faculty assessment may not be limited to those currently in senior research. Even with these caveats, identifying these differences in perceived mastery is useful for the faculty of record and for research mentors to design better assignments, feedback, and interventions. Improvements to the design of assignments that promote self-regulation and the feedback associated with each assignment may influence this difference. Webb et al. suggest that people may exhibit the “ostrich problem” in which they intentionally fail to evaluate the implications of information presented to them on their progress towards their goals (26). Making the progress towards the students’ goals more personal to them and combining with positive feedback prior to and during the senior research sequence are future goals for the course. 326 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Current students, former students who have graduated, and faculty members were also surveyed about how the senior research experience will improve (or has improved, for graduated students and faculty) their research skills. These results are presented in Table 9.

Table 9. Survey Results Regarding Average Reported Improvement in Research Skills (5 = Significant Improvement, 1 = No Improvement) Criteria Average (Interquartile Range)

Current Students (n = 6)

Former Students (n = 9)

Faculty (n = 5)


4.8 (5 – 5)

4.6 (5 – 5)

4.8 (5 – 5)


4.3 (4 – 5)

4.3 (4 – 5)

3.8 (3 – 4)


4.8 (5 – 5)

4.6 (4 – 5)

3.8 (3 – 5)


4.0 (4 – 4)

4.8 (5 – 5)

3.8 (3 – 4)


4.3 (4 – 4.75)

4.1 (3 – 5)

3.4 (3 – 4)


4.2 (4 – 4)

4.8 (4 – 5)

4.0 (4 – 4)


4.3 (4 – 5)

4.0 (4 – 5)

3.8 (4 – 4)


4.5 (4 – 5)

3.8 (3 – 5)

3.4 (3 – 4)


3.7 (3 – 4)

3.9 (4 – 5)

3.8 (4 – 4)


3.7 (3 – 4)

4.0 (4 – 5)

3.6 (3 – 5)


Our current students all expect the course to help improve most of their research skills, with ethical conduct ranking lowest in anticipated improvement. This may reflect confidence in their ethical abilities as these same students reported significant mastery of ethical conduct, on average, coming into the senior research courses. The expectations of current students tended to match up reasonably well with the reported gains from former students with three notable exceptions: After completing the senior research sequence, former students reported larger gains in their data analysis skills and in their ability to work independently compared to what the current students expect to gain. The current students rated themselves highly in these areas coming into the senior research courses, and this result may reflect student growth and maturity. Former students reported lower gains in understanding how knowledge is constructed than expected by current students, and this may point to an area for improvement. Aside from oral communication and providing feedback to students, the faculty again rated the improvement in each criteria lower than both former and current students. This may mean that the faculty have a specific expectation that has not been fully met and/or that they continue to identify areas for potential improvements in the course. Oral communication is highly stressed throughout the course, but the written assignments are only submitted a few times a semester. Written communication is one area that we will be working on improvements, but the students’ prior experience in pre-requisite or supporting courses may need strengthening with respect to writing. In general, the faculty see growth in the students on presentation and data analysis skills, but writing skill improvement is more student-specific. There is a wide range of improvement, but in general, students take the weekly suggestions for revision of subsequent evaluation of data and presentations. The written communications are one area faculty report that needs improvement. The grades on the final proposal and research papers range from the low 90s to 60s (the average of the faculty evaluations). After the first semester, students are given corrections to revise their proposal. It may be helpful to track the comments and revisions made based on the comments. Former students were asked to identify which of the research skills they currently use or anticipate using in their careers. These results are presented in Table 10. While the sample size is small, it appears that our former students have found that these research skills are more broadly applicable to their post-graduation careers. More than 75% of our former students reported that communication (both oral and written), working independently, safe conduct of work, data analysis, providing feedback and ethical behavior skills were most commonly used. Not all the former students surveyed went on to graduate school, so evaluation of literature skills and understanding how knowledge is constructed may not be particularly relevant skills in the industries where these students are working. Current and former students were also asked to comment on their perception or recollection of positive and negative aspects of having all the faculty members participating in the senior research sequence. A selection of their comments is included here: 328 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Table 10. Survey Results from Former Students Reporting Which Research Skills They Have Used since Graduation Criteria

Percentage reported (n = 9)


100.0


100.0


66.7


77.8


55.6


100.0


66.7


55.6


77.8


77.8

Safe conduct of experimental work

88.9

Current Student Comments “Having experienced scientists who have all spent years designing and developing research projects gives us a good resource to guide us toward creating one that will be successful for us when we don’t really have an idea of what to do. They will be able to help us become comfortable with answering questions and thinking outside the box and on our toes as we would have to working in the field or presenting at a conference.” “The faculty’s presence will be more stressful than if the course were only students and instructor because we are subject to more criticisms and denunciations. In the long run, this will make presentations and overall research better but can also be detrimental to the students’ well-being academically.” Former Student Comments “For me their presence made sure that I knew my research and was confident in my reporting. I was shy in the beginning, but now that I’m in grad school I jump to be first to present! “Their presence certainly added to the stress of the course but this was not a negative, simply an obstacle to overcome that made the course more challenging and thus more rewarding.” All the students reported some anxiety about the presentations for a variety of reasons, including not being sure how to please each professor, the depth of questions from different sub-disciplines, and general nervousness associated with the presence of more than the instructor of record. More recent former 329 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

students reported less anxiety or fear of the senior research sequence, which may be the result of intentionally incorporating more positive feedback along with suggestions for improvement. These students likely had more interactions earlier in their academic career with other students going through the senior research sequence, and this may have helped them develop a better expectation of what was expected in the course. Similarly, to Finklestein and Fishbach’s study, students may also be more respective due to positive feedback also given (22). Additionally, weekly progress reports have been used for the past two years. While students have always been expected to meet with their research mentors, there was no formal assessment of their participation or preparation for the upcoming week. Incorporation of the progress reports helped to facilitate discussions between the student and mentor as well as a mechanism for honest feedback on a regular basis. Developing a sense of community between the faculty and students and providing multiple perspectives are some of the major goals of the presence of the faculty. Though the broader faculty participation causes some anxiety to students, they do see value in these interactions as seen above. Some improvements to the rubrics that include specific prompts for positive feedback may be helpful, and this may be an area for discussion between the student and the faculty mentor. It may also help to demonstrate to students that expectations among the faculty are similar enough and not to be distressed over, and that these interactions are a mechanism to obtain multiple perspectives on a problem. As stated before, this chapter has an emphasis on the assessment methods used in our research courses at FSC. As with any assessment, it is key to review the progress made in the course and to identify areas for improvement. The faculty were asked to comment on which element of the senior research sequence was least effective, why they felt it was less effective, and how it could be improved. Selected comments are included below:

Faculty Comments “The writing component of senior research is the least effective. Students do not process comments that are given to them and continue to write in the style that they started the course with. It could be improved by focusing on writing periodically, not just in reports but in short writing assignments or analysis of good and bad writing.” “Ethical conduct is not really addressed in this course, and I don’t think that we do a very good job of helping students evaluate the quality of their literature sources. With regard to assessing literature, the students often don’t have articles chosen sufficiently in advance for the faculty and student to meet and discuss it in depth, and it is difficult for faculty members with little or no background knowledge of the project and/or paper to help provide in depth, critical analysis with the students. This could be improved by devoting more time to activities to develop these skills, either within the senior research sequence or embedded in prior courses.” 330 Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

Skills such as ethical conduct, writing, and critical evaluation of literature are key areas that need revisions in the course, as echoed in the faculty statements above. We speculate that the students treat the senior research assignments in the course the way they would in their other courses – the work needs to be submitted by a certain deadline, with little thought paid to it once it is submitted. While the nature of research mentorship is understood regarding experimental design and implementation, it may not be apparent to students that the mentorship and collaboration ideally extends to discussions of literature, revisions of written assignments, and development of presentations as well. This could be addressed through the development of assignments for completion by the student with the mentor, akin to the weekly progress reports, but targeted at discussing a research paper or critiquing examples of written work. Alternatively, revisions to written work could be accompanied by statements by or discussions with the student regarding how each comment or request for improvement was addressed. The mentorship experience varies somewhat between different faculty members and different students because of time constraints and personalities. The success of students in the senior research sequence is often attributed to the individual student, but mentors may be having effective interactions or identifying practices that work well for them that are not being shared with the other faculty. This is most likely because of a lack of time or more pressing issues. While we are implementing assignments to help students’ metacognition, the same may be needed for the faculty to help further harmonize expectations and to share successes. Sharing successful mentoring strategies, guided by literature (27) may be key to the next phase of revisions for the courses. Additional issues related to scalability of this format have also been identified by the faculty. Only a limited number of students can present during any given class period, and we feel comfortable with the development that students have demonstrated while presenting every other week. As the course enrollment grows, this will stress the availability of students and mentors to have quality interactions. Offering multiple sections with smaller enrollments and rotating faculty members between sections for the weekly presentations can help provide an equivalent experience without necessarily increasing faculty workloads. Even with these needed improvements and considerations, the senior research sequence developed over the past five years has been broadly successful. Our alumni report that the skills they developed through their senior research experience have been useful in their post-undergraduate careers. These survey results point to areas for improvement, both in skills that the faculty anticipated as well as in how we further develop those student skills. Although it requires a concerted departmental effort, expanding the number and roles of faculty involved in the senior research sequence has helped to make it a signature experience of the chemistry program at Florida Southern College.


References 1.

2. 3. 4.

5.

6. 7.

8.

9. 10.

11.

12.

13. 14.

15.

Weaver, G. C.; Russell, C. B.; Wink, D. J. Inquiry-based and research-based laboratory pedagogies in undergraduate science. Nat. Chem. Biol. 2008, 4, 577–580. Green, W. J.; Elliott, C.; Cummins, R. H. "Prompted" inquiry-based learning in the introductory chemistry laboratory. J. Chem. Educ. 2004, 81, 239. Rhoad, J. S. Written assignments in organic chemistry: critical reading and creative criting. J. Chem. Educ. 2017, 94, 267–270. Walker, J. P.; Sampson, V. Argument-driven inquiry: Using the laboratory to improve undergraduates’ science writing skills through meaningful science writing, peer-review, and revision. J. Chem. Educ. 2013, 90, 1269–1274. Weaver, G. C.; Donald, W.; Varma-Nelson, P.; Lytle, F.; Morris, R.; Fornes, W.; Russell, C.; Boone, W. J. Developing a new model to provide first and second-year undergraduates with chemistry research experience: early findings of the center for authentic science practice in education (CASPiE). Chem. Educ. 2006, 11, 125–129. Hinkhouse, H.; Del Carlo, D.; Isbell, L. Undergraduate research in chemistry: A comparison of two narratives. Chem. Educ. 2008, 13, 381–391. Novak, M.; Wang, Y.-T.; Ambrogio, M. W.; Chan, C. A.; Davis, H. E.; Goodwin, K. S.; Hadley, M. A.; Hall, C. M.; Herrick, A. M.; Ivanov, A. S.; Mueller, C. M.; Oh, J. J.; Soukup, R. J.; Sullivan, T. J.; Todd, A. M. A research project in the organic instructional laboratory involving the SuzukiMiyaura cross coupling reaction. Chem. Educ. 2007, 12, 414–418. Winstead, A. J. An Uundergraduate research based independent project: The optimization of the classic Diels-Alder microwave reaction. Chem. Educ. 2017, 15, 28–31. Russell, S. H.; Hancock, M.; McCullough, J. Benefits of undergraduate research experiences. Science 2007, 548. Hu, S.; Scheuch, K.; Schwartz, R.; Gayles, J. G.; Li, S. Reinventing undergraduate education: Engaging college students in research and creative activities. ASHE High. Educ. Rep. 2007, 33, 1–101. Hu, S.; Kuh, G. D.; Gayles, J. G. Engaging undergraduate students in research activities: Are research universities doing a better job? Innov. High. Educ. 2007, 32, 167–177. Hunter, A. B.; Laursen, S. L.; Seymour, E. Becoming a scientist: The role of undergraduate research in students’ cognitive, personal, and professional development. Sci. Educ. 2007, 91, 36–74. Bauer, K. W.; Bennett, J. S. Alumni perceptions used to assess undergraduate research experience. J. High. Educ. 2003 (2), 210. Crowe, M.; Brakke, D. Assessing the impact of undergraduate-research experiences on students: An overview of current literature. CUR Q. 2008, 28, 43–50. Jones, R.; Davis, S. Assessing faculty perspectives on undergraduate research: Implications from studies of two faculties. CUR Q. 2014, 34, 37–42. 332

Gourley and Jones; Best Practices for Supporting and Expanding Undergraduate Research in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2018.

16. Kremer, J. F.; Bringle, R. G. The effects of an intensive research experience on the careers of talented undergraduates. J. Res. Develop. Educ. 1990, 24, 1–5. 17. Kardash, C. M. Evaluation of undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors. J. Educ. Psychol. 2000, 92, 191–201. 18. Tanner, K. D. Promoting student metacognition. CBE Life Sci. Educ. 2012, 11, 113–120. 19. Bromfield Lee, D. Book of Abstracts, 245th ACS National Meeting, New Orleans, LA, United States, April 7-11, 2013; American Chemical Society: Washington, DC, 2013; CHED-1732. 20. Oliver-Hoyo, M. T.; Bromfield Lee, D. C. Multi-sensory chemical equilibrium investigation as a learning lab experience. Chem. Educ. 2010, 15, 181–183. 21. Kovarik, M. L. Use of primary literature in the undergraduate analytical class. Anal. Bioanal. Chem. 2016, 408 (12), 3045–3049. 22. Finkelstein, S. R.; Fishbach, A. Tell me what I did wrong: Experts seek and respond to negative feedback. J. Consum. Res. 2012, 39, 22–38. 23. AACU - Association of American Colleges & Universities. https:// www.aacu.org/ (accessed May 2017). 24. American Association of Colleges and Universities. Oral Communication Value Rubric. https://www.aacu.org/sites/default/files/files/VALUE/ OralCommunication.pdf (accessed October 2017). 25. American Association of Colleges and Universities. Written Communication Value Rubric. https://www.aacu.org/sites/default/files/files/VALUE/ WrittenCommunication.pdf (accessed October 2017). 26. Webb, T. L.; Chang, B. P. I.; Benn, Y. “The Ostrich Problem”: Motivated avoidance or rejection of information about goal progress. Soc. Personal. Psychol. Compass. 2013, 7, 794–807. 27. Thiry, H.; Laursen, S. L. The Role of student-advisor interactions in apprenticing undergraduate researchers into a scientific community of practice. J. Sci. Educ. Technol. 2011, 771.


Senior Undergraduate Research and Assessment ... - ACS Publications

Recommend Documents