Institutional Assessment

Our philosophy on assessment is guided by our commitment to our students and the questions we ask to ensure quality. What are our students learning? Are our services the best they can be? The questions we ask about ourselves will vary from program to program and department to department. Whether they deal with students learning specific content, skills or attitudes or perhaps issues of student motivation and ability to monitor their own learning. Our assumption is that the key assessment questions are best known by the faculty and staff themselves. Finding ways to answer these questions is key to our student success.

Academic assessment seeks to answer the broad question, "What and how well do our students learn what we are attempting to teach them?" As such, academic assessment is not designed to evaluate individual faculty or even individual courses. It is designed to evaluate individual programs as a whole and to determine where the programs might be strengthened in order to improve our students' abilities to learn. The primary audience for academic assessments is not administrators or accrediting agencies, but, rather, the program faculty themselves.

Similarly, operational efficiency and effectiveness is the capability of SUNY Oneonta to deliver services to students in the most cost-effective manner possible while still ensuring the high quality of products and support. This is achieved by streamlining core processes in order to more effectively respond to student needs in a cost-effective manner. By focusing on continuous progression toward a meaningful but ambitious target, assessment methods are used to give feedback and guide SUNY Oneonta’s planning efforts.

Biology Program (BS) 21/22 Cycle

1. Outcome(s) assessed and related assessment question(s)

Biology Department PLOs

PLO 1: Diversity of Life. Apply an understanding of the connection of molecular genetics and diversity of life to recognize that diversity can be defined in many ways.

PLO 2: Morphology: Recognize the connection of the genetic code and morphological characteristics of an organism as it applies to molecules, cells, tissues, organs, organ systems and organisms to their structural features.

PLO 3: Homeostasis. Apply an understanding of regulation and maintenance of the genome; control of the flow of regulation of gene expression and epigenetics.

PLO 4: Scientific Inquiry. Generate questions, formulate predictions, and evaluate outcomes.

PLO 5: Social and Ethical Implementations of Science. Recognize contentious issues and compare competing viewpoints.

PLO 6: Data Analysis. Utilize basic math skills including algebra and statistical methods to recognize and evaluate sources of variation in data as well as interpret graphs and tables.

For this academic year, 2022-23, we completed 2 assessment protocols. For the first protocol, we collected data across all 6 PLOs as a set of online quizzes taken as a graduation requirement for Biology majors with senior status. We collected data on 77 graduating seniors from both Fall 2022 and Spring 2023. The senior data were collected at the end of each semester.

For our 2nd protocol, we more intensively collected data on PLOs 2, 4, and 6. PLO2 was also evaluated last year, but it is the 1st year for PLOs 4 and 6. We collected data on PLO2: Morphology in the three freshmen core courses in Fall 2022 and Spring 2023. Based on how our courses are now structured with PLO2 introduced in the freshmen core courses and then reinforced in all classes, we felt it important to assess the freshmen courses for an introductory-level of understanding and to assess some senior-level courses for more advanced knowledge. We used three short-answer questions at the introductory-level; these were presented to both the first-years and seniors. We also wrote four short-answer questions that targeted more advanced understanding; these were only administered to seniors. Students in Biol 1002, Biol 1004 and Biol 1006 were presented the 3 questions as a part of a graded homework assignment. The responses were collected and then randomly sorted. Four responses from each course were pulled from the group to evaluate. PLOs 4 and 6 are first introduced in the core 2000-level courses, so students in these courses were asked a set of multiple choice plus short-answer questions. Data were collected from 3 of the 4 courses each semester for a total of 54 evaluated responses.

Students in 4000-level courses were asked to respond to the three sets of PLO questions as a part of a graded homework assignment. All 50 collected responses were evaluated by the members of the assessment committee. The actual questions and rubrics used are attached at the end of the report.

2. Executive Summary of findings (500 words max):

Table 1: Senior Knowledge Quiz Results. Values in the table represent the percentage of 77 seniors whose answer totals fall into the four performance categories. The values in each row may total to more than 100% due to rounding.

Senior Knowledge Quiz Results

	Failed to Meet	Approached	Met	Exceeded
PLO1: Diversity	27.3	23.4	42.9	6.5
PLO2: Morphology	18.2	14.3	50.1	16.9
PLO3: Homeostasis	23.4	22.1	46.8	7.8
PLO4: Scientific Inquiry	20.8	28.6	20.1	29.9
PLO5: Social & Ethical Implications	20.8	18.2	44.2	16.9
PLO6: Data Analysis	16.9	15.6	46.8	20.1
Overall Score Based on total of 6 quiz sections	19 students (24.7%) scored below 50% overall	22 students (28.6%) scored 50-59% overall	35 students (45.8%) scored 60-79% overall	1 student (1%) scored 80% or higher overall

Across the 6 sections there were a total of 60 questions. Many students did very well in 1, 2 or 3 of the six PLOs, but struggled with other sections and therefore did not score well overall. Twenty-nine of the 36 students who met or exceeded expectations scored above 6 out 10 on all six quizzes. The median value for the senior scores was 35/60, in the ‘approached expectations’ range. So overall, we are not satisfied with the results as fewer than 50% met our expectations.

Many students struggled with PLO1 – Diversity. One possible explanation for this, according to the students, was that they started with that set of questions, and it took a while for them to ‘get into a groove;’ they felt very nervous when they started and then relaxed a bit. Another possible issue is the subset of questions each got from the test bank; the questions ask about diversity at various organizational levels, and it is possible that their set had many which related to concepts that they may not have thought of in some time. We also note that many of the questions in the set relate to common misconceptions with those answers favored by students.

We really expected a much better performance on PLO4 – Scientific Inquiry. This is a concept that students have covered starting in middle or high school. When we looked at some of the student answers, we are considering that the 10 minute limit may have been a bit tight given that so many of the questions asked students to interpret novel information.

Table 2: Short-answer questions for PLO2 Results. Values in the table represent the percentage of students whose answer totals fall into the four performance categories. We evaluated a total of 16 seniors in three senior seminar courses plus one other 4000-level class (BIOL 4502 Animal Behavior), randomly selecting 4 students per section to evaluate. We collected data from 25 freshmen, 4 responses pulled from each of 6 Perspectives in Biology sections plus 1 response from the sole Biology major in a seventh section. Questions 1-3 were the same for freshmen and seniors.

Short-answer questions for PLO2 Results

	Freshmen	Seniors – Questions 1-3	Seniors – all Questions
Exceeded	36	68.8	37.5
Met	40	18.8	43.8
Approached but did not meet	12	12.5	12.5
Failed to Meet	12	0	6.3

[Note: four answers were pulled per section because that was the maximum number of artifacts in one section, and we chose to have a similar amount from each so as not to bias the data by instructor or course.]

More than 75% of first-year students met expectations. Morphology was specifically taught in the three freshmen courses, but may have been the 1st time the concept was presented to the students. The average score for each question ranged from 2.6 for questions 1 and 3 to 3.3 for question 2. The scores for question 1 were quite low because many students were scored a ‘0’ for plagiarizing/copying and pasting their answers. This year, we rephrased question 1 to lessen the temptation for students to copy and paste their answers, and it seems to have been effective in reducing plagiarism. The average overall score for the freshmen was 8.6, in the range of "met expectations.”

Nearly 90% of seniors met or exceeded expectations on the first three questions, and no seniors from our sample failed to meet expectations on those questions. More than 80% of seniors met expectations when we included all questions, and only one senior student from our sample failed to meet expectations. Last year, several senior students failed to meet expectations on questions related to this PLO, so we are encouraged by the improvement students have shown.

Table 3: Short-answer questions for PLO4 Results. Values represent the percentage of students whose answer totals fall into the four performance categories. We evaluated a total of 50 senior-class students. We collected data from 54 sophomore class students. Questions 1-13 were the same for freshmen and seniors.

Short-answer questions for PLO4 Results

	Sophomore-lvl	Seniors: Questions 1-13	Seniors: all Questions
Exceeded	40.7	60.0	46.0
Met	38.9	30.0	42.0
Approached but did not meet	18.5	10.0	10.0
Failed to Meet	1.9	0	2.0

Students in the sophomore-level courses are already doing well on aspects of the process of scientific inquiry. This is a general education concept and includes ideas emphasized in their science courses in high school. Our seniors show improvement on the same concepts presented to the sophomores but their performance overall was not much higher than the sophomores. This indicates that we should to examine the delivery of what we consider more advanced content in our courses. It may also reflect the amount of online exposure senior students received in their freshmen and sophomore years due to the COVID restrictions; this PLO would have been directly impacted with the lack of in-person labs.

Table 4: Short-answer questions for PLO6 Results. Values in the table represent the percentage of students whose answer totals fall into the four performance categories. We evaluated a total of 47 senior-class students. We collected data from 68 sophomore class students. Questions 1-11 were the same for freshmen and seniors.

Short-answer questions for PLO6 Results

	Sophomore-lvl	Seniors: Questions 1-11	Seniors: all Questions
Exceeded	4.4	21.7	28.3
Met	55.9	58.7	47.8
Approached but did not meet	29.4	6.5	15.2
Failed to Meet	10.3	13.0	8.7

Data analysis and anything related to mathematical concepts are generally weak areas for our students, but more than 75% of seniors met expectations and 60% of second-year students also met expectations. The seniors showed improvement in this area both on the sophomore set of 11 questions and overall. We know that many of our biology courses do not emphasize these skills as they focus on other lab-based skills, and plan to have a more concerted effort to think about how data analysis skills are presented in our upper-level courses.

3. Criteria for success:

A. Senior Knowledge Quiz

The six PLOs had 10 questions each. The questions in the PLOs were selected to cover biology across Domains and Kingdoms as well as from the cellular to the ecosystem level. Our standards for 10 questions total were:

8, 9 or 10 Exceeded expectations

6 or 7 Met expectations

5 Approached but did not meet expectations

≤4 Failed to Meet expectations

We set the expectations for overall performance on the exam based on what we expect of our seniors. Out of a total of 60 points, here are the total scores we used to evaluate our students:

≥48 Exceeded expectations (equivalent to 80% or higher)

36-47 Met expectations (above 60%)

30-35 Approached but did not meet expectations (minimum of 50%)

≤29 Failed to Meet expectations (below 50%)

We chose 60% for ‘meeting expectations’ because for a multiple choice exam, setting the bar where we would for a pass in a pass/fail course seemed most appropriate.

B. Short-Answer Assessment focused on Reassessing PLO2

For the Freshmen level, there were a total of 3 questions that we felt should be able to be answered at the end of a course where this information was 1st introduced. For the Seniors, there were a total of 7 questions, with the 1st 3 questions being the same as for the freshmen. The latter 4 questions were designed to ask students to have a greater understanding of aspects of Morphology. The questions in the PLO were written to allow students the freedom to answer based on their course background; e.g. if a student focused on cellular-level courses, then they could answer most questions from that point of view, whereas a student who focused on organismal courses could answer from an organismal point of view. Each question was evaluated out of 4 points.

4 Exceeded expectations: fully-developed and accurate explanations

3 Met expectations: met bench-mark answer, mostly accurate and complete response

2 Approached but did not meet: inaccurate and/or rather incomplete response

1 Failed to Meet expectations: attempted to answer, but incorrect; may contain logical errors or invalid conclusions

0 Did not answer, obviously plagiarized/ cut-pasted answers, cannot evaluate

For the freshmen, asked 3 questions, their total possible was 12.

10-12 Exceeded expectations

8-9 Met expectations

6-7 Approached but did not meet

≤5 Failed to Meet expectations

For the seniors, asked 7 questions, their total possible was 28.

23-28 Exceeded expectations

18-22 Met expectations

14-17 Approached but did not meet

≤13 Failed to Meet expectations

C. Short-Answer Assessment focused on PLO4

Sophomore level students in the core courses that first introduce PLO4 were asked 11 multiple choice questions and two short answer questions with multiple parts. These questions targeted the basic ideas in the scientific process.

Senior level students in a variety of 4000 level courses were asked to answer the same questions as the sophomores plus another 3 questions with multiple parts that ask for more extensive understanding of parts of the scientific process.

All of the multiple choice questions were graded as 0 or 1. The individual parts of the short answer questions were graded as 0, 1, 2 for Failed, Approached or Met expectations.

For the sophomores, their total possible was 21.

17-21 Exceeded expectations

13-16 Met expectations

9-12 Approached but did not meet

≤8 Failed to Meet expectations

For the seniors, their total possible was 39.

31-39 Exceeded expectations

24-30 Met expectations

16-23 Approached but did not meet

≤15 Failed to Meet expectations

D. Short-Answer Assessment focused on PLO6

Sophomore level students in the core courses that first introduce PLO6 were asked 9 multiple choice questions and two short answer questions with multiple parts. These questions targeted the basic ideas in data analysis.

Senior level students in a variety of 4000 level courses were asked to answer the same questions as the sophomores plus another 4 questions with multiple parts that ask for more extensive understanding of data interpretation.

All of the multiple choice questions were graded as 0 or 1. The individual parts of the short answer questions were graded as 0, 1, 2 for Failed, Approached or Met expectations.

For the sophomores, their total possible was 17.

14-17 Exceeded expectations

10-13 Met expectations

7-9 Approached but did not meet

≤6 Failed to Meet expectations

For the seniors, their total possible was 31.

25-31 Exceeded expectations

19-24 Met expectations

12-18 Approached but did not meet

≤11 Failed to Meet expectations

4. How, when, and to whom were results distributed and discussed?

Results were distributed to all Biology faculty via email after final grades were due in May. All faculty were given 14 days to read the summary data and comment via email to the department. About 50% of the faculty responded.

Example comments from faculty in reference to the Senior Knowledge Quiz results:

Our 4000 level course offerings are not representative of all of the PLOs, so students focusing on certain areas may be skewed in their exposure to the PLOs in the upper-level courses.

Students always seem to take the PLO1 quiz first, even when they can and are encouraged to do the quizzes in any order. It could be the numbers are lower since it is the first, and they are the most nervous.

Because this is set up as a graduation requirement, the students have related that they are really stressed going into the quiz, and this may contribute to the lower performance.

I like that students can get this immediate feedback on how well they learned the material when they take the quizzes. It can help them feel good when they know how they did. Some have told me it feels like preparation for the GREs or MCATs.

Example comments from faculty on the short-answer results:

For PLO2 at the freshmen level, it was distinctly noticeable that students who had taken or were concurrently taking BIOL 1004 did significantly better with their responses to PLO2 when in my BIOL 1006 course. While I did cover some aspects of morphology, it was obvious they got much more content in 1004. I suspect something similar will be true with PLO1 (diversity) in 1004 and 1006 being stronger but PLO3 (homeostasis) being stronger in 1002.

Example comments from faculty on assessment generally:
We acknowledge that most of our senior students started college in 2019 and ended up with the end of their freshman year and all of their sophomore year online. This means that they missed many of the core courses and many laboratory experiences. We know that this likely had an impact on senior performance on the Learning Outcomes.

Many upper-level courses with labs address lab techniques and bench skills but not as much data interpretation (PLO6). This is not a weakness and is often linked to the course content.

I wish we could find a way to capture data on freshmen and transfers separately. I would be interested to know just how “seamless” transferring really is.

What we may be seeing in these data are that some students are getting reinforcement on the PLOs and others are only getting a topic once or twice.

5. Describe changes you plan to implement, use of results, and program or assessment modifications:

The department plans to review the current list of required courses and the PLOs. Many are not as pleased with the results of the new plan with students choosing 2 of 3 1000 level lecture courses (unless they are pre-health, in which case there is no choice) plus only a single lab experience freshmen year. We also need to relook at where and how the different PLOs are being covered in the upper-level courses to be sure that no matter which courses are taken, students obtain the experience expected.

As we look at rearranging the curriculum, it will be important to consider a backwards design. What do we want them to be able to do? Given those goals, how are we going to reinforce those goals multiple times as they move through the curriculum?

6. Timeline for implementation and reassessment

PLOs 4 and 6 will be re-evaluated next year. These concepts are initially introduced at the sophomore level in the core courses. Because we found a large number of seniors and juniors taking our 2000-level core courses, we do not want to collect preliminary data in those courses again. Instead, we feel that we should be collecting data on our incoming freshmen. As such, we will collect data in Fall 2023 from BIOL 1002, 1004 and 1006.

Senior Seminar is no longer a required course for students, so instead, we will use the time for the Senior Exit Exam to collect artifacts on PLOs 4 and 6 from our seniors. We will not be using the Senior Knowledge Quiz next year as we do not feel that we have gotten data as useful as we would like from it; meaning the department has not felt it could use the information to make any changes. Having only 10 questions per topic and all multiple choice seemed to be a poor representation of their knowledge. Even if we removed the time limits, which was suggested, we have not been pleased with the questions and the format. We also had many faculty who did not like putting a minimum score on student performance for an outcome that reflected more on the department than on the student. However, we had to have some incentive for the students to put in their best effort. This has led to a lot of tension and anxiety in the faculty and the students.