Core Area Assessment Report Set 5 - C2: History, D2: Theology, & F: Visual and Performing Arts

History of Core Assessment Effort

The Core Assessment Working Group (CAWG) was formed in 2015 by the Core Advisory Committee (CAC), a committee made up of Department Chairs who represent each Core Area, faculty representatives from the School of Management and the School of Nursing and Health Professions, as well as the Associate Dean of Academic Effectiveness. CAWG was created with the support of the College Council, for the College of Arts and Sciences (CAS) in response to CAS Dean Marcelo Camperi’s call for an assessment of the Core Curriculum. That call was issued in response to faculty concerns surrounding the growth and coherence of the Core, and because the WASC Senior College and Universities Commission (WSCUC) requires that colleges and universities engage in regular curricular assessment in order to retain their academic accreditation. This is the first time that the USF Core Curriculum has been assessed since its inception in 2002.

In Spring 2015, the CAC, with the guidance of Associate Dean for Academic Effectiveness June Madsen Clausen, asked that a working group be created to design and implement a process for assessing the Core curriculum. CAWG was then constituted by Dean Camperi, with representation from different areas of the Core curriculum. The initial membership of CAWG included Tracy Benning (Sciences), Christine Young (Arts), Yaniv Stopnitzky (Social Sciences), and Ronald Sundstrom (Humanities, CAWG Chair). In Spring 2017, Joshua Gamson replaced Yaniv Stopnitzky; in Fall 2017, Ryan Van Meter replaced Ronald Sundstrom. In Spring 2018, Eve-Anne Doohan replaced Joshua Gamson. Thus the current membership of CAWG is Tracy Benning (Sciences, CAWG Chair), Christine Young (Arts), Ryan Van Meter (Humanities) and Eve-Anne Doohan (Social Sciences).

CAWG, with Associate Dean Clausen, created a timeline for assessing the Core curriculum (see Appendix A for Core Area Assessment Master Timeline; See Appendix B for C2, D2 and F Assessment Process Timelines), and concurrently began to investigate and design materials to support an assessment of the Core curriculum. The group conferred with a consultant, Carol Gittens (Associate Dean, Santa Clara University). Based on Gittens’s recommendation, CAWG consolidated the 48 learning outcomes from the 11 Core Areas (A1 through F) into a simplified and more measurable set of Higher Order Learning Goals (HOLGs) corresponding to each Core Area (see Appendix C). The HOLGs were then used to design a draft rubric for each Core Area, with the goal of developing rubrics specific enough to offer a meaningful measure of student learning in relation to Core learning outcomes and general enough that they could be applied to student work products from a variety of courses and disciplines within a Core Area.

The Core Areas were divided into five sets of 2-3 Core Areas, with each set due to be assessed once during a five-year period. This assessment process was broken into five phases, with staggered start dates for the different Core Area sets. The process includes the following steps:

1. Faculty in each Core Area are asked to align their Core courses with the respective Core learning outcomes;
2. Rubrics for each area are developed with input from faculty teaching in the relevant Core Area, and assessable student work products are identified;
3. Workshops are conducted to inform faculty about the assessment process and to recruit faculty raters;
4. Student work products are gathered and rated by paid faculty raters;
5. The assessment results are interpreted by CAWG and shared with faculty and administration;
6. Departments in the related Core Area reflect on the report and submit responses as described in the Reporting Protocol (see Appendix D).

CAWG has previously released reports for Sets 1, 2, 3 and 4. The Set 1 report (which includes Core Areas B2 and D1) was released on May 2, 2018; the combined Set 2 and 3 report (which includes Core Areas A1, A2, B1 and D3) was released on December 4, 2018; the Set 4 report (which includes Core Areas C1 and E) was released on May 21, 2019. All reports are archived at the CAS Office of Academic Effectiveness website.

Process and Methodology for Core Area Assessments - Set 5

Core Areas C2 (History), D2 (Theology & Religious Studies) and F (Visual & Performing Arts) are included in Set 5.

To begin the assessment of these Core Areas, syllabi from all courses in each of these Core Areas were reviewed during the Spring 2018 semester to check for alignment with the Core Learning Outcomes (CLOs). Alignment was defined as whether the CLOs and methods of assessment for each of the CLOs were included on each syllabus. The alignment check was organized by the Core Advisory Committee and headed by the Chair of each Core Area. For some Core Areas, the Core Area Chair reviewed all syllabi to check for alignment. For others, the Core Area Chair enlisted the help of the other members of the Core Area Committee (made up of Chairs and Directors of all Departments and Programs that teach in that particular Core Area). Faculty were notified via their Department Chair/Program Director if their syllabus was not in alignment, with the expectation that CLOs and methods of assessment would be included in future syllabi.

In Fall 2018, following the syllabi alignment check, all full-time and part-time faculty teaching C2, D2 and F classes were invited to attend rubric feedback sessions in their Core Area, to ensure that rubrics remained true to the intentions of the existing CLOs, would make sense to faculty raters, would reflect the language and practices of the Core Area, and when applied to student work products would provide an accurate measure of whether and to what degree the learning outcomes were achieved. The rubrics were each reviewed by faculty teaching in the Core Area during two or more rubric feedback sessions, before their final approval by the CAC in December 2018. (see Appendix E for the Rating Rubrics).
 
Additionally, at the rubric feedback sessions, faculty in each Core Area helped identify what type of student work products would be available and useful for assessment. For C2, submitted work products included analytical papers and exams. For D2, these included analytical and argumentative papers. For F, these included analytical papers as well as original artwork, both visual and performative (captured on video). In most cases in areas C2 and D2, a single work product was submitted for each student; for many students in area F, two products were submitted. Student work products were then randomly sampled using a stratified approach based on overall course enrollments. (See Appendix F for details on the numbers of courses, student work products, and sampled student work products).

Respective faculty in each Core Area were invited to apply to serve as faculty raters during a daylong assessment of student work products, for which they received a $250 honorarium. Four C2 faculty participated in the rating session held December 5, 2019. Five D2 faculty participated in the rating session held May 30, 2019. Four F faculty participated in the rating session held August 14, 2019 (see Appendix B for a list of participants). Rating was preceded by a calibration exercise, in which participants rated the same student work products and discussed any discrepancies in their application of the rubric. At all three sessions, a recalibration exercise was held after the lunch break.

A portion of the work products were also rated by a second faculty rater to check inter-rater reliability (this procedure is explained later in this report). In total, raters assessed about 15% of the submitted C2 work products, about 13% of D2 work products and about 10% of the submitted F work products.

Assessment Results

The established benchmark for Core assessment is that 70% or more of assessed student work products should achieve a rating of 3 or higher for each criterion in every Core Area. For every rubric, each criterion (while specific to that Core Area) uses the same 4 point rating scale as follows: 1= Below Expectations, 2= Needs Improvement, 3= Meets Expectations, 4= Exceeds Expectations.

C2 History

Results shown in Figures 1-4 reveal that a slight majority of students leave C2 courses able to explain a significant span of history over a particular geographic area, analyze histories of past societies and civilizations in their own context, articulate how historical forces shape the development of societies and civilizations, and apply historical thinking to ethical issues in the past and present.

Student performance is strongest on Criterion 2 at 65.7%, and lowest on Criterion 4 with 51% of student work products meeting the benchmark. None of the 4 criteria for this Core Area reached the 70 % benchmark. Specifically:

●    Criterion 1: Explain a significant span of history over a particular geographic area. Results of work product assessment for this criterion indicate that about 59% of student work met or exceeded the benchmark. This criterion had the second highest number of work products that rated below expectations at about 18% (Figure 1, Table 1).

Figure 1. Rating score distribution for sampled C2 work products. A score of 3 or higher indicates that a student has met Criterion 1 competency expectations.

●    Criterion 2: Analyze histories of past societies and civilizations in their own context. As stated previously, this criterion achieved the highest percentage of student work products rated as meeting or exceeding expectations in this area at 65.7 %, just slightly below the 70% benchmark. Less than 6% of work products failed to demonstrate this criterion (Figure 2, Table 1).

Figure 2. Rating score distribution for sampled C2 work products. A score of 3 or higher indicates that a student has met Criterion 2 competency expectations.

●    Criterion 3: Articulate how historical forces shape the development of societies and civilizations. About 60% percent of student work products were rated as meeting or exceeding expectations for this criterion. Twenty-six percent of work products were rated as needing improvement, with an additional 14% failing to demonstrate this criterion (Figure 3, Table 1).

Figure 3. Rating score distribution for sampled C2 work products. A score of 3 or higher indicates that a student has met Criterion 3 competency expectations.

●    Criterion 4: Apply historical thinking to ethical issues in the past and present. A slight majority of student work products were rated as meeting or exceeding expectations for this criterion at 51%. It achieved the lowest percentage of work products meeting the benchmark. In addition, about 23% were rated as needing improvement and 26% of the work products sampled failed to demonstrate this criterion, the highest percentage in this category for any of the criteria (Figure 4, Table 1).

Figure 4. Rating score distribution for sampled C2 work products. A score of 3 or higher indicates that a student has met Criterion 4 competency expectations.

Criteria	Percentage of Students Scoring 3 or Above for C2 Criteria
1	58.8
2	65.7
3	59.8
4	51.0

Table 1. Percentage of students meeting expectations on C2 assessed work. The percentage is based on the number of work products with a rating score of 3 or higher divided by the total number of rated products overall.

D2 Theology and Religious Studies

Results from the Core D2 assessment exhibit variable levels of proficiency; however, the majority of students leave Core D2 courses achieving below the desired benchmark of 70%. The one exception was criterion 2, where almost 76% of student work products assessed met or exceeded expectations and surpassed the 70% benchmark. Figures 5-8 illustrate the results for the four criteria assessed.
Specifically,

●    Criterion 1. Explain the beliefs and/or practices of at least one religious tradition. Results for this criterion indicate 59.1% of student work products met or exceeded the benchmark. About 16.5% of student work assessed failed to demonstrate this criterion (Figure 5, Table 2).

Figure 5. Rating score distribution for sampled D2 work products. A score of 3 or higher indicates that a student has met Criterion 1 competency expectations.

●    Criterion 2. Analyze how religion, theology, and/or spirituality interrelates with human experience. This criterion yielded the highest percentage of students meeting or exceeding the benchmark, with 75.7% of work products assessed scoring a 3 or 4, with only 3.5% of the work products assessed failing to demonstrate this criterion (Figure 6, Table 2).

Figure 6. Rating score distribution for sampled D2 work products. A score of 3 or higher indicates that a student has met Criterion 2 competency expectations.

●    Criterion 3. Discuss how religious and/or theological traditions can work effectively for social justice and the greater good. Only 32.2% of student work products sampled for assessment for this criterion scored 3 or above. A little over 35% of the work products failed to demonstrate this criterion; when combined with products that scored a 2 (needs improvement), a little over 67% of the work products did not meet expectations for this criterion (Figure 7, Table 2).

Figure 7. Rating score distribution for sampled D2 work products. A score of 3 or higher indicates that a student has met Criterion 3 competency expectations.

●    Criterion 4. Articulate one’s own spirituality. This criterion resulted in the lowest rating scores, with about 50% of the student work products failing to demonstrate this criterion (1). About 21% of the work products were scored as needs improvement (2). Thus overall, about 29% of student work products met or exceeded the benchmark for this criterion (Figure 8, Table 2).

Figure 8. Rating score distribution for sampled D2 work products. A score of 3 or higher indicates that a student has met Criterion 4 competency expectations.

Criteria	Percentage of STudents Scoring 3 or Above for D2 Criteria
1	59.1
2	75.7
3	32.2
4	28.7

Table 2. Percentage of students meeting expectations on D2 assessed work. The percentage is based on the number of work products with a rating score of 3 or higher divided by the total number of rated products overall.

F Visual and Performing Arts

Results from the Core F assessment indicate the majority of students leave Core F courses meeting or exceeding expectations the standard met for 2 of the 3 criteria. Figures 9-11 illustrate the results for the three criteria assessed. Specifically,

●    Criterion 1. Demonstrates an understanding of the significance of arts content (e.g., art works, artists, periods, genres, processes) in context ( e.g., formal analysis, historical, socio-political). Results for this criterion indicate 75.8% of student work products met or exceeded the 70% benchmark. Only 4% of student work assessed failed to demonstrate this criterion (Figure 9, Table 3).

Figure 9. Rating score distribution for sampled F work products. A score of 3 or higher indicates that a student has met Criterion 1 competency expectations.

●    Criterion 2. Uses technical and conceptual skills to produce creative or scholarly content. Results for this criterion indicate a Core Area high of 80.2% of student work products met or exceeded the benchmark of 70%. Only 1 of the student work products assessed failed to demonstrate this criterion (Figure 10, Table 3).

Figure 10. Rating score distribution for sampled F work products. A score of 3 or higher indicates that a student has met Criterion 2 competency expectations.

●    Criterion 3. Demonstrates a critical understanding of the cultural, social, or ethical meanings of artistic works. Assessment of this criterion resulted in 61.5% of the student work products meeting or exceeding the 70% benchmark. This was the lowest percentage for Core F criteria; however, 61.5% is not too far below the desired benchmark. About 14% of the work products assessed failed to demonstrate this criterion (Figure 11, Table 3). This was the highest percentage in this category for Core F criteria.

Figure 11. Rating score distribution for sampled F work products. A score of 3 or higher indicates that a student has met Criterion 3 competency expectations.

Criteria	Percentage of Students Scoring 3 or Above for F Criteria
1	75.8
2	80.2
3	61.5

Table 3. Percentage of students meeting expectations on F assessed work. The percentage is based on the number of work products with a rating score of 3 or higher divided by the total number of rated products overall.

Reflections on Assessment Results

C2 History

None of the C2 criteria met the 70% benchmark of meeting or exceeding expectations, though Criterion #2 came very close. In the debriefing that followed the rating session and in written feedback, faculty raters reported that their largest challenge in assessment was that many work products didn’t map well onto the rubric. For most raters, applying the rubric to a research paper, longer essay or essay exam was straightforward. For other types of assignments ranging from multiple choice exams (many submitted without the test key) to weekly reading summaries, the application of the rubric was quite challenging.
 
Indeed, raters also noted that such a wide variety of work products was a hindrance of its own. We recommend incorporating these observations into any future assessment of C2.

Criterion #4 had the fewest number of work products rated as meeting or exceeding expectations. This was also the criterion for which there were the most products rated below expectations. It is difficult to precisely account for this finding. On the one hand, the C2 rubric dictates that work products are rated “below expectations” when the learning activity described in the criterion is not present (in this case, “applying historical thinking to ethical issues”) or is present but contains excessive errors. So it’s possible that the work products submitted for assessment did not ask students to enact this learning activity, and therefore it wasn’t present. It is also difficult to imagine a set of work products where excessive errors are present only for one learning activity but no others. On the other hand, at least one faculty rater noted the difficulty of assessing work products for the application of ethical considerations in the first place. It was suggested that this learning activity is often performed in classroom discussion or other in-person activities. Raters also discussed the redundancy of requiring students to apply ethical considerations in C2 when an entire core area is already devoted to ethics (D3).

It’s also worth noting that during the C2 assessment process, from rubric development to the rating session, C2 faculty raised the possibility of submitting and assessing more than only 2 work products for each student. CAWG limited the number of work products for each student to 1 or 2 for the sake of faculty rater ease. In light of the concern that many of the work products seemed ill-suited to the rubric, for any future assessment, this limit should be revisited.

D2 Theology and Religious Studies

Only one of the Core D2 criteria met the 70% benchmark of meeting or exceeding expectations. The comments from the Theology and Religious Studies faculty raters provide some insight into why this may be the case. Several faculty raters mentioned that criteria 3 and 4, tapping into social justice and personal spirituality respectively, were difficult to assess. Mentioned specifically was that many of the work products that were rated did not seem to include either of these elements. This could have been a work product issue (i.e., perhaps the assignments submitted did not include a reflection on social justice or one’s personal spirituality), or it could be because of a larger issue. Perhaps the courses in D2 do not include a focus on social justice and an articulation of students’ own personal spirituality. These results may suggest that the faculty in D2 may wish to discuss whether these two elements of social justice and personal spirituality are important and necessary for Core courses to include. If they are important, then this may necessitate a discussion of how to best include these elements in Core courses. One faculty rater in particular said that the department faculty should discuss pedagogical techniques for how to implement criteria 3 and 4.

Another potentially problematic finding was that the first criterion of explaining the beliefs and/or practices of at least one religious tradition did not meet the benchmark of 70% of students meeting or exceeding expectations. This criterion reflects the knowledge in the discipline. Faculty raters expressed that this criterion was easier to apply to the rubric, which may imply that the lower level of achievement is associated with students not being able to do this to the extent that they should (rather than the lower rate of achievement being due to an inappropriate work product issue).

F Visual and Performing Arts

The Core F area has been quite successful in teaching students to demonstrate an understanding of the significance of arts content in context and to use technical and conceptual skills to produce creative or scholarly content. Student learning was rated well above the benchmark in both Criteria 1 and 2, providing a solid foundation for creating even more rigorous benchmarks for future student learning. Criteria 3 was less successfully achieved, with 61.5% of students demonstrating a critical understanding of the cultural, social, or ethical meanings of artistic works. While it is encouraging that this figure is still approaching the benchmark, there may be more to be done to enhance student learning in this area. It should also be noted that the highest number of student works products scored a “1” in this area, suggesting that some of the work products may not have been designed to reflect the learning described by Criteria 3. As was noted previously, Core F submitted two work products for most students (which was the limit that CAWG allowed for the sake of faculty rater ease.) However, for future assessments, it may be relevant to adjust the number of work products submitted for Core F students and/or to coach faculty on work product selection so that rating is more effective using the current rubric.

Inter-Rater Reliability Analysis for Set 5 Raters

Inter-rater reliability is a numerical estimate that measures the degree of agreement among raters when assessing the same work product. Inter-rater reliability was examined to ensure that the assessment process was both accurate and consistent. We used a basic two-rater model to calculate percent agreement with both exact and adjacent ratings scored as agreement. Out of the 102 work products rated for C2, a sample of 23 work products (or 22.5 %) was rated twice; out of 115 total work products rated for D2, a sample of 29 work products (or 25.2%) was rated twice; and out of a total of 91 work products rated for F, a sample of 24 work products (or 26.4%) was rated twice.

Rule of thumb benchmarks for rater data containing 4 or fewer categories is that a percent agreement of 90% (or higher) constitutes a high agreement, and 75% is considered minimal agreement. Using these benchmarks, rater agreement on C2 criteria was variable, with Criterion 2 having the highest percent agreement at 91.3%, slightly above the high agreement benchmark (Table 4). This was followed by Criterion 3 at 82.6% (Table 4). Criteria 1 and 4 had the lowest percent agreement with 78.3%, just above the minimal agreement benchmark (Table 4). Results from inter-rater reliability for C2 reflect the challenges expressed by raters with the difficulties of applying the rubric to the submitted work products.

Criteria	C2 Rater Percent Agreement
1	78.3
2	91.3
3	82.6
4	78.3

Table 4. Inter-rater reliability for C2 criteria based on a two rater method of agreement.
 
Percent rater agreement for D2 criteria ranged from 85.2% to 93.1%, resulting in high agreement for 2 out of 4 criteria (Table 5). Criterion 2 had the lowest percent agreement at 85.2% but was still well above the minimal agreement benchmark, as was the case for Criterion 1, the second lowest percent agreement at 88.9%. Overall, there was good inter-rater reliability among D2 raters for all criteria.

Criteria	D2 Rater Percent Agreement
1	88.9
2	85.2
3	93.1
4	93.1

Table 5. Inter-rater reliability for D2 criteria based on a two rater method of agreement.

Finally, percent rater agreement for F criteria ranged from 87.5% to 95.8%, resulting in a high agreement for 2 out of 3 criteria (Table 6). Criterion 2 had the lowest percent agreement at 87.5%, but was only slightly below the high agreement benchmark. Criterion 1 had the highest percentage agreement at 95.8%, followed by Criterion 3 at 91.7%. Overall, there was very good agreement among the Core F raters.

Criteria	F Rater Percent Agreement
1	95.8
2	87.5
3	91.7

Table 6. Inter-rater reliability for F criteria based on a two rater method of agreement.

Next Steps for Core Assessment Reports

Following the release of the Core assessment report and any department-specific data, departments and programs will be required to offer their interpretations of the results and to specifically evaluate the Core Learning Outcomes (CLOs) as they apply to courses they teach. Departments and programs will be required to provide feedback on the current set of CLOs for their Core Area and comment on both strategies to address deficiencies that were identified in the assessment process and whether CLOs should be modified as a result of the assessment. Potential outcomes of department or program level analysis of Core assessment results might include (but are not limited to): 1) modification of current CLOs, 2) selection of more appropriate student work products for future assessment efforts, 3) suggestion of modifications to the rubric used for the assessment and finally, 4) identification of changes to specific Core courses to better align with CLOs. This information will be collected in a simple Google form designed to capture faculty sentiment at the department or program level after discussion of the assessment results. Once submitted, this form will be sent to the relevant Core Area Chair, the Core Advisory Committee (CAC) co-chairs and the Associate Dean of Academic Effectiveness. A timeline for reporting feedback will be established to align with other required assessment activities and reports for the College. Information from these reports will be used to inform Core Area Chairs and the CAC on the state of the Core curriculum and should provide the data necessary to guide any subsequent changes to the Core curriculum.

APPENDIX A: CORE AREA ASSESSMENT MASTER TIMELINE

CORE ASSESSMENT TIMELINE

Appendix B: Assessment Process Timeline

FT = Full-time, PT = Part-time, PHP = Preferred Hiring Pool

• 9/11/18 CAWG attends A+A department meeting to review assessment process
• 9/12/18 CAWG attends History department meeting to review assessment process
• 9/25/18 CAWG attends THRS department meeting to review assessment process
• 9/28/18 CAWG attends PASJ department meeting to review assessment process
• 10/5/18 CAWG Meeting —D2 Rubric Review - Participating faculty: Mark Miller (FT - THRS), Cathal Doherty, SJ (FT - THRS), Vijaya Nagarajan (FT - THRS), Tsering Wangchuk (FT - THRS)
• 10/11/18 CAWG Meeting —C2 Rubric Review - Participating faculty: Michael Stanfield (FT - History)
• 10/19/18 CAWG Meeting—F Rubric Review - Jessica Snow (PT - A+A), Christine Young (FT - PASJ), Gabe Maxson (FT - PASJ), Sergio De La Torre (FT - A+A), Rachel Beth Egenhoefer (FT - A+A)
• 10/23/18 D2 Rubric Review - Participating faculty: Vijaya Nagarajan (FT - THRS), Aaron Hahn Tapper (FT - THRS), John Nelson (FT - THRS)
• 11/2/2018 CAWG Meeting — C2 Rubric Review —Participating faculty: Heather Hoag (FT - History)
• 11/7/2018 CAWG attends History department for C2 Rubric review
• 11/15/18 CAWG Meeting – F Rubric Review - Participating faculty: Kate Luscheck (FT - A+A), Karen Fraser (FT - A+A), Amie Dowling (FT - PASJ), Steven Doctors (PT - A+A), Stephanie Hunt (PT - PASJ), Eric Hongisto (FT - A+A)
• 11/30/18 CAWG Meeting — C2 Rubric Review - Participating faculty: Martin Clausen (FT - History)
• 12/7/18 CAWG attends PASJ department meeting – F Rubric Review - Participating faculty: Amie Dowling (FT - PASJ), Alexandra Amati (FT - PASJ), Byron Au Yong (FT - PASJ), Christine Young (FT - PASJ), Roberto Varea (FT - PASJ)
• 12/13/18 Email vote by Core F faculty on Rubric Criterion #3 — Amie Dowling (FT - PASJ), Alexandra Amati (FT - PASJ), Byron Au Yong (PT - PASJ), Christine Young (FT - PASJ), Roberto Varea (FT - PASJ), Steven Doctors (PT - A+A), Rachel Beth Egenhoefer (FT - A+A), Jessica Snow (PT - A+A), Eric Hongisto (FT - A+A), Kate Luscheck (FT - A+A)
• 12/5/18 C2 and D2 Rubrics approved at CAC meeting
• 12/14/18 F Rubric approved by CAC
• 4/5/19 Faculty Rater Applications Due for D2 and F
• 5/30/19 D2 Rating Workshop - Led by CAWG. Participating faculty: Natalie Latteri (PT - THRS), Mark Thomas Miller (FT - THRS), Vijaya Nagarajan (FT - THRS), Stephen Black (PT - THRS), Andrei Antokhin (PT - THRS)
• 8/14/19 F Rating Workshop - Led by CAWG. Participating faculty: Mark Taylor (PT - Media Studies), Jessica Snow (PT - A+A), Karen Fraser (FT - A+A), Natacha Ruck (PT - Media Studies)
• 12/5/19 C2 Rating Workshop - Led by CAWG. Participating faculty: Uldis Kruze (FT - HIST), Elliot Neaman (FT - HIST), Katrina Olds (FT - HIST), Heather Hoag (FT - HIST).

APPENDIX C: HIGHER ORDER LEARNING GOALS (HOLGS)

Created by CAWG to simplify the Core Area Learning Outcomes into more consistent and more measurable terms.

Note: the Higher Order Learning Goals (HOLGs) do not replace the Core Learning Outcomes (CLOs). CLOs will still be used to evaluate whether a course should receive Core designation. HOLGs will be used exclusively for assessing student learning in Core courses.

WASC	ILO	HOLG	CLO
A1: PUBLIC SPEAKING
Written Communication Oral Communication Critical Thinking	ILO 1 ILO 3	Analyze, interpret, and evaluate using rhetorical concepts and principles, the effectiveness of their own and others' communication in both academic and civic contexts, and identity ethical problems in public address.	A1: 4 A1: 5
Written Communication Oral Communication	ILO 4	Compose and present well-organized speeches, and well-reasoned, appropriately supported oral arguments.	A1: 1 A1: 2 A1: 3
A2: RHETORIC AND COMPOSITION
Written Communication Oral Communication Critical Thinking	ILO 1 ILO 3	Analyze, interpret, and evaluate linguistic and rhetorical strategies used in a variety of texts, and connect multiple texts in an argumentative essay, making comparisons and contrasts between them.	A2: 1 A2: 2
Information Literacy	ILO 5 ILO 6	Compose sophisticated research questions and arguments in response to those questions, conducting library research, and using academic documentation methods.	A2: 3 A2: 4 A2: 5
B1: MATH
Critical Thinking Quantitative Reasoning Information Literacy	ILO 3 ILO 6	Design and implement mathematical solutions to algebraic, algorithmic, statistical, numerical, or computational problems.	B1: 1 B2: 2
Critical Thinking Quantitative Reasoning Information Literacy	ILO 6	Evaluate the validity of a solution and its relevance to the original problem using quantitative reasoning as the norm for decision making.	B1: 3
B2: SCIENCE
Critical Thinking Quantitative Reasoning Information Literacy	ILO 3	Demonstrate literacy in the content and principles of a scientific discipline.	B2: 1
Critical Thinking Quantitative Reasoning Information Literacy	ILO 1 ILO 3 ILO 6	Conduct laboratory or field procedures that explore content, principles and application of scientific disciplines in a socially responsible manner.	B2: 2 B2: 3 B2: 4
C1: LITERATURE
Critical Thinking Information Literacy	ILO 1 ILO 3	Analyze, interpret, and evaluate the historical, social, and cultural influences that inform diverse literary works.	C1: 1 C1: 2 C1: 3
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 4	Articulate the ideas, plural meanings, moral and social implications, and formal features of literary works.	C1: 2 C1: 3 C1: 4
C2: HISTORY
Critical Thinking Information Literacy	ILO 1 ILO 3	Analyze, interpret, and evaluate a significant span of history over a wide geographic area, and the histories of past societies and civilizations using the values and standards of their own contexts and times.	C2: 1 C2: 3
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 4	Articulate how significant historical forces shape the development of societies and civilizations, and use historical thinking to consider ethical issues in the past and present.	C2: 2 C2: 3 C2: 4
D1: PHILOSOPHY
Critical Thinking Information Literacy	ILO 1 ILO 2 ILO 3	Analyze, interpret, and evaluate central philosophical issues.	D1: 1 D1: 2
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 4 ILO 5	Articulate using philosophical methods primary philosophical themes and issues found in the writings of the major philosophers.	D1: 3 D1: 4
D2: THEOLOGY AND RELIGIOUS STUDIES
Critical Thinking Information Literacy	ILO 1 ILO 2 ILO 3	Analyze, interpret, and evaluate the value of how religion, theology, and spirituality underlies and correlate with a broad range of human experience.	D2: 1 D2: 2
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 4 ILO 5	Articulate the similarities and differences among diverse religious traditions and their ethical and social implications.	D2: 3
D3: ETHICS
Critical Thinking Information Literacy	ILO 1 ILO 2 ILO 3	Analyze, interpret, and evaluate central ethical issues concerning right and wrong; good and bad; and equality, justice, and rights.	D3: 1 D3: 2 D3: 3
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 4 ILO 5	Articulate ethical theories and values and apply them in professional and personal decision making.	D3: 4 D3: 5 D3: 6
E: SOCIAL SCIENCES
Critical Thinking Information Literacy	ILO 1 ILO 3	Analyze, interpret, and evaluate issues regarding humans and the processes that shape their relationships, institutions, and interactions with their environments.	E: 1 E: 2 E: 3
Critical Thinking Quantitative Reasoning Information Literacy	ILO 3 ILO 6	Use qualitative or quantitative data, analysis, or theory to evaluate causal arguments in the social sciences.	E: 2 E: 3 E: 4
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 4	Articulate social science arguments that recognize connections between the social, economic, political, and environmental spheres of human life in a socially responsible manner.	E: 1 E: 5 E: 6
F: VISUAL AND PERFORMING ARTS
Critical Thinking Information Literacy	ILO 1 ILO 3	Analyze, interpret, and evaluate the aesthetic, historical, socio-political, and cultural influences that inform diverse art works.	F: 1 F: 2
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 6	Apply conceptual and technical skills related to an artistic discipline by engaging in creative and scholarly processes.	F: 2 F: 3
Written Communication Oral Communication Critical Thinking	ILO 3 ILO 4	Articulate the ethical and socio-political significance of the content and form of artistic works and the processes used to create them.	F: 1 F: 2 F: 4

APPENDIX D: CORE ASSESSMENT REPORTING PROTOCOL

1. In the reporting process, CAWG prepares a draft report. If Core Areas are made up of more than one department, then individual department data will be reported separately as an addendum, and only aggregated data for the Core Area will be included in the main report.
2. The draft main report and addenda reports will be shared with CAC, and CAC will provide feedback to CAWG about the reports.
3. CAWG will incorporate feedback and submit final drafts of the main report and addenda to the CAC for approval.
4. Once approved, the CAC co-chairs will submit the final reports to the CAS Dean for approval.
5. Once approved, the CAC co-chairs will share the main report with all Arts and Sciences faculty. However the addenda, with department-specific data, will be shared exclusively with each respective department, ideally via an in-person meeting with members of CAWG, in order to facilitate the “closing the loop” process.
6. Departments will share a written response to the main report and appropriate addendum with the Core Area Chair, the CAC co-chairs, and the ADAE, to capture their interpretations of the findings and their action plans for completing the “closing the loop” part of the assessment process.
7. Requests for course-specific data reporting from departments may be approved by the CAC if faculty within the department consent to its release using voting procedures outlined in their department bylaws. If any faculty member disagrees with, or has concerns about, the department consent to release course-specific reporting data, they may submit a statement of concern to the CAC along with the report of the department vote.

Developed 11/7/17 by CAWG Committee, Approved by CAC 11/29/17, Revised 2/2/18 by CAWG Revision approved by CAC 2/2/18

Appendix E: Rating Rubrics

C2 History Higher Order Learning Goals (HOLGs)

Students will:

1. Analyze, interpret, and evaluate a significant span of history over a wide geographic area, and the histories of past societies and civilizations using the values and standards of their own contexts and times.
2. Articulate how significant historical forces shape the development of societies and civilizations, and use historical thinking to consider ethical issues in the past and present.

C2 (History) HOLG Rubric
Criteria	Performance Standards
	Exceeds Expectations (4)	Meets Expectations (3)	Needs Improvement (2)	Below Expectations (1)
Explain a significant span of history over a particular geographic area.	Explain a significant span of history over a particular geographic area with exceptional clarity and accuracy.	Explain a significant span of history over a particular geographic area with clarity and accuracy.	Explain a significant span of history over a particular geographic area with limited clarity and accuracy.	Did not explain a significant span of history over a particular geographic area or did so with excessive errors.
Analyze histories of past societies and civilizations in their own contexts.	Analyze histories of past societies and civilizations in context with exceptional understanding and insight (e.g., depth of analysis, originality).	Analyze histories of past societies and civilizations in context with understanding and insight (e.g., depth of analysis, originality).	Analyze histories of past societies and civilizations in context with limited understanding and insight (e.g., depth of analysis, originality).	Did not analyze histories of past societies and civilizations in context or did so with excessive errors.
Articulate how historical forces shape the development of societies and civilizations.	Articulate how historical forces shape societies and civilizations with exceptionally effective argumentation, composition, technical skill, clarity, and appropriate sources.	Articulate how historical forces shape societies and civilizations with effective argumentation, composition, technical skill, clarity, and appropriate sources.	Articulate how historical forces shape societies and civilizations with partially effective argumentation, composition, technical skill, clarity, and appropriate sources.	Did not articulate how historical forces shape societies and civilizations with effective argumentation, composition, technical skill, clarity, and appropriate sources or did so with excessive errors.
Apply historical thinking to ethical issues in the past and present.	Apply historical thinking to ethical issues in the past and present with exceptional insight (e.g., depth of analysis, originality).	Apply historical thinking to ethical issues in the past and present with insight (e.g., depth of  analysis, originality).	Apply historical thinking to ethical issues in the past and present with limited insight (e.g., depth of analysis, originality).	Did not apply historical thinking to ethical issues in the past and present or does so with excessive errors.

Developed by CAWG Committee - July 2018 -– Approved by the CAC December 5, 2018

C2. History Core Learning Outcomes (CLOs)

1. Demonstrate a basic understanding of a significant span of history over a wide geographic area. (Criterion 1)
2. Articulate in writing and discussion their understanding of the ways significant historical forces (e.g., colonization, industrialization, war, social movements) shape the development of societies and civilizations. (Criterion 3)
3. Exhibit historical consciousness by understanding past societies and civilizations in their own contexts and times. (Criterion 2)
4. Demonstrate a sense of how historical thinking can establish a valuable framework for considering ethical issues in the past and present. (Criterion 4)

D2. Theology & Religious Studies Higher Order Learning Goals (HOLGS)

Students will:

1. Analyze, interpret, and evaluate the value of how religion, theology, and spirituality underlie and correlate with a broad range of human experience.
2. Articulate the similarities and differences among diverse religious traditions and their ethical and social implications.

D2 (Theology & Religious Studies) HOLG Rubric
Criteria	Performance Standards
	Exceeds Expectations (4)	Meets Expectations (3)	Needs Improvement (2)	Below Expectations (1)
Explain the beliefs and/or practices of at least one religious tradition.	Explain the beliefs and/or practices of at least one religious tradition with exceptional specificity and accuracy.	Explain the beliefs and/or practices of at least one religious tradition with specificity and accuracy.	Explain the beliefs and/or practices of at least one religious tradition with limited specificity and accuracy.	Does not explain the beliefs and/or practices of at least one religious tradition or does so with excessive errors.
Analyze how religion, theology, and/or spirituality interrelates with human experience.	Analyze how religion, theology, and/or spirituality interrelates with human experience with exceptional clarity and accuracy.	Analyze how religion, theology, and/or spirituality interrelates with human experience with clarity and accuracy.	Analyze how religion, theology, and/or spirituality interrelates with human experience with limited clarity and accuracy.	Does not analyze how religion, theology, and/or spirituality interrelates with human experience or does so with excessive errors.
Discuss how religious and/or theological traditions can work effectively for social justice and the greater good.	Discuss how religious and/or theological traditions can work effectively for social justice and the greater good with exceptional insight (e.g., depth of analysis, cleverness, originality).	Discuss how religious and/or theological traditions can work effectively for social justice and the greater good with insight (e.g., depth of analysis, cleverness, originality).	Discuss how religious and/or theological traditions can work effectively for social justice and the greater good with limited insight (e.g., depth of analysis, cleverness, originality).	Does not discuss how religious and/or theological traditions can work effectively for social justice and the greater good or does so with excessive errors.
Articulate one’s own spirituality.	Articulate one’s own spirituality with exceptional clarity and insight.	Articulate one’s own spirituality with clarity and insight.	Articulate one’s own spirituality with limited clarity and insight.	Does not articulate one’s own spirituality.

Developed by CAWG Committee – July 2018 -– Approved by the CAC December 5, 2018

D2. Theology & Religious Studies

1. Understand their own spirituality and recognize how religion, theology, and spirituality underlie and correlate with a broad range of human experience. (Human Dimensions of Religion, Theology, and Spirituality) Criteria 2 and 4
2. Understand, differentiate, and appreciate various religious traditions, as encouraged by Vatican II's stance on the Catholic Church's relationship with other faiths. This understanding will entail the creedal vision, moral teachings, historical context, social expression, and key rites and symbols of these faith traditions. (Religious Diversity) Criteria 1
3. Investigate and discuss how religious and theological traditions can work effectively for social justice and for the good of the entire human family and the environment that sustains it. (Social Justice) Criteria 3

F. Visual and Performing Arts (HOLGS)

Students will:

1. Analyze, interpret, and evaluate the aesthetic, historical, socio-political, and cultural influences that inform diverse art works. (Criterion A)
2. Apply conceptual and technical skills related to an artistic discipline by engaging in creative and scholarly processes. (Criterion B)
3. Articulate the ethical and socio-political significance of the content and form of artistic works and the processes used to create them. (Criterion C)

F (Visual & Performing Arts) HOLG Rubric
Criteria	Performance Standards
	Exceeds Expectations (4)	Meets Expectations (3)	Needs Improvement (2)	Below Expectations (1)
A. Demonstrates an understanding of the significance of arts content (e.g., art works, artists, periods, genres, processes) in context (e.g., historical, social, political).	Demonstrates an understanding of the significance of arts content in context with exceptional specificity and accuracy.	Demonstrates an understanding of the significance of arts content in context with specificity and accuracy.	Demonstrates an understanding of the significance of arts content in context with limited specificity or accuracy.	Does not demonstrate an understanding of the significance of arts content in context or does so with excessive errors.
B. Uses technical and conceptual skills to produce creative or scholarly content.	Uses technical and conceptual skills to produce creative or scholarly content with exceptional understanding, skill and insight (e.g., depth of analysis, astuteness, originality).	Uses technical and conceptual skills to produce creative or scholarly content with understanding, skill and insight (e.g., depth of analysis, astuteness, originality).	Uses technical and conceptual skills to produce creative or scholarly content with limited understanding, skill and insight (e.g., depth of analysis, astuteness, originality).	Does not use technical and conceptual skills to produce creative or scholarly content or does so with excessive errors.
C. Demonstrates a critical understanding of the cultural, social, or ethical meanings of artistic works.	Demonstrates a critical understanding of the cultural, social, or ethical meanings of artistic works with exceptional insight (e.g., depth of analysis, astuteness, originality).	Demonstrates a critical understanding of the cultural, social, or ethical meanings of artistic works with insight (e.g., depth of analysis, astuteness, originality).	Demonstrates a critical understanding of the cultural, social, or ethical meanings of artistic works with limited insight (e.g., depth of analysis, astuteness, originality).	Does not demonstrate a critical understanding of the cultural, social, or ethical meanings of artistic works or does so with excessive errors.

Developed by CAWG October 2018, approved by CAC 12/14/19

Area F: Visual and Performing Arts

Students will:

1. Demonstrate an understanding of the broad historical and theoretical foundations of an area of the Performing Arts: Dance, Music, and Theater; Fine Arts; Art History; Architecture or Design; through an examination of specific cultural, social, economic and political contexts. Topics should be ones that consider the field of study with depth and breadth. (Criterion A)
2. Demonstrate orally, in writing, or through production, an understanding of critical approaches to evaluating key works in the respective area and the specific movements and periods associated with it. (Criterion A)
3. Develop technical and conceptual skills related to the area by engaging in individual and/or collaborative classroom activities. (Criterion B)
4. Demonstrate a broader, critical understanding of specific communities of makers and users by attending presentations related to the subject art (lectures, performances, and museum or other exhibits.) (Criterion C)

Appendix F: Expanded Methodology and Numbers for Assessment Process

In Spring 2019 through Fall 2019 , the assessment time frame for C2, D2, and F, there were a total of 703 students enrolled in 21 sections of 14 different C2 courses, 858 students enrolled in 27 sections of 21 different D2 designated courses and 865 students enrolled in 36 sections of 23 different courses designated as Core F. At the beginning of the semester, a call went out to all Core faculty in these areas for student work products to be assessed. Corie Schwabenland Garcia, Academic Data and Assessment Analyst for the Office of Academic Effectiveness, was charged with collecting all of the student work products submitted by faculty. A total of 691 student work products were submitted for C2 courses. This number is less than the 703 students enrolled in Core C designated courses. A handful of C2 instructors submitted more than one work product per student and the number of submitted C2 products is even less than the submission number would indicate. Two Core C2 faculty failed to submit work products for 99 students enrolled in their courses. A total of 912 work products were submitted for Core D2, slightly more than the 858 students enrolled in Core D2 courses. The number of submitted products is higher than enrollment because several instructors submitted more than one work product for assessment.

Three faculty did not respond to the submission request (-136 work products) and one faculty member failed to submit a work product for every student in their course. The partial submissions reduced the work product pool by an additional 57 samples. A total of 916 work products were submitted for Core F, slightly more than the 865 students enrolled in Core F courses. As in Core D2 submissions, the number of submitted products is higher than enrollment because several instructors submitted more than one work product for assessment. Four faculty did not respond to the submission request which totaled 158 work products. A summary of enrollment numbers and overall assessment metrics are presented in Table 1. For the C2, D2 and F collected work products, Dean’s office staff then went through the submitted items and paired the submitted work products where appropriate. Approximately one-third of the submitted work products from each Core Area were then randomly selected to be available for assessment. For all work products, student names and any identifying information were redacted, and any grading that appeared on the products was removed. Out of the available pool of samples for C2, the 4 faculty raters assessed 102 student work products or 15% of the total number of products submitted on December 15, 2019; 115 student work products or about 13% of the total number of D2 products submitted were assessed on May 30, 2019 by 5 faculty raters; and 91 student student work products or about 10% of submissions were assessed by 4 faculty raters on August 14, 2019 for Core F. An inter-rater reliability check was performed during each rating session by having a subset of work products evaluated by two faculty raters. For C2, 22.5 % or 23 student work products were double-rated, for D2, 25.2% or 29 products were double-rated and 24 products or 26.4% were double rated for F. Those ratings were then used to perform an inter-rater reliability check as detailed in the report.

Fall 2018	Student Enrollment	Number of Work Products Submitted	% Compliance
C2	703	691 (837*)	82.6%
D2	858	912 (1407*)	64.8%
F	865	916 (1249*)	73.4%

Table 1. Core C2, D2 and F assessment metrics for student work products and faculty compliance. Student work products were requested for every student enrolled in Core C2, D2 and F courses. Faculty in each Core Area were asked to report what student work product(s) would be submitted and how they would be collected on an electronic work product submission form. Based on faculty responses to the form an expected number of product submissions was established; those numbers are shown in parentheses and denoted by an asterisk (*). Percent compliance for C2, D2 and F was calculated by taking the actual number of products submitted divided by the expected number of products as reported on the faculty submission form.