Appendix A
Based on recommendations from the 2019 assessment, the Writing and Rhetoric Program Director revised the rubric in consultation with the University Assessment team, focusing on clarifying criterion descriptors and performance-level language. The Program’s curriculum committee piloted the revised rubric using a small set of practice artifacts and provided feedback, which informed final refinements prior to scoring for the 2025 assessment summit.
FWR Assessment Rubric
Criteria |
Highly Proficient (4) |
Proficient (3) |
Competent (2) |
Developing (1) |
|---|---|---|---|---|
|
Critical Inquiry |
Thoroughly explores questions and issues with insightful, complex, or personal responses and deep reflection. |
Explores questions and issues with complex or personal responses and reflection. |
Partially explores questions and issues with some general responses and some reflection. |
Minimally explores questions and issues with general or unrelated responses and minimal reflection. |
|
Rhetorical Awareness |
Responds insightfully and with nuance to rhetorical context and purpose; consistently, coherently, and appropriately addresses audience, genre, and/or media. |
Responds thoughtfully to rhetorical context and purpose; addresses audience, genre, and/or media. |
Responds predictably to rhetorical context and purpose; partially addresses audience, genre, and/or media. |
Responds trivially to rhetorical context and purpose; barely addresses audience, genre, and/or media. |
|
Sources & Evidence |
Consistently and skillfully uses sources or evidence to pursue the writer’s primary goal or support the writer’s primary claim. |
Mostly uses sources or evidence to pursue the writer’s primary goal or support the writer’s primary claim. |
Uses some sources or evidence to pursue the writer’s primary goal or support the writer’s primary claim. |
Rarely use sources or evidence to pursue the writer’s primary goal or support the writer’s primary claim. |
|
Organization & Structure |
Consistently and skillfully executes structural or organizational features that are appropriate to the task and support the writer’s primary goal. |
Mostly executes structural or organizational features that are appropriate to the task and support the writer’s primary goal. |
Executes some structural or organizational features that are appropriate to the task and support the writer’s primary goal. |
Rarely executes structural or organizational features that are appropriate to the task and support the writer’s primary goal. |
|
Sentence-Level Style & Readability |
Consistently uses stylistically sophisticated sentences and reader comprehension is never impeded. |
Almost always uses stylistically clear sentences and reader comprehension is rarely and minimally impeded. |
Usually uses stylistically clear sentences and reader comprehension is occasionally impeded, though not critically. |
Uses unclear stylistic sentences and critically impedes reader comprehension. |
Appendix B
Rater Reliability
Rater agreement indicated strong reliability. Across the 300 artifacts and five rubric criteria, 92% of rating pairs produced identical scores or scores within one point, and 49% were identical. Rater agreement by rubric criterion is presented in Table B.1a. Only nine artifacts required a third read. The intraclass correlation coefficient (average measures) was 0.506, indicating that approximately half of the variance in scores reflected true differences in student performance rather than rater error (see Table B.1b).
Criterion |
% Identical Agreement |
% Agreement within 1 Level |
|---|---|---|
|
Critical Inquiry |
43 |
91% |
|
Rhetorical Awareness |
42 |
92% |
|
Sources and Evidence |
48 |
87% |
|
Organization and Structure |
49 |
94% |
|
Sentence-Level Style and Readability |
64 |
98% |
Reliability indicator |
Result |
|---|---|
|
Interclass correlation coefficient (average measures) Reliability indicator |
0.506 |
|
Rating pairs identical or within 1 point |
92% |
|
Rating pairs identical |
49% |
|
Artifacts requiring a third read (of 300) |
9 |