As we approach the end of the semester, we anticipate final grades and course evaluations. What do these rituals signify? How do they function to crystallize the telos of our 15 weeks together? What are their purported functions – and do their methods of execution serve those functions? What institutional features are measured and ranked? What academic work gets evaluated, and how? What doesn’t get evaluated, and why? (Faculty service, “citizenship,” and advising are among those responsibilities that are rarely, if ever, assessed. What does such disregard tell us about institutional priorities?) What qualities are we assessing, for what purposes, and with what effects?
Today, we’ll look at four areas of evaluation: (1) student assessment: grades, crits, comprehensive exams, and the thesis or dissertation “defense” (which implies the existence of an offense!); (2) peer review; (3) faculty performance reviews (which include student course evaluations); and (4) ethics review of “human subjects” research. We’ll consider the ostensible purposes, protocols and methods, implicit values, inherent biases, and politics of each – and we’ll imagine how things could be done differently, to enact different values (perhaps abolitionist, decolonial, feminist values) and to serve different (perhaps less punitive, more generative) functions.
Academic Object Analysis Presentations: Clemente, Elizabeth, Meera + Grace
In-Class Activity: Scoring the Finale: We’ll work collaboratively – first in small groups, then as a whole class – to develop a process of peer review for our classmates’ final projects, and to propose a format for our final presentations: traditional solo+sequential presentations, a mini-conference, a world cafe, something else? We might even want to create an invitation, a “call for presenters” (recall our discussion of CFPs!), to the event, to give ourselves an opportunity to think about the type of interaction and hospitality we want to cultivate.
To Prepare for Today:
- You’ll recall that we were briefly introduced to anthropologist Marilyn Strathern in Week 2, when we developed “rubrics” for evaluating syllabi. Let’s read the Intro to her edited collection: “New Accountabilities: Anthropological Studies in Audit, Ethics and the Academy,” in Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy (Routledge, 2000): 1-18 – update 5/19 (sorry!): you’re welcome to focus on pp. 1-5!
On Grading:
- You’ll recall that Cathy Davidson spoke briefly about the history of grading in Week 5. Let’s dig a bit deeper: please read Jeffrey Schinske and Kimberly Tanner, “Teaching More by Grading Less (or Differently),” CBE Life Sciences Education 13:2 (2014): 159-66. See also Jesse Stommel’s “Ungrading: A Bibliography” blog post (March 3, 2020).
- For those of you with studio education backgrounds, let’s think about the pedagogical model of the “critique.” If you’re unfamiliar with crit culture, read An Le, “What Is an Architecture Crit?” Portico (2016) and check out my “On Design Crits” Arena channel. This Twitter thread (May 6, 2020), about my own crit confusion, generated some discussion! 😉
- Let’s also consider the thesis or dissertation “defense.” PhD students, you might want to skim William Clark’s “The Examination,” from his Academic Charisma and the Origins of the Research University (University of Chicago Press, 2006): 93-140 🙂 Clark on “charisma” echoes sociologist Pierre Bourdieu’s discussion, in Homo Academicus, about the subjective, affective qualities of academic judgment.
On Peer Review [note: all of the below are short blog posts and guides!]:
- Robert Harington, “The Rise of Peer Review: Melinda Baldwin on the History of Referring at Scientific Journals and Funding Bodies,” The Scholarly Kitchen (September 26, 2018).
- Aaron E. Carroll, “Peer Review: The Worst Way to Judge Research, Except for All the Others,” New York Times (November 5, 2018).
- COPE, “Ethical Guidelines for Peer Reviewers” [pdf] (n.d.).
- Seth Denbo, “Open Peer Review in the Humanities,” The Scholarly Kitchen (March 4, 2020).
- Civic Laboratory for Environmental Action Research, “Community Peer Review,” CLEAR (n.d.).
OPTIONAL: On Faculty Review:
- Skim my blog post about my own review for promotion to full professor (2018).
- Student evaluations are often the primary instrument for assessing faculty teaching. You’ll recall that Jonathan Zimmerman talked a bit about evaluations in Week 5. He also referenced Scott Gelber’s Grading the College: A History of Evaluating Teaching and Learning (Johns Hopkins University Press, 2020). Please read Jenela Brankovic’s review of the book, published in Higher Education Quarterly 75:4 (2021): you’re welcome to stop at “the evaluation of student learning” on page 2. And of course there’s voluminous research about the biases of student teaching evaluations.
- What if we develop new evaluative metrics that embody more personally and institutionally meaningful values? Please explore the HuMetricsHSS project.
OPTIONAL: On Human Subjects (Ethics Board) Review:
- Sheeva Sabati, “Upholding ‘Colonial Unknowing’ Through the IRB: Reframing Institutional Review Ethics,” Qualitative Inquiry 25:9-10 (2019): 1056-64.
- Madeline Whetung (Nishnaabeg) and Sarah Wakefield, “Colonial Conventions: Institutionalized Research Relationships and Decolonizing Research Ethics” in Linda Tuhiwai Smith, Eve Tuck, and K. Wayne Yang, eds., Indigenous and Decolonizing Studies in Education (Routledge, 2019): 146-58.
Supplemental Resources:
On Audits, Metrics, and Status:
- Kathleen Fitzpatrick, Generous Thinking: A Radical Approach to Saving the University (Johns Hopkins University Press, 2019).
- Bruno Latour and Steve Woolfar, “Cycles of Credit” in Laboratory Life: The Construction of Scientific Facts (Princeton University Press, 1986): 187-34.
- Deborah L. Rhode, In Pursuit of Knowledge: Scholars, Status, and Academic Culture (Stanford University Press, 2006).
- *Chris Shore and Susan Wright, “The Kafkaesque Pursuit of ‘World Class’: Audit Culture and the Reputational Arms Race in America,” in Sharon Rider, Michael Peters, Mats Hyvönen, and Tina Besley, eds., World Class Universities: A Contested Concept (Springer, 2020): 59-76.
- *Marilyn Strathern, “Introduction: New Accountabilities” in Strathern, ed., Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy (Routledge, 2000): 1-18.
- *Marilyn Strathern, “Bullet-Proofing: A Tale from the United Kingdom” in Annelise Riles, ed., Documents: Artifacts of Modern Knowledge (University of Michigan Press, 2006): 181-205.
- *Marilyn Strathern, Commons and Borderlands: Working Papers on Interdisciplinarity, Accountability and the Flow of Knowledge (Sean Kingston, 2003).
On Grading:
- Susan D. Blum, ed., Ungrading: Why Rating Students Undermines Learning (and What to Do Instead) (West Virginia University, 2020).
- Boston University, “Grading Equitably: Rethinking Grading, from Contract Grading to Ungrading, Specs Grading, and More,” Faculty Seminar (Spring 2021).
- Christine Lee, “What Is the History of Grading?” TurnItIn (October 14, 2020).
- “Pass / Fair: An American History of Testing,” Backstory Radio #0153 (April 8, 2016).
- Mary Lovett Smallwood, Examinations and Grading Systems in Early American Universities (Harvard University Press, 1935).
On Faculty Review:
- American Association of University Professors, “Tenure.”
- American Association of University Women, “Fast Facts: Women Working in Academia.”
- Pierre Bourdieu, “Postscript: The Categories of Professional Judgment” in Homo Academicus (Stanford University Press, 1988):
- Robert M. Diamond, Preparing for Promotion, Tenure, and Annual Review (Josey-Bass, 2004).
- Pamela Leggett-Robinson, Demystifying Promotion & Tenure: A Resource for Black Women (Independently Published, 2021).
- Juan Pablo Pardo-Guerra, The Quantified Scholar: How Research Evaluations Transformed the British Social Sciences (Columbia University Press, forthcoming 2022).
- Joanna Wolfe, “Let’s Stop Relying on Biased Teaching Evaluations,” Inside Higher Ed (January 21, 2022).
On Peer Review:
- Mieke Bal, “Let’s Abolish the Peer-Review System,” Media Theory (September 3, 2018).
- Melinda Baldwin, “Peer Review,” Encyclopedia of the History of Science (2019).
- Melinda Baldwin, “Scientific Autonomy, Public Accountability, and the Rise of ‘Peer Review’ in the Cold War United States,” Isis 109:3 (September 2018).
- Jessica McCrory Calarco, from “Peer Review Process” through “Moving Forward…” in A Field Guide to Grad School: Uncovering the Hidden Curriculum (Princeton University Press, 2020): 228-48.
- Kathleen Fitzpatrick, “Beyond Metrics: Community Authorization and Open Peer Review,” Debates in the Digital Humanities (University of Minnesota Press, 2012) or “Opening Up Peer Review,” European Network for Cinema and Media Studies (2021).
- “A History of Academic Peer Review,’ Frontiers: Science News (September 15, 2016).
- Gregory Jusdanis, “The Oppression of Peer Review,” Arcade (2010).
- +Wolfgang Kaltenbrunner, Kean Birch, and Maria Amuchastegui, “Editorial Work and the Peer Review Economy of STS Journals,” Science, Technology, & Human Values (2021) [thanks to Hannah Zeavin!].
- *+Michèle Lamont, How Professors Think: Inside the Curious World of Academic Judgment (Harvard University Press, 2010).
- Alison Muditt and Karin Wulf, “Peer Review in the Humanities and Social Sciences: If It Ain’t Broke, Don’t Fix It?” The Scholarly Kitchen (September 21, 2016).
- +Jack Schneider and Ethan Hutt, “Making the Grade: A History of the A-F Marking Scheme,” Journal of Curriculum Studies 46:2 (2014): 201-24.
- +Kate E. Snyder, “How to Become a More Effective Peer Reviewer,” Gifted Child Quarterly 62:2 (2018).
- Daniel Souleles, “An Interview With Reviewers 1, 2, and 3,” Savage Minds (May 17, 2016).
- Jasmine Wallace, “How to Be a Good Peer Reviewer,” Scholarly Kitchen (September 17, 2019).
- Gareth Watkins, “Peer Review in the Humanities and Social Sciences,” Wiley (January 10, 2017).
On IRB Review:
- Nathan Emmerich, “Reframing Research Ethics: Towards a Professional Ethics for the Social Sciences,” Sociological Research Online 21:4 (2016).
- Jacob Metcalf and Kate Crawford, “Where are Human Subjects in Big Data Research? The Emerging Ethics Divide,” Big Data & Society (2016): 1-14.
- Zachary M. Schrag, “The Case Against Ethics Review in the Social Sciences,” Research Ethics 7 (2011): 120-31.
- Angela Yost, “A Brief History and Insight on Human Subjects Research,” Society of Research Administrators International (August 26, 2016).