Week 12, April 19: On Evaluation

Brian Cantoni, Scantron (just looking at it makes me nervous!), via Flickr, CC BY 2.0

As we approach the end of the semester, we anticipate final grades and course evaluations. What do these rituals signify? How do they function to crystallize the telos of our 15 weeks together? What are their purported functions – and do their methods of execution serve those functions? What institutional features are measured and ranked? What academic work gets evaluated, and how? What doesn’t get evaluated, and why? (Faculty service, “citizenship,” and advising are among those responsibilities that are rarely, if ever, assessed. What does such disregard tell us about institutional priorities?) What qualities are we assessing, for what purposes, and with what effects? 

Today, we’ll look at four areas of evaluation: (1) student assessment: grades, crits, comprehensive exams, and the thesis or dissertation “defense” (which implies the existence of an offense!); (2) peer review; (3) faculty performance reviews (which include student course evaluations); and (4) ethics review of “human subjects” research. We’ll consider the ostensible purposes, protocols and methods, implicit values, inherent biases, and politics of each – and we’ll imagine how things could be done differently, to enact different values (perhaps abolitionist, decolonial, feminist values) and to serve different (perhaps less punitive, more generative) functions. 

Academic Object Analysis Presentations: Clemente, Elizabeth, Meera + Grace

In-Class Activity: Scoring the Finale: We’ll work collaboratively – first in small groups, then as a whole class – to develop a process of peer review for our classmates’ final projects, and to propose a format for our final presentations: traditional solo+sequential presentations, a mini-conference, a world cafe, something else? We might even want to create an invitation, a “call for presenters” (recall our discussion of CFPs!), to the event, to give ourselves an opportunity to think about the type of interaction and hospitality we want to cultivate.

To Prepare for Today: 

  • You’ll recall that we were briefly introduced to anthropologist Marilyn Strathern in Week 2, when we developed “rubrics” for evaluating syllabi. Let’s read the Intro to her edited collection: “New Accountabilities: Anthropological Studies in Audit, Ethics and the Academy,” in Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy (Routledge, 2000): 1-18 – update 5/19 (sorry!): you’re welcome to focus on pp. 1-5!

On Grading:

  • You’ll recall that Cathy Davidson spoke briefly about the history of grading in Week 5. Let’s dig a bit deeper: please read Jeffrey Schinske and Kimberly Tanner, “Teaching More by Grading Less (or Differently),” CBE Life Sciences Education 13:2 (2014): 159-66. See also Jesse Stommel’s “Ungrading: A Bibliography” blog post (March 3, 2020). 
  • For those of you with studio education backgrounds, let’s think about the pedagogical model of the “critique.” If you’re unfamiliar with crit culture, read An Le, “What Is an Architecture Crit?Portico (2016) and check out my “On Design Crits” Arena channelThis Twitter thread (May 6, 2020), about my own crit confusion, generated some discussion! 😉
  • Let’s also consider the thesis or dissertation “defense.” PhD students, you might want to skim William Clark’s “The Examination,” from his Academic Charisma and the Origins of the Research University  (University of Chicago Press, 2006): 93-140 🙂 Clark on “charisma” echoes sociologist Pierre Bourdieu’s discussion, in Homo Academicus, about the subjective, affective qualities of academic judgment. 

On Peer Review [note: all of the below are short blog posts and guides!]

OPTIONAL: On Faculty Review:

  • Skim my blog post about my own review for promotion to full professor (2018). 
  • Student evaluations are often the primary instrument for assessing faculty teaching. You’ll recall that Jonathan Zimmerman talked a bit about evaluations in Week 5. He also referenced Scott Gelber’s Grading the College: A History of Evaluating Teaching and Learning (Johns Hopkins University Press, 2020).  Please read Jenela Brankovic’s review of the book, published in Higher Education Quarterly 75:4 (2021): you’re welcome to stop at “the evaluation of student learning” on page 2. And of course there’s voluminous research about the biases of student teaching evaluations. 
  • What if we develop new evaluative metrics that embody more personally and institutionally meaningful values? Please explore the HuMetricsHSS project

    OPTIONAL: On Human Subjects (Ethics Board) Review: 

Sophie Taeuber-Arp @ MoMA; photo by me

Supplemental Resources: 

On Audits, Metrics, and Status: 

  • Kathleen Fitzpatrick, Generous Thinking: A Radical Approach to Saving the University (Johns Hopkins University Press, 2019). 
  • Bruno Latour and Steve Woolfar, “Cycles of Credit” in Laboratory Life: The Construction of Scientific Facts (Princeton University Press, 1986): 187-34. 
  • Deborah L. Rhode, In Pursuit of Knowledge: Scholars, Status, and Academic Culture (Stanford University Press, 2006).
  • *Chris Shore and Susan Wright, “The Kafkaesque Pursuit of ‘World Class’: Audit Culture and the Reputational Arms Race in America,” in Sharon Rider, Michael Peters, Mats Hyvönen, and Tina Besley, eds., World Class Universities: A Contested Concept (Springer, 2020): 59-76. 
  • *Marilyn Strathern, “Introduction: New Accountabilities” in Strathern, ed., Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy (Routledge, 2000): 1-18. 
  • *Marilyn Strathern, “Bullet-Proofing: A Tale from the United Kingdom” in Annelise Riles, ed., Documents: Artifacts of Modern Knowledge (University of Michigan Press, 2006):  181-205.
  • *Marilyn Strathern, Commons and Borderlands: Working Papers on Interdisciplinarity, Accountability and the Flow of Knowledge (Sean Kingston, 2003). 

On Grading: 

On Faculty Review: 

  • American Association of University Professors, “Tenure.” 
  • American Association of University Women, “Fast Facts: Women Working in Academia.”
  • Pierre Bourdieu, “Postscript: The Categories of Professional Judgment” in Homo Academicus (Stanford University Press, 1988): 
  • Robert M. Diamond, Preparing for Promotion, Tenure, and Annual Review (Josey-Bass, 2004). 
  • Pamela Leggett-Robinson, Demystifying Promotion & Tenure: A Resource for Black Women (Independently Published, 2021). 
  • Juan Pablo Pardo-Guerra, The Quantified Scholar: How Research Evaluations Transformed the British Social Sciences (Columbia University Press, forthcoming 2022).
  • Joanna Wolfe, “Let’s Stop Relying on Biased Teaching Evaluations,Inside Higher Ed (January 21, 2022).

On Peer Review: 

On IRB Review: