On February 8-9, I was very happy to meet with colleagues at the University of Manitoba, during a visit organized by the Institute for Humanities. In presentations and discussion sessions, we covered topics relating to the measurement and appraisal of humanities research. A summary of some of the themes raised in those discussions follows.
When asked to provide evidence or descriptions of research impact, humanities researchers typically face two related difficulties. The first is the prevalence and influence of research metrics that do not capture humanities research accurately; the second is the difficulty of proposing characterizations of research impact that do capture humanities research accurately.
We discussed ways in which both difficulties can be addressed through an open, interdisciplinary conversation about research cultures and practices. An extremely valuable tool in preparing for and conducting such a conversation, I suggested, is the Federation for the Humanities and Social Sciences’ 2014 working paper “Impacts of Humanities and Social Science Research,” the first product produced through the Federation’s Impacts Project.
Some research reporting and measurement schemes in the university sector are based on assumptions rendering them ill-suited to capture the power and significance of research in the humanities (and in broad swathes of the social sciences as well, though our focus in these meetings was on the humanities). Problematic assumptions can include metrics or evaluation processes that are unduly sensitive to:
- rapid onset impacts, those that emerge in the short or medium term frame after publication/creation;
- impacts linked directly to commercialization; or
- publication and citation practices characteristic of relatively few disciplines in the academy – especially those favouring journal publication over books and conference proceedings, those in which it is common to include large numbers of citations in published articles, and those with highly liberal co-authorship standards.
Humanists are justifiably uneasy about the use of inaccurate metrics. On the other hand, it’s reasonable to want some ways of characterizing and observing research impacts in the humanities in order to understand the effects of humanities research, to be able to defend its value to the polities that materially support it, and to make strategic institutional decisions about research goals. Jointly these considerations speak to the value of a campus-wide conversation aiming at a shared understanding of scholarly practices and research outcomes. If there is a gap between the reality of humanities research and the accuracy of some research metrics, it is important to address both sides of the matter: pointing out weaknesses in the proposed metrics, but also making a positive case about humanities research that will enable better characterizations of it.
The positive case for characterizing humanities research has traditionally been hard to make, simply because it is difficult to spell out the extremely broad sweep of impact forms that humanities scholarship generates. This is where the FHSS’s Impact Project becomes especially valuable, proving to be one of the most useful resources for humanities professors and organizations in Canada (or internationally, for that matter). My presentation brought out some of the respects in which the Federation’s Impacts resources can play a role:
- building understanding and sharing knowledge within the humanities and social science community of what counts as an impact, and how it may be demonstrated;
- engaging the research community in a conversation about how best to discuss HSS impacts;
- developing tools to help SSH researchers demonstrate the impact of their work; and
- advocating for institutional and systemic policies and practices that characterize the SSH research community accurately.
Of particular importance, I suggested, are the contents of the “bins” of research impact indicators provided in the 2014 impact document. These are collections of impact indicators, grouped by types, which provide researchers, students, staff, institutional leaders, stakeholders, and the public with illustrations of the kinds of impact that humanities research can have. (Readers should also be on the lookout for a forthcoming second paper from the Federation, which considers approaches to assessing impacts in more depth.)