Metrics for Research Impact in the Social Science

The Latest Thinking About Metrics for Research Impact in the Social Science
Sage Publishing

Introduction

Traditionally, impact within academia has been synonymous with citation counts, but how do we move beyond this? Although it will always be imperative to have basic, curiosity-driven research in social science, much research must have a significant and positive real-world impact that extends beyond citations in other research. The impact of that more applied research can vary widely—influencing policy or practice, engaging the public, delivering economic benefits, or supporting advances in the physical sciences—but how to show this impact in a widely accepted and replicable manner has escaped our disciplines. There is little agreement, or even understanding, about how to measure the societal impacts of such research (Tregoning, 2018). Social science research is particularly difficult to evaluate given the inherent challenges in understanding and measuring what are social (and often subjective) phenomena, and research outputs can be overlooked as obvious or commonsensical (Reale et al., 2018).

However, governments and other funders increasingly call on researchers to demonstrate the societal impact of their work, generally without specifying any commonly agreed-on set of best practices or definitions. In the United Kingdom, impact case studies for the Research Excellence Framework (REF) 2021 will be used to award 25 percent (up from 20 percent in REF 2014) of about £2 billion per year of public research funding. These case studies are qualitatively peer-reviewed by panels and officially don’t use citations, though they may still have an influence. Although it’s not tied to funding yet, the UK’s emerging Knowledge Exchange Framework (KEF) can also evaluate research impact, alongside the wider effects that universities can have.

In the United States, the Pentagon’s Defense Advanced Research Projects Agency (DARPA) has recently tasked social scientists with creating an artificial intelligence system, known as SCORE, to quantitatively measure the reliability of social science research and “thereby increase the effective use of

literature and research to address important human domain challenges” (Russell, n.d., para. 1).

And the National Science Foundation, the primary funder of basic academic research in social science, requires “broader impacts” to be explicit in the research it funds but offers no metrics for what broader impacts are. Historically, perception of impact in the social sciences has largely had a qualitative cast, for success stories in human behavior and societal good, or in policy drafted and laws enacted, were seen as the definition of impact. Being able to qualitatively describe how investments in the social sciences have paid dividends later—the raison d’être of the American Golden Goose Awards, for example—has been key to “selling” social science to a broader audience. But a more metrics-driven approach, including widely accepted and policy-meaningful measures, will still be required when deploying impact outside of academe.

It’s worth making explicit that neither the REF nor the U.S.-based examples, nor most funders, use literature-based measures like journal citations as a primary measurement. Yet citations and journal impact factors remain the traditional lingua franca of impact for academics, and a majority of social scientists (and in fact most academics) are leery of new methods of showing impact (Bakker et al., 2019). (Any new measures also must address those concerned about what they see as the creeping corporatization of higher education or that science still waiting to be used “by definition has no space for performance metrics” [Prewitt, 2016, para. 9].)

Download>>