Promotion and tenure policies and practices often value researchers' scholarly impact. But what is "impact," and how is it measured?
Below you'll find resources explaining the kinds of metrics used to assess research and researchers, as well as studies and reports exploring what is currently evaluated and rewarded in hiring, review, promotion, & tenure, and how these measures can be problematic.
Academic criteria for promotion and tenure in biomedical sciences faculties: cross sectional analysis of international sample of universities (BMJ, June 25, 2020)
“The evaluation of scientists emphasises traditional criteria as opposed to non-traditional criteria. This may reinforce research practices that are known to be problematic while insufficiently supporting the conduct of better quality research and open science. Institutions should consider incentivising non-traditional criteria.”
Advancing science or advancing careers? Researchers’ opinions on success indicators (PLOS ONE, February 11, 2021)
“We found that indicators related to openness, transparency, quality, and innovation were perceived as highly important in advancing science, but as relatively overlooked in career advancement. Conversely, indicators which denoted of prestige and competition were generally rated as important to career advancement, but irrelevant or even detrimental in advancing science.”
Assessing scientists for hiring, promotion, and tenure (PLOS Biology, March 29, 2018)
“Using more appropriate incentives and rewards may help improve clinical and life sciences and their impact at all levels, including their societal value.”
Games academics play and their consequences: how authorship, h-index and journal impact factors are shaping the future of academia (Proceedings of the Royal Society B, December 4, 2019)
"A large proportion of the responsibility must fall on senior faculty to be role models expressing the highest ethical standards, being superior mentors of only a few students, making the effort to make reviews based on quality and establishing institutional evaluation criteria.... Institutions must also play a role by training mentors to create a healthy and ethically robust culture."
Open and Equitable Scholarly Communication: Creating a More Inclusive Future (ACRL, 2019)
“While content discussions among scholarly communications practitioners often are framed around questions of openness, the most frequent barrier to a more open, inclusive, and equitable system reported by study participants was the current promotion, retention, and tenure (PRT) system and its focus on productivity and quality, defined almost exclusively as multiple traditional publications in venues perceived as high prestige.” (from “Content," p.16ff)
Responsible Science Assessment: downplaying indexes, boosting quality (Annals of the Brazilian Academy of Sciences, February 22, 2021)
“We believe researchers should rethink who benefits financially from the journals they publish, as well as who are the people shepherding this publication venue. Our proposal, the Initiative for Responsible Scientific Assessment, has the main goal to… promote higher quality research assessment, focused on better Science.”
"Saying ‘No’ to Rankings and Metrics" (Chapter 6 in Socially Responsible Higher Education, Brill, 2021)
"[The metrics and rankings] system reproduces researchers that could easily neglect issues unique to their society and that are not able to question structural inequalities in their knowledge production. In such a situation, open access to research will only act as band-aid to an infectious disease – it is aesthetically appealing, but does not address the deep, problematic orientation of researchers and their institutions."
Scholarly Communications Lab, Simon Fraser University. Researchers at the ScholComm Lab have published several papers as part of their Review, Promotion, and Tenure Project, in which they collected and analyzed more than 850 RPT documents from North American universities & assessed the language included in the guidelines. Some recent papers:
How faculty define quality, prestige, and impact of academic journals (PLOS ONE, October 28, 2021)
"This study’s results highlight the subjectivity of common research terms and the importance of implementing evaluation regimes that do not rely on ill-defined concepts and may be context specific."
The value of data and other nontraditional scholarly outputs in academic review, promotion, and tenure (Forthcoming in Open Handbook of Linguistic Data Management, MIT Press)
“We conclude that many current structures for faculty assessment do not explicitly recognize the increasing complexity and demands of faculty work.”
May 13, 2022