The state of research assessment: Insights from a survey of 6,600+ researchers

Author: Una Farr

The landscape of research assessment is increasingly defined by an awareness of the limitations of traditional metrics, such as publication counts and journal impact factors. These measures have historically dominated evaluation practices, but there is a growing push for more holistic approaches that consider a wider range of contributions from researchers, that are more truly reflective of the work that they do and the impact that they make.

A new Springer Nature white paper, The State of Research Assessment: Researcher Perspectives on Evaluation Practices, supports the ongoing discussion on research assessment reform by capturing the views of over 6,600 researchers across a diverse range of regions, disciplines and career stages. This blog summarises the key findings from the white paper, including how researchers are currently assessed, their perceptions of existing practices, and their hopes for the future. 

How are researchers currently assessed?

Initiatives for research assessment reform (including The Leiden Manifesto, The Declaration on Research Assessment, and the Coalition for Advancing Research Assessment) have universally challenged an overreliance on journal and publication metrics. 

Despite the frameworks introduced by these initiatives, our study found that researchers still identify metrics as the dominant factor in how they are assessed – with 55% stating that their work is evaluated entirely or mostly using metrics. Of these, publication metrics (such as citations or article-level impact) were by far the most common, surpassing all other options by a significant margin. 

At present, what is the mix of qualitative and quantitative methods for how your work is evaluated? (n=3,827)

We also asked researchers about how their contributions outside of research outputs are included within assessment processes. Overall, we found that researchers’ positive contributions to society (the effect of their research on society or the economy), and their wider contributions to the research community (e.g. teaching, carrying out peer review) are commonly used within evaluations. But what this means in practice can vary, and many researchers feel that these activities are only considered in certain situations, or are not weighted as highly as research outputs and publication metrics.   

“These activities are usually regarded positively but often not included in evaluation of my research for research advancement. Then almost only metrics regarding publications count. Contrastingly, in grant applications it is often possible to elaborate on less “numeric” measurements of the value of my research.” 

[Denmark, established career stage, Clinical Medicine & Healthcare] 

How do researchers feel about assessment practices?

Perhaps surprisingly based on the above, we also learned that most researchers feel positively about the ways they and their research are assessed. Researchers expressed favourable views on every type of assessment: 75% net-positive for new job/career advancement evaluations, 71% net-positive for external institutional evaluations, and 64% net-positive for grant applications. 

Whether or not it was successful, did you feel positively about your experiences of the following types of evaluation? (n=5,291)

However, when asked to consider assessment practices more broadly, many researchers expressed notable concerns, including: a lack of trust in the process (opaque systems, bias and subjectivity); an overreliance on publication metrics; and the difficulty in assessing the societal impact of research. Some respondents also expressed frustration that research assessment often judges things that they cannot control or influence – relying on the downstream impacts of their work rather than the effort they put in. 

“The evaluation criteria are based on quantitative indicators or on indicators that are not depending on my effort and that I cannot improve by simply working more and better.” 

[Romania, established career stage, Physical Sciences & Engineering] 

What do researchers envision for the future?

Finally, we asked respondents to consider how their work is currently assessed, versus their preferences for the future. In line with our findings above, researchers feel that their research outputs currently make up the majority (60%) of how they are evaluated. However, in an ideal scenario we see this drop to 46%, with researchers desiring a greater weighting on their contributions to society (“the public good”), and their wider contributions to research and research culture.

How would you say the importance of each of these areas is weighted in terms of how you are currently assessed, and in an ideal evaluation process what would you suggest should be the weighting? (n=6,616)

We observed a similar shift when we asked researchers about their preferences for the usage of metrics during assessment processes. While 55% of respondents identified metrics as the dominant factor in their current evaluations, in an ideal scenario 52% of researchers favour an equal balance between quantitative and qualitative methods (compared to 27% preferring metrics). 

“A comprehensive and balanced evaluation process is crucial for fostering high-quality research and recognising the diverse contributions of researchers.” 

[Algeria, mid-career stage, Physical Sciences & Engineering]

Emerging questions for research assessment reform

Though we see a clear desire from researchers to move towards more holistic forms of assessment, our study also uncovered some tensions between the best practices provided by research assessment reform initiatives, what is happening in reality, and researchers’ perceptions and preferences for the future. For example:

• Around half of researchers did not know whether their institution was signed up to any research assessment reform initiatives – and of those who did, 40% perceived little to no change within institutional practices. 
• Researchers had mixed views on the ease of preparing assessment requirements, as well as access to institutional support - around 45% positive compared with around 35% negative. 
• A high proportion of researchers acknowledged the value of metrics, with only 18% believing that evaluations should be entirely or mostly qualitative. This goes against recommendations made by research assessment reform initiatives, and also does not consider the fact that many activities are difficult or impossible to measure quantitatively.

To review the full survey findings, with further insights and conclusions for the research community, read the full white paper, The state of research assessment: Researcher perspectives on evaluation practices.  

Related content

• Supporting DEI in research: What researchers need from institutions
• Navigating the evolving role of research managers: Insights from INORMS Council members
• Global research collaboration strategies that support researchers
• The impact of good science communication, and how organisations can support
• Supporting open science: Exploring barriers and solutions to data sharing
• Navigating research integrity in the age of AI
• Three ways Generative AI can give Research Managers “superpowers”
• Virtual versus in-person seminars: Which training support is most beneficial?

Don't miss the latest news & blogs, subscribe to The Link Alerts!