Responsible reform of research assessment could deliver more value for South Korea

Author: Soon Kim and Ayako Miyazaki

Springer Nature held its inaugural Korea Research Advisory Forum (KRAF) on August 20, 2024, bringing together a diverse group of influential figures from South Korean research organisations. 

This forum was established to discuss key challenges and priorities for research in South Korea and to explore how we can work together to address these issues. It follows similar research advisory bodies setup with key stakeholders in the USA, Japan, Africa and Europe. The inaugural KRAF roundtable included representatives from the Korean ARPA-H Initiative, Seoul National University, POSTECH, Korea University, Yonsei University, IBS, and GISTeR and focussed on the future of research assessment in South Korea. Convened by Dr. Magdalena Skipper, Editor in Chief, Nature and Dr. Nick Campbell, VP Academic Affairs, Springer Nature, KRAF was chaired in person by the latter with Steve Riddell, Director of Consulting, Nature Research Intelligence, and Ikuko Oba, Commercial Director moderating discussions.

Advancing research assessment practices for open science and social impact

South Korea has shown strong commitment to science, technology and innovation, with a relatively high number of researchers per capita and significant spending on R&D compared to other countries^[1]. However, as members of the forum discussed, there is a disconnect between its spending and its research output, as reflected in analyses included in the Nature Index 2024 South Korea supplement, published in the week that KRAF was held. Reforms and innovations in research assessment practices are a key  priority for South Korea to help address this issue. The approaches being deployed by the Korean ARPA-H funding initiatives are an important example of innovation in this area: high-risk, high-reward research is the explicit goal with a more considered, diverse approach to assessing impact. Changmo Sung, Director of the Korean ARPA-H’s Mission PM Center, discussed the new evaluation methodologies ARPA-H will use, emphasising the importance of qualitative evaluation methods that prioritise social impact, innovation, and interdisciplinary research over traditional publication counts and citation metrics. The forum members also emphasised the need to incorporate stakeholder feedback and develop innovative methodologies that align with the principles of open science and social impact. 

Ensuring transparency in performance evaluation

Other key challenges faced by the research community in South Korea include ensuring transparency in performance evaluation, where striking a balance between qualitative and quantitative measures remains complex, particularly for high-risk, high-reward research programs, a point raised by Hyun Woo (Henry) Park, Associate Professor, Yonsei University. Chulhong Kim, Professor, Pohang University of Science and Technology suggests that researchers and institutions should consider metrics relevant to commercialization (patents, licences etc), in addition to established measures of academic performance (publications and citations). Kim also emphasises the need to evaluate research leadership and the long-term impact of innovations, which are often neglected in current evaluation systems. Ilhwan Lee, Director of the Center for National R&D Evaluation, Korea Institute of S&T Evaluation and Planning (KISTEP), shared insights into the need for a comprehensive approach to evaluating the performance of government-funded research institutes in South Korea. The diverse missions and goals of over 1,500 R & D programs, make it challenging to apply a one-size-fits-all set of performance indicators.

Researchers often experience significant stress and workload associated with preparing for evaluations, which can adversely impact their overall productivity and well-being. Ensuring transparency and accessibility in the evaluation processes is essential, as it helps maintain trust and engagement within the research community. Michelle Randall, Operations Director at the Institute for Basic Science (IBS) Center for Quantum Nanoscience, shared insights regarding the high stakes of the IBS three-year review process and the intense preparations required for effective evaluations. She also noted the importance of addressing stress and challenges faced by research centres during these evaluations.

Moving away from traditional quantitative metrics

The forum agreed that minimising bureaucratic burdens and enhancing transparency in research evaluations is highly desirable. Providing incentives for data sharing and collaboration is crucial, alongside the removal of disincentives for team science. Moreover, there is a strong consensus emerging around the need to move away from an over-reliance on quantitative metrics in favour of more holistic and context-sensitive evaluation methods. Ensuring effective implementation of qualitative assessment approaches is vital for fairness, with social science research aiding this process. Ultimately, fostering collaboration and team science is essential, suggesting that research assessments should consider team contributions rather than solely individual performance.

Yu Jin Jung, Head of the Division of Research Services at the Institute for Basic Science (IBS), emphasised the importance of fostering autonomy among research teams and using qualitative assessments to evaluate their activities comprehensively. The IBS evaluation system focuses on scientific excellence and originality, assessing research centres based on peer review rather than solely on quantitative metrics.

Diverse approaches to evaluating government-funded research organisations were shared, highlighting the need for evaluations to align with strategic national and institutional goals. Developing effective tools to assess the social impact of research can align scientific endeavours with the needs of the broader community. 

Supporting diversity and inclusion in research, and capturing the true value of research in society

The importance of diversity and inclusion in research practices was also emphasised by Heisook Lee, President, GISTeR, addressing key areas for improvement, including the integration of gender and gender-based analysis as a vital step towards fostering more inclusive research environments. Global initiatives that support diversity were also raised, advocating for these considerations to be embedded into research policies. The discussions reinforced the need for a diverse approach across various dimensions to enhance the effectiveness and relevance of research in today’s complex landscape.

As South Korea continues to navigate its research landscape, it faces significant challenges, including gender disparity in the research workforce. The insights shared during the KRAF roundtable were both timely and critical, reflecting some of the discussions and data published in this year’s Nature Index 2024 South Korea supplement. Both the report and the roundtable highlighted the importance of addressing gender representation in research and fostering a more diverse research ecosystem. The result, the forum discussed, would be a strengthened research culture, and broaden perspectives, leading to enhancing the country's standing as a leader in global science. Moreover, the forum members shared the understanding that building diverse research teams through strengthened global partnerships has the potential to yield more innovative and robust results. 

The roundtables reached a consensus on the necessity for diverse and contextualised evaluation approaches. The forum members expressed a strong desire to transition towards more qualitative and impactful research evaluations that better capture the true value of research in society.

The roundtables, the first in a series of engagements with key stakeholders in South Korea, gave a diverse set of insights into key issues and opportunities currently facing the research landscape within the region. Most critically the discussions highlighted the importance of ongoing innovation in research assessment practices and the need for responsible reform to support a more impactful research landscape. As we look ahead, Springer Nature is excited to continue these vital discussions and collaborate with the research community to create a more inclusive and effective research evaluation environment in South Korea.

[1] World Bank Group - Research & development expenditure (% of GDP)  

About the Springer Nature Korean Research Advisory Forum (KRAF)

Springer Nature is committed to opening the doors to discovery, and ensure that we provide the best possible services to the research community, through understanding their needs. As part of that ongoing commitment, following the launch of the US Research Advisory Council in 2021, in 2024, Springer Nature has organized the Korean Research Advisory Forum, comprised of relevant members across various sectors in the academic community. The forum members are as follows: 

• Changmo Sung, Director, Mission PM Center, Korea ARPA-H
• Chulhong Kim, Professor, Pohang University of Science and Technology
• Eunmi Chae, Associate Professor, Korea University
• Heisook Lee, President, GISTeR
• Hyun Woo (Henry) Park, Associate Professor, Yonsei University
• Ilhwan Lee, Director, Center for National R&D Evaluation, Korea Institute of S&T Evaluation and Planning (KISTEP) 
• Je Kyung Seong, Professor, Seoul National University
• Jooyoung Park, Associate Professor, Seoul National University
• Michelle Randall, Operations Director, IBS Center for Quantum Nanoscience
• Yong Sik Ok, Professor, Korea University
• Yu Jin Jung, Head of the Division of Research Services, Institute for Basic Science
• Sang Yup Lee, Professor, Korea Advanced Institute of Science and Technology