Research and innovation to provide evidence that support reforms of research assessment

Expected Outcome:

The successful proposal will deliver on the following impacts: “More responsible assessment of researchers, research projects and research organisations based on a broader range of research outputs, practices and activities than publications and publishing” and “Improved quality, performance and impact of research, researchers and institutions due to more responsible research assessment systems”.

Project results are expected to contribute to all of the following expected outcomes:

• Increased scientific knowledge on assessment tools and methods, their potential biases and limitations, and corresponding solutions;
• Increased scientific knowledge on methods to reform research assessment;
• Evidence supporting the implementation of the commitments of the Agreement on Reforming Research Assessment available to policy makers and research organisations involved in research assessment.

Scope:

A global movement is underway to significantly enhance the evaluation process for researchers, research projects and research institutions. The San Francisco Declaration on Research Assessment (DORA), the Leiden Manifesto, the Latin American Forum on Research Assessment (FOLEC-CLACSO), and the Agreement on Reforming Research Assessment^[1] and the Coalition for Advancing Research Assessment (CoARA) under Action 3 of the European Research Area Policy Agenda 2022-2024^[2] are the main initiatives that strive to shift away from inappropriate reliance on metrics such as the Journal Impact Factor. Instead, they acknowledge the diverse range of research practices, activities and outputs that go beyond publications. They aim to reward practices, activities and outputs that contribute to a higher quality and impact of research, while also recognising and mitigating systemic biases, particularly gender biases, within traditional assessment metrics.

CoARA Working Groups carry out in-depth work to enable signatories of the Agreement on Reforming Research Assessment to exchange experiences, learn from each other and identify good practices, thus supporting their efforts to reform research assessment.

The identification of good practices and the implementation of changes to research assessment require solid scientific evidence. This concerns evidence about assessment tools and methods, and exploring issues across disciplines and types of research organisations, such as: best practices in qualitative assessment; making peer review practices less bias-prone and more transparent; lessons learned from pilots with narrative CVs; inappropriate uses of metrics and alternatives for different levels of assessment (of research, of researchers and of research institutions); relevant indicators (that are open) for developing new assessment frameworks; infrastructures and tools needed to base research assessment on FAIR research digital objects (e.g., publications, data, software); ensuring that assessment methods promote diversity and inclusion across genders and underrepresented groups. This also concerns evidence about reform methods and about institutional changes and culture, investigating issues, such as: ensuring readiness and engagement of the scientific community and research organisations to changes; including different disciplinary perspectives; piloting actions that can lead to new assessment practices.

The action aims to provide the evidence needed for the practical implementation of research assessment reforms. It will gather and evaluate existing evidence and carry out additional research activities to support the implementation of the commitments of the Agreement on Reforming Research Assessment, in line with the 10 principles for reforms. The action consists of three parts, all of which must be addressed:

• Review existing literature, data and evidence on research assessment practices and evaluate their validity and robustness for implementing the Agreement on Reforming Research Assessment.
• Identify gaps in knowledge and prioritise areas that require further research activities to support reforms of research assessment in line with the Agreement on Reforming Research Assessment.
• Implement research activities in the prioritised areas.

The proposals should ensure synergies and complementarities, develop close cooperation, and share knowledge and evidence with CoARA and its Working Groups, the CoARA Boost project, DORA, and other relevant initiatives. The proposals should provide open access to all relevant results as early as possible, with specific efforts to ensure that these results are accessible and relevant to diverse and underrepresented groups.

This action may require the effective contribution of SSH disciplines and the involvement of SSH experts, institutions as well as the inclusion of relevant SSH expertise, in order to produce meaningful and significant effects enhancing the societal impact of the related research activities^[3].

The duration of this action should not exceed 3 years.

[1] https://coara.eu/app/uploads/2022/09/2022_07_19_rra_agreement_final.pdf.

[2] https://commission.europa.eu/system/files/2021-11/ec_rtd_era-policy-agenda-2021.pdf.

[3] https://research-and-innovation.ec.europa.eu/research-area/social-sciences-and-humanities/ssh-integration_en.