A group of European funding agencies and research councils is calling for organizations to adopt new measures to evaluate researchers and grant proposals, expanding beyond conventional metrics such as publication records or previous grant success.
The suggested measures include assessing a researcher’s output or grant proposal for its potential for economic and societal impact or commercialization, and its contribution to knowledge or policy.
In a report, the Brussels-based consortium Science Europe also calls for the elimination of bias towards characteristics such as ethnicity and disability in peer review and researcher evaluations, increasing transparency around all assessment processes and standardizing grant-application procedures across the European Union.
In its study, the group examines how member organizations make decisions about grant awards, hiring, promotions and peer-review assignments. The report also includes recommendations that members can pass on to universities in their countries. “Before you can make any change, you have to understand how the system works or doesn’t work,” says study chair Bonnie Wolff-Boenisch . “It’s like a baseline for deciding the next steps.”
James Morris, senior policy officer at Science Europe, says that although the consortium cannot enforce the recommendations, he thinks that research organizations will probably adopt them voluntarily, particularly because they can now see how their counterparts in other countries deal with similar issues. Many members have contributed to the report, he says, which “gives the recommendations some power”.
In 2019, Science Europe surveyed its members and some outside groups to identify the most difficult challenges they faced in assessing researchers and their scientific output. It also asked how each organization addressed problems such as gender bias and transparency in the peer-review process. It received responses from agencies in 27 countries, including 33 research-funding organizations, 4 research organizations and one that both funds and performs research.
The consortium’s report identifies bias in peer and promotion reviews as an area for improvement, noting that around two-thirds (68%) of funding organizations deliberately address diversity on their review panels, mostly in terms of gender and area of expertise. Fewer organizations were assessing representation of panel members from minority ethnic groups or those with disabilities, which Science Europe says needs improvement. Another major area is transparency and clarity in the selection of peer reviewers and in the criteria for grant review.
The report also encourages grant recipients to publish in open-access journals and to make software and research data available. The consortium further recommends that European countries standardize their grant- and job-application procedures, making it easier for researchers to find their footing if they move to a new country.
Wolff-Boenisch was pleasantly surprised to find that 81% of members that fund research were already doing qualitative assessments of grant applicants. Criteria they examine include science communication, mentorship, commitment to open science, and impact on policy and culture, rather than publication records alone.
Giorgio Chiarelli, research director of the National Institute for Nuclear Physics in Pisa, Italy, says that it was useful to have funding and research agencies at the same table when providing input for the report. Agencies that carry out research, for instance, might be more directly concerned with their scientists’ interests — preventing them from being overwhelmed by grant-review requests, for instance. Funding agencies, by contrast, could focus on ensuring diversity on review panels.
The report also allows countries to learn from one another, and for relatively small organizations to highlight their own experimental methods. For example, for the past two years, the Brussels-based funding agency Research Foundation Flanders has allowed researchers the chance to rebut negative peer-review comments. Secretary-general Hans Willems says this can help to correct misunderstandings and increase transparency about why grants are or are not funded. In turn, members can learn about methods being trialled by other groups, such as the Dutch Research Council, which is piloting a programme in which applicants work with other specialists in their field to write project proposals, and can then skip part of the review process.
The next phase of the project, Wolff–Boenisch says, will involve studying the research culture and the incentives that motivate scientists to contribute to innovation and other types of work.