The Metric Tide | Steering group | Review scope | Media coverage
The review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the UK’s Research Excellence Framework (REF). It has explored the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture.
To give structure and focus to our efforts, clear terms of reference were established at the outset. The review was asked to examine:
- The relative merits of different metrics in assessing the academic qualities and diverse impacts of research;
- The advantages and disadvantages of using metrics, compared with peer review, in creating an environment that enables and encourages excellent research and diverse impact, including fostering inter- and multidisciplinary research;
- How metrics-based research assessment fits within the missions of universities and research institutes, and the value that they place on published research outputs in relation to the portfolio of other activities undertaken by their staff, including training and education;
- The appropriate balance between peer review and metrics in research assessment, and the consequences of shifting that balance for administrative burden and research cultures across different disciplines;
- What is not, or cannot, be measured by quantitative metrics;
- The differential impacts of metrics-based assessment on individual researchers, including the implications for early-career researchers, equality and diversity;
- Ethical considerations, and guidance on how to reduce the unintended effects and inappropriate use of metrics and university league-tables, including the impact of metrics-based assessment on research culture;
- The extent to which metrics could be used in novel ways by higher education institutions (HEIs) and research funders to support the assessment and management of research;
- The potential contribution of metrics to other aspects of research assessment, such as the matching of reviewers to proposals, or research portfolio analysis;
- The use of metrics in broader aspects of government science, innovation and industrial policy.
Our final report The Metric Tide starts by tracing the history of metrics in research management, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises.
The review has drawn on a diverse evidence base to develop its findings and conclusions. These include: a formal call for evidence; a comprehensive review of the literature (Supplementary Report I); and extensive consultation with stakeholders at focus groups, workshops, and via traditional and new media. The review has also drawn on HEFCE’s recent evaluations of REF2014, and commissioned its own detailed analysis of the correlation between REF2014 scores and a basket of metrics (Supplementary Report II).