Research Impact vs Citations: Who Pays the Price?

Critics of citation-based academic evaluations advocate social impact as an alternative, an approach the UGC now endorses. But in this race for impact, those whom scientific research is meant to serve could become its casualties.

Each autumn, a database compiled by John Ioannidis of Stanford University and the publisher Elsevier identifies the most cited two per cent of the world's researchers, and Indian universities respond with press releases enumerating the faculty members who have made the cut. The release is followed, with equal regularity, by a debate in which defenders of the list read it as evidence of scholarly standing, and critics argue that citations reward a narrow and easily manipulated kind of success and that academics ought to be assessed on the social impact of their research instead, on public engagement and outreach.

The second view now carries official weight in India, with the draft regulations circulated by the University Grants Commission (UGC) in January 2025 proposing to discontinue the Academic Performance Indicator, which rested largely on journal publications, and assess teachers on nine categories of "notable contributions" that include community engagement and service, entrepreneurship, and contributions to society. A competition among academics is, however, safest when it is based on citations alone, because academics are adept at meeting whatever criterion they are judged by.

A clever population that is ranked on the basis of a number learns to produce the number. Citations are inflated through self-citation, cartels of authors and journals that cite one another, editors who press authors to cite the editor's own journal, and the division of a single result into several thin papers. The Stanford database itself reports its indicators both with and without self-citations and adjusts for co-authorship, an acknowledgment that the measure has been under pressure ever since it acquired consequence. The American social scientist Donald Campbell anticipated the pattern in the 1970s when he observed that the more a quantitative indicator is used to make social decisions, the more it invites corruption and distorts the processes it was designed to monitor. Once manipulation is accepted as a permanent feature of any system of evaluation, the choice between measures should be based on who bears the harm that manipulation causes, a consideration that the debate over the Stanford list has largely ignored.

Judged on this criterion, a citation-based measure has the considerable advantage that the costs of its manipulation fall mostly within the academy. When a group of chemists inflates its citation counts, the immediate losers are other chemists who are passed over for grants and promotions, and the longer-run loser is the reliability of the chemical literature, which the profession polices through replication, retraction, and the scrutiny of rivals. Impact behaves differently because its raw material lies outside the university. To manufacture impact, a researcher needs a village in which to intervene, a ministry to advise, a patient group to enroll, or a public to persuade, and the incentive is in selecting these according to how readily they yield a change that can be documented and attributed to the researcher. A development economist rewarded for demonstrable impact has reason to favour interventions whose effects appear within a grant cycle, to overstate the part played by the research in decisions that a government would have taken in any case, and to cultivate a research relationship with officials who are willing to sign testimonials. The costs of each of these practices fall on households, patients, and taxpayers who never entered the competition.

British universities have had an official measure of impact since 2014, when the Research Excellence Framework (REF), which governs the allocation of a substantial part of public research funding in the UK, began to assess impact through narrative case studies. The weightage given to these case studies rose from 20 per cent of the exercise in 2014 to 25 per cent in 2021, and impact, now combined with engagement, retains that weightage in the round due in 2029. One university's estimate from the 2014 data put the value of a top-rated case study at no less than £324,000 in research funding over the REF cycle, roughly what five or six research outputs would otherwise earn. Universities responded by appointing impact managers, and when Richard Watermeyer, a professor of higher education at the University of Bristol, and his colleagues interviewed members of the assessment panels, the assessors described evaluations in which persuasive prose counted for more than actual evidence, reported that they had been advised to consult corroborating material only when a claim looked doubtful, and wondered whether the authors of supportive testimonials were friends of the researcher.

A distortion of outcomes

The draft UGC regulations would create conditions for a larger market of this kind in India, since the categories of notable contribution are broad and difficult to verify; the selection committees that apply them are small; and the distance between a well-connected professor and a district official, a panchayat, or a non-governmental organisation willing to certify an outcome is short.

A list of the world's top two per cent of lovers, for example, would strike most readers as absurd, and the absurdity illustrates how a ranking changes the thing it ranks. Were such a list to exist, love would become a prize which those with the most money, leisure, and social connection would find ways to win, and the conduct it rewards would bear a diminishing resemblance to love. The social researcher Richard Titmuss made a related argument about blood donation more than half a century ago, and economists now discuss the general phenomenon as the crowding out of intrinsic motivation. The case against ranking impact therefore depends on the distortion of outcomes that Campbell described, which affects beneficiaries no matter what the motives of the researcher are. A competition for impact would lead academics to select, design, and report their engagements with society according to what scores well, and the society on the receiving end would have to absorb whatever difference exists between what scores well and what helps.

Researchers inspecting the drought-hit areas of Yadgir, Karnataka, on November 17, 2018. A competition for impact would lead academics to select, design, and report their engagements with society according to what scores well. | Photo Credit: By Special Arrangement

Citation competition is, admittedly, less sealed from the wider world than it appears. Citation counts influence hiring, the allocation of grants, and the choice of research questions, and a literature inflated by citation cartels or research paper mills can find its way into the systematic reviews that inform clinical and policy decisions. Capital matters here as well, since the sociologist Robert Merton's account of the Matthew effect showed that recognition accrues disproportionately to scholars who are already recognised. Academics, moreover, already compete for impact through consultancies, media visibility, and advisory appointments, and not ranking impact would still leave that competition in place.

The harm that citation gaming does outside the academy, however, passes through peer review, editorial correction, and the evidence committees of medical and policy bodies, whose function is to filter the literature before it reaches practice, whereas the harm from gamed impact occurs in the field before any comparable filter can operate. Inequality in citations is an inequality among academics, which the profession has the means and the standing to address. While the existing market for impact is held in check by the absence of official weight, a formal ranking would convert an activity that some academics choose to pursue into a requirement imposed on all of them, with a corresponding increase in the scale of manipulation.

Economists Bengt Holmström and Paul Milgrom's analysis of agents who perform several tasks, only some of which can be measured, highlights the trade-off involved. When competing tasks call for an agent's time and one of them is rewarded while another cannot be measured reliably, strong incentives to carry out the first task draw effort away from the second. Leaving impact unranked would therefore reduce the effort that academics devote to it, and the effort that remains would be guided by their own judgment of what helps, free from the pressure to produce what can be documented. For an activity whose consequences fall on people who have no part in academic competition, a smaller quantity of undistorted engagement would serve society better than a larger one shaped by a scoreboard. The Stanford list, whatever its defects, is a contest whose casualties are mostly academics. Those who wish to reform academic evaluation in India would do well to keep the measure based on it, and to resist turning the lives of the people that research is meant to serve into the material on which academic careers are scored.

Anirban Ghatak is Assistant Professor of Economics at the Indian Institute of Management Kozhikode. Views are personal.