Counting scientists’ productivity with numbers undermines science | Explained Premium

Stanford University’s John Ioannidis recently ranked the top 2% of scientists in various fields. This list is based on 6 citation indicators, including total citations and single-author citations. However, this system has drawbacks, such as not accounting for quality/impact of work, not extrapolating between fields, and incentivising unethical practices. Ranking systems and metrics should not be the only way to evaluate scientists’ work. In fact, reading their papers is the best way to do so.

Scientists at Stanford University recently ranked the ‘top’ 2% of scientists in a variety of fields. The ranking contained an up-to-date list of the most highly cited scientists in these disciplines. That is, the list consists of the top 100,000 scientists based on an aggregate of numerical indicators, equal in this case to those scientists who have authored papers whose citation count lies in the top 2% in each field.

The presence of Indian scientists in this list has garnered substantial public attention. They have been accompanied by institutional press releases, news features, and award citations.

Given this fanfare, it’s important that we understand what the top 2% ranking system actually measures and how well the measure correlates with real-world scientific achievement.

The 2% ranking system is based on standardised citation metrics across all scientific disciplines. In the scientific research setting, a citation is a reference to a piece of information, typically an already published entity like an article, book, or a paper in a journal. For a scientist, being cited means that some scientific publication that they have authored has served as a reference, basis or source for some parts of subsequent research in the field.

Using scientific publication data from the Scopus database of published papers, maintained by the publisher Elsevier, the 2% ranking system uses a composite citation index that is based on six citation indicators. These are: (i) the total number of citations for a paper, (ii) the total number of citation for a paper where the scientist is a single author (i.e. no co-authors), (iii) the total number of citations for papers where the scientist is a single or the first author, (iv) the total number of citations for papers where the scientist is a single, first or last author, (v) the number of papers for which the scientist has been cited at least the same number of times (h-index), and (vi) the number of citations per author for all the authors of a paper.

By combining the values of these indicators, the 2% ranking system assesses scientists’ citation impact in a single calendar year as well as throughout their careers.

This way, using different scientometric indicators, many scientists have attempted to numerically quantify their peers’ scientific achievement. For example, the ‘AD scientific index’ measures the productivity coefficient of a scientist using the h-index, the i-10 index (number of publications with at least 10 citations), and other numbers; the ‘h-frac index’ tracks the fractional allocation of citations among co-authors; and the ‘Author Contribution Scores’ computes a continuous score that reflects a scientist’s contributions relative to those of other authors over time. There are many others.