Challenges in measuring quality of research

Assessing faculty research output through citation index raises questions about the true quality of academic contributions.

Today, the performance of faculty members in most academic institutes, especially in Engineering, is being judged by their research output. However, this begs certain pressing questions. Why is so much thrust being put on the ‘research’ component for career advancement when a large number of institutions do not have a sound research infrastructure, unlike some premier government-funded tech institutes? Why is it not sufficient if faculty focus on imparting skills to students through innovative teaching and lab-related work, along with additional responsibilities such as mentoring and career guidance? Of course, with rapid developments and changes in technology, faculty members have to upgrade their knowledge to help students learn and evolve.

As of now, there are not too many effective guidelines to assess a researcher’s true potential. A key yardstick of assessment is the citation index. Many institutes have been rewarding faculty researchers based on their citation index, which translates to having a good citation score in indexing sites such as Google Scholar, Scopus, Web of Science, Research Gate, and others. This system is a catalyst to some faculty researchers producing quantitative research and even being mentioned as the top 1% or 2% researchers in a list by a top-ranking world institute.

Evaluating research contributions through citations can be erroneous. Assume that a researcher makes a crazy claim and writes a number of bad research papers. It is obvious that an avalanche of criticism from expert researchers in the area will follow. This will enable the researcher to become one of the most cited researchers with a good citation index. The moot question is: how do we evaluate what is ‘good’ and what is a ‘less acceptable’ research. There are additional issues of ghost writing of research papers and the problem of predatory journals.

There have been instances of researchers routinely flaunting their Google citation count and h-indexes of their research on social media, which are higher than the citation count of few internationally-acclaimed Indian researchers. It would be a stretch of imagination to conclude that such researchers are superior than the internationally-acclaimed ones. One can always come up with a counter-argument as to why not?

The objective is not to belittle researchers having an impressive citation count. As outlined earlier, high citation count can arise out of not-so-great research. It can also increase when researchers form a group and keep referencing each other within the group, often with little reason. Many universities/institutes have started rewarding researchers based on the number of index papers and citation counts in the absence of a more comprehensive measure for qualitative research.

Unless we develop a good enough metric to evaluate constructive research contribution, it will be difficult to bring ethical practices in academic publications in most institutions. There is a need for the people at the helm of policy-making committees in these institutions to ponder over and stem the flow of a large number of publications with insignificant contributions, written merely to increase the count number.

The writer is a Professor of Computer Science and Engineering with three decades of experience and based in the National Capital Region. Emailprofessor.sghosh@gmail.com