New tools aim for nuanced analysis of academic research citations

Citations have long been the default measure of academic influence, offering a seemingly simple way to track a scholar’s impact. The logic is straightforward: the more a piece of research is cited, the higher its perceived value.

But the limitations of this tool are increasingly apparent. Citations focus on immediate academic influence and often miss the bigger picture, especially the broader societal impact that research can have.

“What citations tell us is the downstream impact on the academic world, but they do not tell us whether those ideas are being taken up in practice,” says Ludo Waltman, professor of quantitative science studies at Leiden University in the Netherlands.

So why do citations retain such appeal? Diana Hicks, a professor in the School of Public Policy at Georgia Institute of Technology, says: “Citations have held sway because they provide a means for outsiders to get a sense of the contributions made by researchers — evidence that they have advanced knowledge.” She warns, however, that relying too much on this single metric is “naive”.

For one, self-citations, whereby academics reference their own work, can distort the numbers. “It is completely reasonable to self-cite if you are building on a body of research,” says Elizabeth George, editor of the Academy of Management Annals and a professor at Cambridge Judge Business School. But she also points to “gaming” of the system, when self-citations are used to artificially inflate the tally.

What is more, citation practices vary across disciplines, making comparisons difficult. Fields such as medicine tend to attract more citations because of its vast research community, clinical relevance and interdisciplinary nature.

Another concern is that research in regions where English is not the primary language can be overlooked because English-language journals dominate academic publishing. “Such work will disappear if citation counts become the only thing that matters in research assessment,” warns Hicks.

In response to these concerns, new tools are emerging that offer a more nuanced view of academic influence. OpenAlex, a platform launched in 2021, uses a metric to adjust for differences between fields, aiming to give a more balanced picture of research impact across disciplines. Scite, another tool, goes further by evaluating the quality of citations, whether they support or challenge the work.

Scite’s focus on quality as well as quantity

Brooklyn-based start-up Scite is looking to change how academia views citations. Instead of simply counting how often a paper is cited, Scite tracks the nature of those citations — whether they support, challenge or merely mention the work.

“The ability to consider the content of citations, rather than their number, is a big shift from traditional approaches,” says Sean Rife, Scite’s head of academic relations. “We need to consider not just the quantity of academic work, but its quality as well.”

The ongoing “replication crisis”, where many studies fail to be reproduced, inspired Scite’s creation in 2018. Rife warns that relying too much on any single metric, such as citation counts, can encourage bad behaviour in academia.

Scite supplements traditional tools such as the h-index — a number that shows how many of a researcher’s papers have been mentioned by others at least that same number of times — but adds depth, bringing qualitative analysis to citation tracking.

According to Rife, this approach, combined with platforms such as OpenAlex, marks a shift towards more transparent and nuanced ways to measure academic impact. “We are seeing a shift away from relying on a small number of data providers towards using openly available data in creative ways,” he observes.

Despite the rise of alternatives, traditional citation metrics still hold sway in hiring and tenure decisions. “Performance evaluation based on academic citations is key, but needs to be completed by other metrics that provide adequate proof that research was helpful to practitioners and students,” says Emmanuel Métais, dean of Edhec Business School in France.

To ensure research is making a broader societal impact, Edhec also factors in external funding, media exposure and student outcomes. “We value and encourage our faculty to leverage the knowledge they create . . . to drive progress beyond campus walls,” Métais says.

For now, therefore, citations remain an important, albeit flawed, part of academia’s evaluation process. But, as the sector comes to terms with the limitations of these metrics, the push for more meaningful and balanced measures is likely to continue.

Some are already gaining ground. Metrics such as “contextualised” citations (which account for different disciplines) and “positive” citations (tracking how research is applied and built upon) offer a more nuanced take on scholarly impact. Below, we spotlight some of the top academic articles excelling in these new metrics, providing a fresh perspective on influence and impact across fields.

Aggregate confusion: the divergence of ESG ratings

This paper tackles a growing issue in sustainable investing: why different environmental, social and governance (ESG) ratings can paint radically different pictures of the same company.

With some $100tn in assets tied to ESG criteria, this matters. Investors and companies rely heavily on these ratings to inform decisions, from equity investments to corporate strategies.

The study, co-authored by Florian Berg and Roberto Rigobon of MIT Sloan School of Management, along with Julian Kölbel, at the University of St Gallen, found that measurement differences between rating agencies are the main source of divergence, accounting for 56 per cent of the variation. Scope and weight differences make up the rest. 

This creates a problem: companies receive mixed signals about what actions to prioritise, leading to potential under-investment in sustainability initiatives. It also makes it harder for investors to assess real ESG performance, and for markets to accurately price ESG factors into equity values.

The paper says the divergence could also hamper efforts to link chief executive pay to ESG performance and undermine research in sustainable finance, where results can vary depending on which rating is used.

By exposing these inconsistencies, the research highlights the need for standardisation, noting that regulators could help by harmonising ESG disclosure practices. That would make ratings more reliable and useful for decision makers. 

Corporate immunity to the Covid-19 pandemic

The paper explores how different corporate characteristics influenced stock price reactions during the early stages of the pandemic in 2020. 

By poring over data on some 6,000 companies across more than 50 economies, the study finds that companies with stronger pre-pandemic financial health — those with more cash, less debt and higher profitability — experienced milder stock declines. 

Additionally, companies with more exposure to global supply chains and customers in highly affected regions saw greater stock price drops, according to the paper from Wenzhi Ding from Hong Kong Polytechnic University, Ross Levine from Berkeley Haas, Chen Lin from HKU Business School and Wensi Xie from CUHK Business School.

The paper also highlights the role of corporate social responsibility (CSR). Companies that had invested in CSR prior to the pandemic saw better stock performance during the crisis. This suggests that strong relationships with employees, suppliers and communities may increase corporate resilience in times of crisis. 

Furthermore, companies with more flexible governance structures to enable mergers and acquisitions or leadership changes performed better.

The findings underline the broader importance of corporate health and social responsibility, offering insights for both policymakers and corporate leaders in building more resilient companies.

Artificial intelligence and management: the automation-augmentation paradox

The paper explores the increasing use of artificial intelligence in management, highlighting the tension between two key approaches: automation, where machines take over tasks, and augmentation, where humans and AI collaborate.

The research, by Sebastian Raisch of Geneva School of Economics and Management along with Sebastian Krakowski of Stockholm School of Economics, highlights a paradox: while some management thinkers advocate augmentation as the better strategy, the authors argue that organisations cannot fully separate it from automation.

Instead, they argue that automation and augmentation are interdependent, creating tension as companies try to strike a balance between efficiency and innovation.

The paper argues that focusing too much on either approach can result in negative consequences. Relying solely on automation risks job losses and deskilling, while an overemphasis on augmentation can lead to inefficiencies and reinforce human biases.

The research advocates a balanced, integrated approach, urging management practices that not only boost organisational performance but also deliver broader societal benefits, such as fairer job markets and more responsible AI use.

Corporate green bonds

This research explores the sharp rise and impact of green bonds: financial instruments that companies, including Unilever and Apple, have used to fund environmentally friendly projects, such as renewable energy and resource conservation.

Caroline Flammer’s paper, written during her time at Boston University’s Questrom School of Business, shows that green bonds are gaining momentum, with annual issuance having more than doubled in each year between 2013 and 2016.

Companies that issue green bonds tend to see a strong market response, with noticeable gains in shareholder value. For instance: company market value rises by 2.2 per cent after one year, building on the initial 0.7 per cent shareholder value gain seen at the bond announcement. Profitability, reflected in return on assets, also improves in the years following green bond issuance.

The research from Flammer, now at Columbia University, shows that companies issuing these bonds have typically achieved an 8.8 per cent rise in their environmental ratings and a 27.7 per cent drop in their CO₂ emissions. Pointing to a growing societal impact, the issuing companies’ share of “green” patents for environmentally friendly inventions rises by 3.4 per cent, relative to all patents. 

Flammer’s findings suggest that green bonds can support both environmental progress and long-term value creation.