Why some economists are skeptical of this year’s Nobelists

The Nobel Prize in Economics awarded this week to Daron Acemoglu, Simon Johnson, and James Robinson allows a lot of people to feel like winners. (Especially those in the know who refer to the group by the acronym “AJR.”)

The general public wins in that all three have, unusually for academic economists, written extensively for lay audiences; Acemoglu and Robinson’s 2012 book Why Nations Fail was a bestseller, and Johnson has had several prominent books on financial regulation and innovation. Economic historians get a win, in that Acemoglu, Johnson, and Robinson’s most famous work was on historical processes of development in former colonies. Lefties win, in that Acemoglu has of late become a vocal proponent of policies to expand worker power and has mused about the possibility of “AI-enabled communism.”

That said, you don’t get to 4,400 citations without earning a few critics. The fairest hit against the three is that while their theories are elegant, the data underlying them are shaky at best, and the results don’t hold up to scrutiny.

That’s not as terrible as it might sound. All science progresses through new findings that themselves are later overturned. I think Acemoglu, Johnson, and Robinson are great economists and even flawed empirical findings can be important in advancing a field. The physicist JJ Thomson — another Nobel winner — famously and erroneously proposed that atoms lacked a nucleus, but that doesn’t make his previous discovery of the electron any less important.

But this week, with considerable and mostly uncritical public attention on the Nobel winners’ work, I think it’s important to talk about its shortcomings, and the need to subject influential findings like theirs to further testing.

AJR’s most famous work was an intervention into one of the longest-running debates in economics: Why are some nations so rich and other nations so poor?

The collaborators sought to rebut geographic determinists (notably Jared Diamond) who argued that the characteristics of the land were responsible for, say, Europe being richer than Africa. AJR’s answer was that some countries had better, more “inclusive” institutions that allowed the fruits of economic growth to be broadly shared, whereas others had “extractive” institutions where a small cabal could capture all the gains. The former grow over the long run; the latter don’t.

Their most famous paper, “The Colonial Origins of Comparative Development: An Empirical Investigation,” sought to measure the effect of inclusive versus extractive institutions. To do this, they had to find some factor that caused certain areas to have certain kinds of institutions, but that was otherwise unlinked to their economic development. In econometrics this is called an “instrumental variable,” and the theory is that controlling for such variables lets you isolate the causal effect of the independent variable (in this case, institution type) that you’re studying.

Their instrument was “how often European settlers died.” Think about Australia on the one hand and Nigeria on the other. Both were colonized by the United Kingdom; Australia has, I think it’s fair to say, stronger, less corrupt institutions.

What AJR proposed was that Australia became Australia because it was reasonably hospitable terrain for the European colonizers, the many sharks and spiders notwithstanding; they could, and did, move there in large numbers. They then had incentives to build institutions that benefited white settlers.

In Nigeria, by contrast, diseases like malaria and yellow fever killed off huge numbers of British settlers, so a similar settlement project couldn’t get off the ground. With comparatively few white settlers, the British didn’t give a shit about building fair institutions, because they were effectively building them for Africans — and the British cared far less about black people’s welfare than white people’s.

Sure enough, AJR found that countries with high European settler mortality during colonization have lower per-capita incomes today, which they saw as evidence for their view that institution type is determinative. A nearly as-famous paper the following year, “Reversal of Fortune,” extended the argument, finding that among countries colonized by Europeans, those that were most successful in 1500 (where “success” is measured by urbanization or population density) are disproportionately poor today.

Those results point against geographic explanations, AJR argued, and toward institutional changes wrought by European colonization.

There’s a lot that’s appealing about the AJR worldview. Government institutions do seem important; there’s no other plausible reason why South Korea is one of the richest places on Earth and North Korea is perhaps the poorest. The theory is a hopeful one: While countries cannot change their geographies, they can adopt new, better institutions.

But do the specific empirical claims AJR made hold up? It doesn’t seem like it. Economist David Albouy offered the most persuasive reply to their 2001 “Colonial Origins” paper by digging into the actual data. AJR used a sample of 64 countries, but only had real data for 28 of them. The other 36 had data that were assigned based on “conjectures the authors make as to which countries have similar disease environments.”

As you might expect, making up data on settler mortality for places where we have no data is difficult, and Albouy finds serious flaws in how AJR do it; for six countries, he finds, their estimates “are based on an incorrect interpretation of former colonial names for Mali.”

Once you only look at the 28 countries with non-synthetic data, there’s no relationship between settler mortality and present-day economic outcomes. Worse, even the real data tends to be about soldiers rather than civilian settlers, and soldiers are more likely to die from disease when actively fighting than civilians are.

This biases the results in AJR’s favor, and makes the underlying relationship they posit (that places with higher death rates developed worse institutions) much weaker.

Another reply, by Ed Glaeser, Rafael La Porta, Florencio López de Silanes, and Andrei Shleifer noted that AJR’s data doesn’t distinguish between the effects of institutions and the effects of human capital: Settler colonies like Australia and Canada didn’t merely get more inclusive institutions, but also settlers who were generally much richer and better-educated (at least from a modern capitalist vantage point) than native inhabitants. The researchers conduct their own tests and argue that human capital does a better job explaining growth trajectories than institutions. That doesn’t necessarily make for a bleaker story than AJR (countries can invest in schools and boost human capital), but it’s a different story.

Glaeser and co. also highlight problems with the measure of “expropriation risk” (the risk that the government takes all your stuff) used in AJR’s work. This is an important indicator for AJR of whether institutions are inclusive or extractive, but it turns out to just be a subjective 0 to 10 ratings system AJR took from the private firm Political Risk Services, and one with huge problems. “In 1984, the top ten countries with the lowest expropriation risk include Singapore and the USSR,” Glaeser et al note. Are we really expected to believe that your risk of getting your stuff taken by the government was low in the Soviet Union?

AJR’s claim of a “reversal of fortune,” with the leading nations of 1500 becoming laggards today, has similarly withered under scrutiny.

Areendam Chanda, C. Justin Cook, and Louis Putterman reevaluated the claim but measured what happened to the descendents of those actual 1500s people, not just the geographic places where they lived. There have been huge movements of people from 1500 to the present, and it doesn’t strictly make sense to compare the Incan Empire to Peru today given how wildly different the people in each were. Chanda et al find that fortune has not reversed, but persisted when you account for population movements: People descended from countries thriving in 1500 were doing better in the 21st century. That’s evidence, they conclude, for the Glaeser et al claim that human capital rather than institutions is the crucial factor here.

Again, my takeaway here is not “Acemoglu, Johnson, and Robinson, Nobel-winning economists, are useless.” They’re incredibly useful, and greatly amplified the prestige of these kinds of tough economic history questions within the economics profession.

But I also think their work is a reminder of the old academic cliché that you can either learn something very small about something very big, or something very big about something very small. They were tackling a very big topic, and seemed for a second to grasp something very big about it. Upon inspection, though, it looks a great deal smaller.