Last year, the development nonprofit Evidence Action did something remarkable: It stopped fundraising for one of its programs because it had conducted a study that suggested it was less effective than it had thought.
This charity just canceled one of its poverty programs. That’s a good thing.
By ending a program that was less effective than it thought, Evidence Action offers a model for other nonprofits.
Kelsey Piper is a senior writer at Future Perfect, Vox’s effective altruism-inspired section on the world’s biggest challenges. She explores wide-ranging topics like climate change, artificial intelligence, vaccine development, and factory farms, and also writes the Future Perfect newsletter.
The program was called No Lean Season. It built on evidence that suggested that small subsidies to allow poor rural families in Bangladesh to migrate during periods when there was no agricultural work were a big help. No Lean Season gave Bangladeshis the money for a trip to the city, where they could find more reliable work. It seemed like a great program — until Evidence Action’s study last year found otherwise, and prompted it to put further fundraising for No Lean Season on hold.
Yesterday, Evidence Action announced that it was pulling the plug altogether. The disappointing study wasn’t the whole story — Evidence Action had found other problems and fired its local partner for the program. In light of these discouraging developments, Evidence Action concluded its resources could do more good for the world’s poorest people when devoted to other projects.
Here’s why this is noteworthy: Most development programs are not rigorously studied, and they’re often run by people who want to keep them going — as long as there’s any evidence they could be a good idea, or any reason to wait for more evidence, or any recent program changes that could have made them more effective.
Evidence Action is doing something different. (Disclosure: I donated $5,000 to Evidence Action in 2018.) It’s trying to be quick to test programs, and quick to admit if the programs don’t work or just aren’t likely to be the best possible place to spend money. It’s also being unusually transparent with the public, which means we get a much clearer picture of how it operates, what went wrong, and why EA ended up firing its partner in Bangladesh. It’s exciting to see us moving toward an approach to development that is willing to cancel programs when it can do better elsewhere — even if, in cases like this one, it’s also painful.
What went wrong for No Lean Season?
Launched in 2013, Evidence Action tries to run anti-poverty programs with a strong evidence base. In addition to its core programs, which provide deworming and clean water, it tries to incubate and scale new programs. One of its core programs, Deworm the World, is a GiveWell-recommended top charity — and No Lean Season was briefly one as well.
After the discouraging 2017 trial, the results of which knocked No Lean Season off GiveWell’s top charities list, Evidence Action stopped fundraising for No Lean Season but kept running it, waiting for the results of a 2018 study that might explain why the promising results from small-scale trials hadn’t appeared in a large-scale one.
When I spoke to him last fall, Yale economist Mushfiq Mobarak had a theory about why the results didn’t scale up: the choice of recipients. Trying to meet quotas for how many loan recipients they signed up, program officers may have signed up a population composed almost entirely of people who were already planning to migrate with or without a loan. (To have an impact, the program has to sign up people whose behavior is changed by the loan.)
That’s one of the hypotheses that the 2018 trial was intended to test. But before the results were released, No Lean Season hit other roadblocks. First, Evidence Action leadership in the US received word that the government approval for the program wasn’t actually valid; there are allegations it had been forged by a junior government official. Evidence Action had partnered with a local Bangladeshi organization whose responsibilities included obtaining that approval, and heard an allegation that the organization had sought to pay a bribe to have the program approved.
Evidence Action retained an independent legal and audit firm to look into the allegations, and found one improper reimbursement — approximately $400 to a “consultant” for unclear consulting work. It also found the local organization unwilling to cooperate with the investigation. Evidence Action stopped its relationship with the local organization.
That changed the calculus for No Lean Season. If it wanted to continue the program, it would need to find another local partner. It would take at least a couple of years just to return to the program capabilities it had in 2018. And while there might be a way to make the program work — while the soon-to-be-released study still might show that it works as currently implemented — it simply didn’t look like that was the best possible use of Evidence Action’s organizational capacity and resources.
Rather than asking, “Is No Lean Season so bad that we’re obliged to cancel it?” Evidence Action asked, “Is No Lean Season so good that it’s the best possible way we can spend our money and organizational capacity to address global poverty?”
The answer: It wasn’t. So it’s canceled.
No Lean Season was a great idea to test. It might have worked. It still might work. But there are a lot of urgent problems in the world, and right now, with current budgets and resources, we have to prioritize ruthlessly.
Several times now, Evidence Action has done that even when it meant a big disappointment, and even when it meant publicizing some things — like misuse of donor funds and alleged requests for bribe money — that many nonprofits would keep under wraps.
And so, while this is unfortunate, it’s also a sign of something good — a sign that the world of global development is changing. It raises the bar for how nonprofits test their ideas and how they react when the evidence is against them. And while that might mean disappointments in the short term, it also means that in the long term, resources will be used as effectively as possible. In a world where so many people still live in extreme poverty, that’s what matters.
Sign up for the Future Perfect newsletter. Twice a week, you’ll get a roundup of ideas and solutions for tackling our biggest challenges: improving public health, decreasing human and animal suffering, easing catastrophic risks, and — to put it simply — getting better at doing good.
See More:
