Reading this, very interesting, DFID paper on impact evaluations and causality it strikes me that the beauty if Randomised Control Trials in development isn’t to do so much with their internal validity, or philosophical arguments about causality but rather, simply, that for RCTs the devil isn’t in the details.
With multivariate regressions there’s always somewhere for the bad stuff to hide: weak instrumental variables, Granger Causality, poor quality data, absence or presence of particular control variables, running so many regressions as to eventually get one of those magic stars…
Likewise with qualitative work as a reader you can never be wholly confident that the author hasn’t prioritised some evidence over others, or heard some voices or not others. Or that they’re not extrapolating too far from a small non-representative sample.
With an RCT on the other hand things are pretty simple. External validity issues beyond your population or context of interest, sure. But run enough RCTs in enough places and you start to overcome this. And, crucially, what you do run is simple (usually) treatment versus controls. Fairly simple maths and an obvious effect, or not.
In the tangled world of development research that, I think, is the humble RCT’s most persuasive selling point.
Not the be all and end all, but nice, because they leave a lot less space for hiding things.
Comments Off on Where the Devil Isn’t
Via Development Impact, Tim Hartford on a huge community development programme, and a equally gigantic attempt to evaluate it:
Tuungane (“Let’s Unite”), a programme funded by the UK’s Department for International Development to the tune of £90m. The first £30m, Tuungane I, funded classrooms, clinics and other investments in 1,250 villages with a total population of almost 1.8 million people.
If the project is ambitious, what’s really fascinating is that Dfid has tried to evaluate Tuungane I rigorously, using a randomised controlled trial. Villages were enrolled in Tuungane through a public lottery. With 1.8 million people in the treatment group and a large control group, such an evaluation would be challenging to conduct in a rich country. In the DRC, where enumerators risk death and must wade through water up to their necks to reach the villages they are surveying, it’s mind-blowing.
Tuungane is a community-driven development project, or CDD. CDD is fashionable in development circles these days. The idea is that communities receive cash from donor agencies on condition that they come up with appropriate institutions for deciding how to spend the money. CDD projects will often insist on standards of transparency, democracy, or gender balance. The hope is that not only will the money be spent well, but that it may also catalyse accountable local institutions.
But does CDD deliver on this? The Tuungane I evaluation has now been completed by a team from Columbia University. The evaluation was based on a second project, Rapid, which was separate from Tuungane. Rapid handed out cash with few strings attached to Tuungane and non-Tuungane villages. The evaluators observed what happened: was the money embezzled, or well spent, and how were the decisions reached? This was the acid test of whether Tuungane had helped to promote effective village institutions.
The good news is that the Tuungane project was well executed and the money typically arrived where it was supposed to. In the DRC this is no small achievement.
Yet the results fall short of the CDD hype. The evaluators found that the Rapid cash grants were reasonably well spent whether or not a village had previously been exposed to three years of community building through Tuungane. Local institutions were more accountable than one might expect, but there was no sign that Tuungane could take any credit for this. Neither did Tuungane projects display any economic returns – although arguably there was too little time between spending the cash and evaluating the results for such returns to become apparent.
It’s a shame that the results haven’t matched the expectations of the CDD optimists, but it’s hardly a surprise. Tuungane is large in absolute terms, but it’s just a few pounds per person. It would be odd to expect miracles. The safe arrival of cash in the hands of very poor people is surely worth celebrating.
Equally interesting to me would be (a) whether there was meaningful variation in results depending on conditions in recipient villages (I’ve yet to read the paper) — for example, did the project perform less well in linguistically or tribally heterogeneous villages than in homogeneous ones (b) what might be revealed by process tracing about why various aspects of the project did and didn’t work.
The other thing I’m curious about, having witnessed the travails of a large data gathering exercise in the Pacific is whether you can really trust your survey teams to actually go and gather the data, rather than just claiming that they did. Especially if “enumerators risk death and must wade through water up to their necks to reach the villages they are surveying.” I’m not saying that inaccurate data creation is necessarily an issue in this particular study, I’m just saying it is where I work, and I really wish it wasn’t.
Unless you’re a saint, an already transcended practitioner of a transcendental religion, or not actually undertaking any development related fieldwork yourself, it’s likely that at some point during your data gathering you are going to end up feeling at least a little frustrated with at least a few of your research subjects.
I certainly do from time to time. Although it turns out I’m not even in the big league. The following excerpt comes from Paul Theroux’s book ‘The Happy Isles of Oceania.’ He’s writing about Bronislaw Malinowski, one of the founders of modern Anthropology.
I was reminded of how Malinowski, the most sympathetic of anthropologists, would spend a day among these laughing people [the Trobriand Islanders of PNG] and then go back and scribble vindictively in his private diary.
“The natives still irritate me, particularly Ginger, whom I would willingly beat to death,” he wrote. “I understand all the German and Belgian colonial atrocities.” Or: “Unpleasant clash with Ginger…I was enraged and punched him on the jaw once or twice.” Or: “I am in a world of lies here.”
If you’re lucky, development field research will afford you profound insight into human nature. Even if it doesn’t, as a consolation prize, you’ll almost certainly end up afforded insight into a very important aspect of your own nature: the grouch within.
Comments Off on Quite possibly you don’t want to know what I really think
I know I criticise them (particularly him) a lot, but there’s no escaping the fact that Easterly and his co-blogger Laura Freschi are bright, knowledgeable people, and their blog is a great place to learn. Some days that learning is limited to the perils of polemic. Others, like today, it’s not. It’s simply a really useful education about development. Bill and Laura, thanks for your post on the HDR. 🙂
Comments Off on In praise of…
William Easterly debates Peter Singer on bloggingheads.tv.
Surprisingly – given Singer’s support of aid and Easterly’s non-stop critique – it’s a civil enough affair.
Easterly does an admirable job improving Singer’s drowning child metaphor, but fails elsewhere. In particular, it’s odd how little he seems to know about the actual world of development practice, and of efforts to reform it. After implying that there’s an absence of campaigns to improve aid he waxes lyrical about the need to increase the participation of aid recipients in aid projects. Good idea. But hardly new. Next time he’s in England he should stop through Sussex, and maybe see if he can get hold of these two, almost out of print, books by this, nearly unheard of, author. (He could also read this critique while he’s at it, and possibly this follow up too.)
He also condemns tied aid. Great. But once again not new. Same with fragmentation. Campaigns such as One and Make Poverty History have been raising these issues for years, and they’re at the heart of the Paris Declaration (the donor nations’ own statement of intent).
Of course, none of the issues above have been resolved yet – so all power to Mr Easterly in his efforts to keep the flame burning. But it would be nice – given his propensity to ridicule aid practitioners (the One campaign in particular) – if he acknowledged that he isn’t the only one pushing for change.
More generally, Easterly’s comments are symptomatic of a larger issue – it’s surprising how few academics working on development related work really understand the work of aid agencies and NGOs: the trade offs involved; the role of politics and ideology; and the long, winding road towards (we hope) better practice. Which is a pity, because the two communities – practitioners and academics – have a lot to offer each other.
Comments Off on Easterly v Singer