Waylaid Dialectic

June 16, 2012

Significant?

Filed under: Random Musings — terence @ 8:08 am
Tags: , ,

Reviewing a review of the Randomistas Berk Ozler points to something that troubled me when I read the book: many of the interventions that have been touted on the basis of delivering statistically significant improvements vis a vis control groups when tested in Randomised Control Trials actually haven’t had that significant of an impact in the everyday sense of the term.

Suppose that you’re told that a program reduced the rate of dropping out of school among 15 year-olds by 17% and this reduction was statistically significant. You are also told that the same figure among 12 year-olds is 38%. You would likely take note. Suppose now you’re told that these are the effects of a conditional cash transfer program, where the dropout rate among the control group is 37.7% and 16.8%, respectively for ages 15 and 12, thus the absolute effect sizes are 6.4 percentage points in each case.

When you hear the first sentence, you are likely to miss two things that you might have thought about had you been given the latter facts about the same program effects – simply stated differently. First, in the latter case, you might say: “31.3% of 15 year-olds dropped out of school even though the government offered their families a considerable sum?” Second, you notice the fact that the program averted just six dropouts for every 100 transfers. Would you be surprised if you were then told that these are the effect sizes of the much-heralded PROGRESA? If you are, feel free to examine Table 1 in this paper (gated, sorry no WP versions around) by Behrman, Sengupta, and Todd (2005). [Emphasis mine.]

I think you can still defend PROGRESSA and similar Conditional Cash Transfer (CCT) programmes in light of this. Mounting a defence along the lines of: (a) in the difficult world of development interventions some improvement is better than none, which is often all we get for a our money; and (b) small things add up to big things over time.

But at the same time Berk’s example strikes me as an incredible important reminder of the perils of being blinded by the ‘I have a result!’ effect of statistical significance.

Also, in light of the actual effect magnitude of PROGRESSA it’s hard not to think that the most appropriate response to its findings might not have actually been for us all to race off touting the wonders of CCTs and trying to cultivate them everywhere, but rather to dwell the fact that it turns out that for the poor, whether they send their children to school or not is, for the most part, something that happens regardless of short run financial incentives (obviously only up to a point of course).

In terms of future research regarding education in development quite possibly the most significant finding of those first PROGRESSA studies should have been that about a third of parents weren’t able to get their kids to school, or didn’t think it worth it, even in light of the money bonus that was on offer. Or that a significant majority of poor parents send their kids to school regardless of the financial costs that PROGRESSA was designed to offset. These would seem to be the real stories of the PROGRESSA results, even if we were to conclude that the programme’s small improvements were still significant enough to make it worth it.

April 27, 2011

Read About it Here First! The Next Big Thing in Aid!

Filed under: Aid,Development Theory,Social Justice — terence @ 12:55 pm
Tags: , , , ,

It’s hard to escape the fact that we development folks are, at times, awfully Charlie Brownesque. Like Charlie Brown, who was forever being fooled by Lucy’s promise to hold the football while he kicked, we keep falling for things we shouldn’t. We’re suckers for the Next Big Thing all the while forgetting how the last big thing left us in the lurch, or how the the big thing before it did the same too.

And so we have, Cash transfers (either conditional or unconditional – CCTs and UCTs) and Cash on Delivery Aid (COD) lining up as the latest Big Things offering to hold the ball for us. Micro-finance, the last big thing, is looking rather bruised, while ICT for development has got to be feeling even less loved these days.

I’m no fan of COD aid, but I’ve got nothing against CCTs and UCTs. They were introduced to South America by the Worker’s Party, who I still kindof idolise, and there’s good evidence that they really have worked well in some places. But I’m deeply sceptical of their potential as panaceas. Which is another way of saying I completely agree with Laura Freschi’s measured scepticism at AidWatch.

The good news here at least is that Freschi is making reference to recent DFID funded research. DFID, it would seem, have learnt a few lessons and seem to be systematically gathering evidence before hopping on the Cash Transfer bandwagon.

Which is kind of cool. Imagine if the next big thing in development wasn’t a thing at all, but rather an approach: move slowly based on the evidence you can use. And consider context before adopting stuff which worked elsewhere. Even this wouldn’t be unproblematic but it would surely have to be better than falling for the latest fad time and time again.

On a completely different subject, but linked by the theme ‘these are people are I don’t normally agree with’ I reckon Jonathan Glennie is almost word perfect in his critique of Zizek.

Create a free website or blog at WordPress.com.