Archive for Impact Evaluation

project placement bias in the evaluation of Christianity

October 19, 2009 at 6:01 · Filed under Impact Evaluation, Reading

If Christianity is true then it ought to follow (a) That any Christian will be nicer than the same person would be if he were not a Christian. (b) That any man who becomes a Christian will be nicer than he was before. … Christian Miss Bates may have an unkinder tongue than unbelieving Dick Firkin. That, by itself, does not tell us whether Christianity works. The question is what Miss Bate’s tongue would be like if she were not a Christian and what Dick’s would be like if he became one. …

We must, therefore, not be surprised if we find among the Christians some people who are still nasty. There is even, when you come to think it over, a reason why nasty people might be expected to turn to Christ in greater numbers than nice ones. That was what people objected to about Christ during His life on earth: He seemed to attract “such awful people.” (C.S. Lewis, Mere Christianity [available in full on-line], Book 4, Chapter 10)

Leave a Comment

just dumping computers in schools might not help

March 12, 2009 at 14:29 · Filed under Development, Impact Evaluation

In The Use and Misuse of Computers in Education: Evidence from a Randomized Experiment in Colombia, by Felipe Barrera-Osorio and Leigh L. Linden, the authors examine a program that

aims to integrate computers, donated by the private sector, into the teaching of language in public schools. The authors conduct a two-year randomized evaluation of the program using a sample of 97 schools and 5,201 children. Overall, the program seems to have had little effect on students’ test scores and other outcomes. These results are consistent across grade levels, subjects, and gender. The main reason for these results seems to be the failure to incorporate the computers into the educational process. Although the program increased the number of computers in the treatment schools and provided training to the teachers on how to use the computers in their classrooms, surveys of both teachers and students suggest that teachers did not incorporate the computers into their curriculum.

Two thoughts on this:

  1. This reminds us – and I’d say “as if we needed reminding” except that we do – that you cannot just dump inputs into schools and expect changes.  If inputs don’t get used well, they don’t matter.  Even though this seems like a no-brainer, many development programs are very narrow: build a school or give some books or ….  Same problem, I’m afraid.
  2. That said, a quick look at the tables suggests to me that the authors may be confusing a noisy result with a narrowly bound zero result.  In other words, there seem to be differences in outcomes between kids who got computers and those who didn’t, but there is so much variation in both groups that we cannot be sure.  What this really means is that we don’t know if there is an effect, that there might be a heterogeneous effect, or there might not.  (Either way, clearly this program wasn’t a raging success.)

There is also some evidence from India (I haven’t evaluated the quality) that if you just let kids play with the computers, they’ll learn some stuff.  (One question is, Do they learn things that will help them?)

Leave a Comment

impact evaluation matters: how long do i have to wait to know if my program works?

September 4, 2008 at 19:33 · Filed under Development, Economics, Impact Evaluation

King and Behrman have a paper on Timing and Duration of Exposure in Evaluations of Social Programs. The paper gives a useful (if not adrenaline-fueled – this is a review, after all) discussion of a host of issues to consider when deciding when to look for results of social programs. [For example, what if the program ends up rolling out at different times in different places? etc] They then give lots of examples of papers that have dealt with the issues (and how they’ve done it). Instructive stuff.

Summary: Impact evaluations aim to measure the outcomes that can be attributed to a specific policy or intervention. Although there have been excellent reviews of the different methods that an evaluator can choose in order to estimate impact, there has not been sufficient attention given to questions related to timing: How long after a program has begun should one wait before evaluating it? How long should treatment groups be exposed to a program before they can be expected to benefit from it? Are there important time patterns in a program’s impact? Many impact evaluations assume that interventions occur at specified launch dates and produce equal and constant changes in conditions among eligible beneficiary groups; but there are many reasons why this generally is not the case. This paper examines the evaluation issues related to timing and discusses the sources of variation in the duration of exposure within programs and their implications for impact estimates. It reviews the evidence from careful evaluations of programs (with a focus on developing countries) on the ways that duration affects impacts.

Leave a Comment