Abdul Latif Jameel Poverty Action Lab

Transcription

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu. So for the course, we have four days of lectures. Today we'll try to convince you that it was actually a good idea to come here, why randomized evaluation is such a useful tool and why it's superior to many other kinds of impact evaluation. Once we've convinced you that it's a good idea to come here, then we'll start going through the nuts and bolts of actually how to run randomized evaluations. Tomorrow we'll go over some of the general design possibilities. The following day we'll go into some more of the technical aspects like sample size and measurement. The last day we'll kind of discuss the fact that even in randomized evaluations, things can go wrong, and how you deal with that. And throughout the entire course as you're learning this, you'll be incorporating what you learn by designing in step with the lectures your own randomized evaluation in the groups that you've been preassigned. So if you check out your name tags, you'll see that you have a group number. Find other people with the same color and number and those will be your group members. We tried to put you together in ways that made sense. So we tried to have people who were interested in agriculture work with other people within agriculture. And over the course of the four days, you'll be designing your own randomized evaluation. And on the last day, on Saturday, you will be presenting your evaluation to the entire group. So that is pretty much what to expect over the next five days. Have I forgotten anything? Then let me reintroduce Rachel who will start five minutes early. RACHEL GLENNERSTER: The group the Mark was talking about is a really integral part of the course. So unfortunately that is not something that you'll be able to get online. But hopefully it's a chance for you to go through, each time we present an idea in the lecture, you then need to go and apply it in the case that you're developing in your evaluation you're developing in your group. And that's what all the teaching assistants are here, to help you go through the case studies, but also to develop your own evaluations. So I'm going to start with the general question of what is an impact evaluation and when should we do one. One of the objectives of this lecture is just to make sure that we're all on the same page when we start using terms like process evaluation and impact evaluation. Because I realize the more I spend time with people who are kind of professional evaluators, I realize that economists and professional evaluators use the same terms but to mean different things, which is incredibly confusing. So I'm sure a lot of this will be familiar to you, but on the other hand, we need to make sure that every one is at the same level in using the terms in the same way before we kind of head into the nuts and bolts. But I also have incorporated in this discussion about when you should do an impact evaluation, which is something that comes up an awful lot when I go and talk to organizations who are trying to think through their evaluation strategy. They've heard that they ought to be doing more impact evaluations. There's lots of focus on this. But they're expensive. So how do they decide which of their programs to evaluate and at what stage to evaluate it. They're getting pressure from donors to do this. But they're not quite sure when it's appropriate. So we'll try and cover those ideas. So we'll start with why is it that we here at J-PAL are focused on impact evaluation. Because there's lots of other things in evaluating our programs that are important. But we just do impact evaluation. We also only do randomized impact evaluation. And that's not to say that's the only thing that's worth doing. We certainly don't think that. That's what we do. There's a reason we do it, because we think it's important. But it's certainly not the only thing that you should be doing in your organizations. So step back and look at the objectives of evaluation and a model of change, which is very important in terms of how to think about your evaluation, how to design it, different types of evaluation, how evaluation feeds into cost benefit analysis, and then into why to do an impact evaluation, and putting it all together into an evaluation strategy. And then coming back to how do we learn. How do we make an organization that learns from its evaluation strategy rather than just doing this as something a funder wants me to do or I have to do to tick a box. How do I develop an organization that learns from its evaluations and makes it a better organization? So this is the motivation for what we do. And I think this point is sort of the main point here. If you step back and you think about how much evidence we have in development, to make the decisions that we need to make, it's really quite appalling how little information we have. If you think about some of the biggest challenges in the world in development about how to prevent the spread of HIV/AIDS in Sub-Saharan Africa, how to improve the productivity of small farmers across the world, it's really amazing how little really rigorous evidence we have to make those decisions. And we may know that this project may work or that project may work, but we very rarely know what is the most cost-effective place to put a dollar that I have. If I'm choosing in HIV prevention, if I've got to choose between a lot of different seemingly great projects, what is the project that's going to give me the most bang for my buck? And we really don't have that kind of consistent rigorous impact evaluation data in order to make those decisions. And that was really the reason why J-PAL was started, because of the feeling that we could do so much better if we had that kind of data. And it's also too often the case that decisions about development are based on emotion rather than data. You can see this in proposals that people write and the discussions that people have, very compelling, personal stories, which are important, but aren't really what we should be making all our decisions on. That may be very motivating to get people involved. But when you're talking about trade-offs, you've got to have a lot more rigorous evidence. If we had that kind of evidence, we could be a lot more effective with the money that we have. I also think it's true, sometimes people say oh, well you're just talking about passing a dollar and spending it in a slightly more marginally effective way. But what we really need is more money going into poverty really. But arguably, potentially one of the most important ways to get more money to go into poverty relief is to convince people that the money that's going in is actually used effectively. So I don't see these as either/or. Using the money effectively and raising more money, I think, both can come from having more evidence. So it's also important, I think, in a way, to move from what I think is a very damaging and nonconstructive debate between the aid optimists and the aid pessimists. It's a very kind of polarized debate with Jeff Sachs on one side and Bill Easterly on the other. This is a quote from Jeff Sachs: "I've identified the specific investments that are needed"-- from the previous sentence, you know that this is to end poverty-- "found ways to plan and implement them and show that they can be affordable." Now if you think we know everything about development already-- we know what's needed, we know how to implement it-- then, kind of, this is the wrong course for you. But I think most of us would agree that that's slightly overstating how much information we have about how to end poverty. There's a lot more questions out there than that suggests. His argument is, but we have to get people motivated. So we have got to say that we know everything. I don't think we have to be quite that optimistic. On the other hand, I think this is way too pessimistic. After $2.3 trillion over five decades, why the desperate needs of the world's poor still so tragically unmet? Isn't it finally time to end the impunity of foreign aid? So Bill Easterly is kind of saying, oh, it has not worked. So let's throw it all away. We've got to find a middle ground here. It's just about time that we have a development. And they're talking about aid. And I would argue it's much more about development than aid. Aid is only a small fraction of the money that's spent on reducing poverty in developing countries. Development pessimism is just as bad. We've got to think more strategically about not just that all aid is bad or development funding is wasted, but how do we focus the money on the right things. So that's kind of the motivation for what we're doing. But if you think on a very grand scale, but thinking about the objective of evaluation in general, you can think of it as three things. Accountability, did we do what we say we were going to do? And again, this is true of aid agencies, NGOs, government. And did we have a positive impact on people's lives? So those are two different aspects of accountability that evaluation needs to speak to. Evaluation isn't only about accountability though. I think it's very importantly about lesson learning so we do better in the future. And that's about does a particular program work or not, and what's the most effective route to achieve a certain outcome? Are there similarities? Are there lessons that you can learn across projects? Are there similarities about what we're finding in the evaluation of this project and that project? For example, are there are ways that you're learning about how to change people's behavior in health, and agriculture, and education? Are there similarities, sort of underlying principles, that we're learning about that we can use in different contexts? And ultimately, to reduce poverty through more effective programs is the ultimate objective of evaluation. So using that as a framework, what makes a good evaluation? Well, the key thing is it's got to answer an important question. But it's no good if it answers an important question but it answers it badly. It's got to answer it in an unbiased way. What do I mean by that? I mean that it's got to find the truthful answer to the question. And really to do that, you need to have a model or a theory of change about how the project is working so that you can test the different steps in the model. And that's the best way to then learn the most. If we simply say, we test whether the project worked or didn't work, we learned something, but we learn an awful lot more if you have a specific model of how the project is going to work and you test the different steps along the way. Sometimes people say-- this is something that drives me mad in the evaluation literature-- you hear people saying, well, randomized evaluations are a black box. They can tell you whether something works or not, but they can't tell you why. I hope in the next few days, we're going show to you how you design an evaluation that tells you not just whether it works or not at the end, but why and how, and the steps along the way, and design it cleverly so that you learn as much as you possibly can from the evaluation about the fundamental question. And that's about getting the questions right at the beginning. And it's about doing your model correctly and thinking of indicators along the way that are going to allow you to get to all those steps and really understand the theory of change that's happening. The model is going to start with what is it we're trying to do, who are the targets, and what are their needs. So in an evaluation in development, this would often be called a needs assessment. And what are their needs? But then what's the program seeking to change? And looking at precise and individual bits of the program, what's the precise program or part of the program that's being evaluation, so asking very specific questions. So let's look at an example. And all of this we're going to come back to and do in more detail. How do you do a logframe? Again, maybe some of you have done that before. Maybe you haven't. But hopefully you'll learn more about how we think about doing a logframe. So here's an example of an evaluation that looked, a very simple one, does giving textbooks to children in Kenya improve test scores was the evaluation. But what was the need? What was the problem that the program was trying to test? Well poor children in Busia District in Kenya had low learning levels. They also had low incomes. They had few books. That meant that they couldn't take the books home, and that, the theory was, made it hard to learn. So it was hard to learn because they didn't have a book in front of them in class, but also because there were so few, they couldn't take them home and read up more and do exercises at home. So what was the input? The input was that a local NGO bought additional textbooks. In order to get to your long-term goal, you not only need the books, you need to make sure that they're delivered. Because, again, making this chain along the way will help you understand if it doesn't work, where did it go wrong, if the books were bought but they never got there, or they were stuck in the cupboard. How many times have we been to schools and oh, yes, we have lots of books. And we don't want them to get messy when the children are using them. So they're all nicely in their sealed package. Well you need to be able to distinguish if something doesn't work, is it because it's stuck in the cupboard? Or was it because even when the books are out there they didn't get used or they didn't help? So the books are delivered and used. The children use the books and they're able to study better. And finally, the impact, which is what we're about here, is yes, you got all of those steps done. But did it actually change their lives? Did it actually achieve the impact you are hoping to get is high test scores? The long-term goal would be not just high test scores but higher income. And that long-term goal may be very difficult to test in the evaluation. And you may use some other work that may have linked these in previous studies in the same country to make the assumption that if we got higher test scores, it will have a positive impact. So again, that's a decision when you make in your evaluation how far along this chain you go, if it's a process evaluation you may stop here. If it's an impact evaluation you have to stop here. But you may not have enough money to take it all the way through to the finest, finest level that you would like to. Oh, I didn't do my little red triangles at right point. OK. So I've already, in a sense, introduced some of these concepts. But again, let's review them so we know we're talking about the same thing. There are many different kinds of evaluation. And needs assessment is where you go in and look at a population, see what are the issues. How many of them have bed nets? What are test scores at the moment? How many books are there? What's class size? What are the problems in your target population? In our process evaluation, does someone want to tell me what they would see as a process evaluation? We talked a little bit about it. Someone? Yeah. AUDIENCE: [UNINTELLIGIBLE PHRASE] the chain that you just presented to see how you get from the input to the output from the output to the outcome of this RACHEL GLENNERSTER: Right. AUDIENCE: Are we successful in doing that in transforming our input into output? RACHEL GLENNERSTER: Right. So process evaluation looks at did we buy the textbooks? Were they delivered? Were they used? So moving inputs, outputs, outcomes, but stopping short before we get to the impact. And that's a very useful thing to do, and should be done basically everywhere you do a program, or at least some of those steps need to be measured almost every time you do a program. But it kind of stops short before you get to the impact stage. Have we actually changed people's lives? We wanted to build a school. Did we build a school? We wanted to build a bridge. Did we build a bridge? We wanted to deliver things? Did we deliver things? But it's stopping before you get the point of knowing whether this has actually changed people's lives. So an impact evaluation then goes to the next stage and says, given that we have done what we said we're going to do, has that actually changed things? And this is where there was a big gap in terms of what we know. There's a lot of lesson learning you can do from process. But in terms of knowing what kind of project is going to be successful in reducing poverty, you really need to go this next step. Now we used to just talk about those three. But increasingly, as I say as I have more contact outside economics and the more research side of evaluation to work a lot with DFID and other organizations, other foundations, and other agencies, I realize a lot of what people outside the academic community call evaluation I would call review. It's very weird. Because they would often call what I call something else. But what I mean by review is, it's sort of an assessment. It's sending a knowledgeable person in and reviewing the program and giving their comments on it, which can be extremely helpful if you have a good person going and talking to the people involved, and saying, well, in my experience, it could have been done differently. But it doesn't quite actually do any of these things. It's not just focused in did I build the school. But it's asking questions about was there enough participation. How well organized was the NGO? And a lot of this is very subjective. So I'm not saying that this is bad. It's just kind of different. And if you have someone very good doing it, it can be very useful. My concern with it, is that it's very subjective to the person who's going. Yeah. Logan? AUDIENCE: I think that you see so many reviews simply because the way-- you just mentioned DFID. USAID, I think, is the same way. It's all retroactive. The way that contracts are awarded and things like that, usually it's because it's a requirement to evaluate a certain number of programs. And it's not until after the program is actually done that they decide they're going to evaluate it. And it's obviously cheaper to send one person over and do the simple review. I think it would be interesting. We'll probably get to this when we talk about how you can apply some of the randomized control test methodology to something that you're doing retroactively. RACHEL GLENNERSTER: So just to repeat, the argument is we do a lot of reviews because a lot of evaluation is done retroactively. What you can do at that point is very limited. Yes. So this is a big distinction between the kinds of evaluations, is one that's set up beforehand and one that is after the event. We've got this program. We want to know whether it works. Basically it's really hard to do that. You've already kind of shot yourself in the foot if you haven't set it up beforehand. If we think about what I was saying about it's crucial to have a theory of change, a model about what we're trying to achieve and how we're going to try and achieve it, and measuring each of those steps, if you're coming in afterwards, then you're kind of adhoc-ly making up what your theory of change is. And you haven't set up systems to measure those steps along the way, it's going to be very hard to do. And that's exactly why you end up with a lot of reviews. You're in this mess. And so you just send someone knowledgeable and hope they can figure it out. To answer your specific question though, you can't do a randomized evaluation after the event. Because the whole point is you're moving people into treatment and control based on a flip of the coin. And then after the event, people have been allocated to the treatment or not the treatment. And it's very difficult to know afterwards were these people similar beforehand. It's impossible to distinguish. They may look different now. But you don't know whether they look different now because they were different in the beginning or because they're different because of the program. Yeah? AUDIENCE: I was really interested to read the first case study because it seemed that you were applying randomized control methodology. But it seemed to be actually done retroactively. RACHEL GLENNERSTER: No. It wasn't. It might look that way. But it was set up so the first case study uses a lot of different methodologies, compares different methodologies. But they couldn't use all those methodologies if they hadn't designed it as a randomized study at the beginning actually. If you've set up a randomized evaluation, you can always do a non-randomized evaluation of it. But if you haven't done it as a randomized to start with, you can't make it randomized. Prospective evaluation, setting up the evaluation from the beginning, is very important I would say in any methodology you use. But it's impossible to do it. There are a couple of examples where people have done a randomized evaluation afterwards or an evaluation afterwards. And that is because the randomization happened beforehand. But it wasn't done because it was an evaluation. So if you look at the case on the women's empowerment in India, you will do later in the week, that was not set up as an evaluation. It was set up as a randomized program. And the rationale was they wanted to be fair. So where there's limited resources, sometimes governments are the people who randomized in order to be fair to the different participants. Some people, in this case, will have to get a women's leader. In Colombia Project, that you'll find on our website, the Colombian government wanted to provide vouchers to go to private school. But they couldn't afford it for every one. So they randomized who would get them. So that's the one case where you can do a randomized evaluation after the event, when somebody else is randomized beforehand, but they weren't actually thinking of it as an evaluation. But even then, it would've been nice to have data beforehand. So the last thing on this list is cost-benefit analysis, which is something that you can do with the input from all of these other things. As they say, the piece of information that we have so little of is what's the effect of a dollar here versus a dollar here. And you can only do that if that's one of your ultimate objectives when you're doing these other impact evaluations or these other evaluation methodologies. Because you need to be collecting data about costs. And the benefits will come from your impact evaluation. But you need to get your costs from your process evaluation. And you can put the two together. And you can do a cost effectiveness. Then if somebody else has done that in their other study, you can do a cost effectiveness comparison across studies. Or even you can evaluate a range of different options on your impact evaluation. And that will give you comparative cost-effectiveness across. So going into a bit more detail in some of these, needs assessment. We'll look at who's the target population. Is it all children or are we particularly focused on helping the lowest-achieving in the group? What's the nature of the problem being solved? Many of these communities will have lots of problems. So what are we particularly trying to focus on here? Is a test schools? Is it attendance at school? How will textbooks solve the problem? I was talking about textbooks. Maybe it's because they can take them home. Well, if that's part of your model, your theory of change, you need to be actually measuring that, not just did they arrive. How does the service fit into the environment? How many times have we sat in an office and designed something that we thought made complete sense, and gone out to the field and thought, what was I thinking? This isn't going to work. How does it feel for the teachers? Do they understand the new books? Do they know how to teach from them? How do the books fit into the curricula? What are you trying to get out of this? As they say, you want a clear sense of the target population. So then you want to see are the students responding? If you're particularly worried about low-performing kids, are they responding to the textbooks? Students who are falling behind, a sense of the needs the program will fill, what are the teachers lacking? How are we going to deliver the textbooks? How many textbooks are we going to deliver? And what are the potential barriers for people learning from the textbooks? Then a clear articulation of the program benefits, and a sense of alternatives. And as I say, if you want to look at cost-effectiveness of alternative approaches, it's very important to think through not just this program in isolation, but what are the alternatives that we could be doing? And how does that fit with them? Is this one is the most expensive things we're going to try, one of the cheapest things we want to try, and everything in between? So you may be thinking in this context, is this a replicable program that I'm going to be able to do elsewhere? Is this the gold-plated version that I'll do if I get lots of funding? Or is this something that I can replicate in lots of other places? Process evaluation, I've really sort of talked quite a bit about these. So I'm going through them faster. And when you do an impact evaluation, because the impact evaluation is the last thing on that chain, you need to do all the other bits on the chain as well. You can't do an impact evaluation without a process evaluation or you won't understand what the hell your answer is meaning at the end. So as we say, a process evaluation is asking are the services being delivered? Is the money being spent? Are the textbooks reaching the classroom? Are they being used? And it's also, as I say, important to be asking yourself, what are the alternatives. Could you do this in a better way? Just like a company is always thinking are there ways to reduce costs. You should be thinking are their ways to do this more cheaply. Are the services reaching the right populations? Which students are taking them home? Is it the ones that I'm targeting or only the most motivated ones? And also, are the clients satisfied? What's their response to the program? So an impact evaluation, am i missing a top bit here? No. OK. Here we go. We're out of order. So an impact evaluation is, as they say, taking it from there. So assuming once you've got all the processes working, and it's all happening, but if it happens, does it produce an impact? Take our theory of change seriously. And say what we might expect to change if that theory of change is happening. So we've got this theory of change that says, this is how we expect things to change. These are the processes by which we expect, like the kids taking the books home. So we want to design some intermediate indicators and final outcomes that will trace out that model. So our primary focus is going to be, did the textbooks cause children to learn more. But we might also be interested in some distributional issues. Not just on average, we might also be interested in was it the high achieving kids that learned more? Was it the low-achieving kids? Because very often in development, we're just as interested in the distributional implications of a project as the average. So who is it who learned? How does impact differ from process? In the process, we describe what happened. And you can do that from reading documents, interviewing people in administrative records. In an impact question, we need to compare what happened to the people who got the program with what would have happened. This is the fundamental question that Dan is going to hammer on about in his lecture about why do we use randomized evaluations. We talk about this is the counterfactual. What would have happened if the program hadn't happened? That's the fundamental question that we're trying to get at. Obviously it's impossible to know exactly what would have happened if the program hadn't happened. But that's what we're trying to get at. Just one second. Yeah? AUDIENCE: So one thing that would seem to fit in somewhere with the impact thing but doesn't quite meet the criteria that you've just described that we've use sometimes is this pre-post test. And that isn't necessarily going to say what would have happened. But it will say, well, what were the conditions when you started? And we extrapolate from that looking at where we are when we ended, what can we say about the impact of the intervention? RACHEL GLENNERSTER: Right. So that is one way that people often try and do an impact evaluation and measure are they having an impact. And I guess it can give you some sense of whether you're having an impact or flag problems. It's to say, well, what were conditions at the beginning? What are they like now? Then you have this assumption, which is that all the difference between then and now is due to the program. And often that's not a very appropriate assumption. Often things happen. If we take our example of schools, the kids will know more at the end of the year then they knew at the beginning. Well would they have known more even if we hadn't given them more textbooks. Probably. So that's kind of the fundamental assumption you're making. And it's a difficult one to make. It's also the case that we talked to people who were doing a project in Gujarat. And they were tearing their hair out and saying, well, we seem to be doing terribly. Our program is doing terribly. People now are worse off than when we started. This was, well Mark will know the years of the riots and earthquake in Gujarat. They'd basically taken data when they started. In the meantime, there had been a massive earthquake and massive ethnic riots against Muslims in Gujarat. Of course people were worse off. And that's not because of you. So if can go either way actually. You can assume that your program is doing much better because other things are coming along and helping people. And you're attributing all the change to your program. Or it could be the case in this extreme example. There's a massive earthquake and massive religious and ethnic riots and you attribute all the negative to your program. So it's a way that sometimes people use of trying to get an impact. It's not a very accurate way of getting your impact, which is why a randomized evaluation would help. So, as you say, it doesn't quite fit this criteria. Because it doesn't quite answer. It says what happened over the period. It doesn't say what would have happened. It is not a comparison of what would have happened with what actually had. And that's how you want to get at your impact. So there are various ways to get at it. But some of them are more effective than others. So let's go back to our objectives and see if we get a match these different kinds of evaluations to our different objectives for evaluation and find out which evaluation will answer which question. So accountability: the first question for accountability is just did we do what we said we were going to do. Now that you can use a process evaluation to do that. Because did I do what I said I was going to do. I promised to deliver books. Did I actually deliver books? Process evaluation is fine for that kind of level of accountability. If my accountability is not just did I do what I said, but did what I do help? Ultimately I'm there to help people. Am I actually helping people? That's a deeper level of accountability. And that, you can only answer with an impact evaluation. Did I actually make the change that I wanted to happen? If we look at lesson learning, the first kind of lesson learning is, does a particular program work or not work. So an impact evaluation can tell you whether a particular program worked. If you look at different impact evaluations of different programs, you can start saying which ones worked, whether they work in different situations, or whether a particular kind of program works in different situations or not. Now what is the most effective route for achieving a certain outcome is kind of an even deeper level of learning. What kind of thing is the best thing to do in this situation? And there you want to have a cost-benefit analysis comparing several programs based on a number of different impact evaluations. And then we said an even deeper level is, can I understand how we change behavior? Understand deep parameters of what makes a successful program, of how do we change behavior from health to agriculture? What are some similarities and understanding of how people tick, and how we can use that to design better programs? And again, that's linking our results back to theories. You have got to have a deeper theory understanding it, and then test that with different impact evaluations. And you can get some kind of general lessons from looking across impact evaluations. And then if we want to have our reduced poverty through more effective programs, which is our ultimate objective of doing evaluations, we've got to say, did we learn from our impact evaluations? Because if we don't learn from them and change our programs as a result, then we're not going to achieve that. And I guess to say that solid, reliable impact evaluations are a building block. You're not going to get everything out of one impact evaluation. But if you build up enough, you can generate the general lessons that you need to do that. I've said quite a lot of this already. But needs assessments give you the metric for defining the cost-benefit ratio. So when we're looking at cost-benefit analysis, we're looking at what's the most cost-effective way of achieving x? Well, you need a needs assessment to say what's the x? What's the thing that I should be really trying to solve? Process evaluation gives you the costs for your inputs to do a cost-benefit analysis. And an impact evaluation tells you the benefit. So all of these different inputs and needed to be able to do an effective cost-benefit analysis. AUDIENCE: Rachel? RACHEL GLENNERSTER: Yeah? AUDIENCE: The needs assessment seems to be more of a program design sort of a [UNINTELLIGIBLE], whereas the remaining three are more like the program has already been designed and we are being cautious, we have thought that this is the right program to go with. Please design a process evaluation for this or a program evaluation for this. How has that needs assessment different from the one that feeds into program design? RACHEL GLENNERSTER: Well, in a sense, there's two different concepts here. You're right. There's a needs assessment for a particular project. We're working with an NGO in India called [UNINTELLIGIBLE] working in rural Rajasthan. And they said, we want to do more on health in our communities. We've done a lot of education and community building. But we want to do a lot more in health. But before we start, we want to know what are the health problems in this community? It doesn't make sense to design the project until you know. So we went in and did., what are the health problems? What's the level of services? Who are they getting their health from? We did a very comprehensive analysis of the issues. And that was a needs assessment for that particular NGO in that particular area. But you can kind of think of that in a wider context of saying, what are the key problems in health in India or in developing countries? What are the top priority things that we should be focusing on? Because again-- and I'm going to get on to strategy in a minute-- if you're thinking as an organization, you can't do an impact evaluation for everything. You can't look at comparative cost-effectiveness for outcomes in the world. Or at least you've got to start somewhere. You've got to start on, what do I most want to know? What's the main thing I want to change to see what's the cost of changing that thing? So is it test scores in schools? Or is it attendance? Am I most concerned about improving attendance? If you look at the Millennium Development Goals, in the sense, that's the world's prioritizing. They're saying, these are the things that I most want to change in the world. And there they made the decision, rightly or wrongly, on education, that they wanted to get kids in school. And there isn't anything about actually learning. And whether your needs is getting kids in school or learning, you would design very different projects. But you would also design different impact evaluations, because those are two very different questions. So the needs assessment is telling you what are the problems? What am I prioritizing for my programs, but also for my impact evaluations? Yeah? AUDIENCE: Do you need to make a decision early on whether you're interested in actually doing a cost-effectiveness analysis as opposed to a cost-benefit. RACHEL GLENNERSTER: [INTERPOSING VOICE] AUDIENCE: [UNINTELLIGIBLE PHRASE]. efficiency measure, where as cost [UNINTELLIGIBLE}-- RACHEL GLENNERSTER: So I'm kind of using those too easily interchangeably. I don't think it's so important here. How would you define the difference between them? AUDIENCE: As I understand it, but [UNINTELLIGIBLE PHRASE] they getting a better answer. But cost-effectiveness is a productivity measure. And it would mean that you would have to, in an evaluation say, OK, I'm going to look at I put one buck into this program and I get how many more days of schooling out of it. Right? RACHEL GLENNERSTER: Right. AUDIENCE: Whereas cost-benefit requires that it all be in dollars or some other [UNINTELLIGIBLE]. RACHEL GLENNERSTER: So you've got to change your benefit into dollars. So I'll give you an example of the difference. AUDIENCE: Like [INAUDIBLE PHRASE]. RACHEL GLENNERSTER: Let's make sure everybody's following this discussion. A cost-effectiveness question would be to say, I want to increase the number of kids in school. How much would it cost to get an additional year of schooling from all of these different programs? And I'm just assuming that getting kids in school is a good thing to do. Right? I want to do it. So I'm asking what's the cost per additional year of schooling from conditional cash transfer, from making it cheaper to go to school by giving free school uniforms, or providing school meals. There are many different things I could that will encourage children to come to school. But I know I want children to come to school. I'm not questioning that goal. So I just want to know the cost of getting a child in school. Cost-benefit kind of squishes is it all. And it really asks the question, is it worth getting kids in school? Because then you can say, if I get kids in school, they will earn more and that will generate income. So if I put a dollar in, am I going to get more than a dollar out at the end? I'm not going to flick all the way back to it. But if you remember that chart that went through the process and impact, and then the final thing of high test scores was higher income, ultimately am I getting more money out of it than I'm putting in? I think that's sort of a philosophical decision for the organization to make. It's very convincing to be able to say, for every dollar we put in, I think for the deworming case that you've got and you do later in the week, they do both, cost-effectiveness and cost-benefit. And the cost-effectiveness says, this is the most cost-effective way to get children in school. But they also then go further and say, assuming that these studies that look at children in school in Kenya earn higher incomes are correct, then given how much it costs to get an additional year of schooling, and given an assumption about how much extra kids will in the future because they went to school, then for every dollar we put in, I think it's you get $30 back. So you kind of have to make an awful lot more assumptions. You have to go to that final thing and put everything on income. Now if I was doing the women's empowerment study, then I'm not sure that I would want to reduce women's empowerment to dollars. I might just care about it. I might care that women are more empowered whether or not it actually leads to higher incomes. So it kind of depends on the argument that you're making. If you want to try and make a this is really worth it, this is a great program, not just because it's more effective than another program, but that it generates more income then I'm putting in. That's a great motivation. But I wouldn't say you always have to reduce it to dollars. Because you have to make an awful lot of assumptions. And we don't necessarily always want to reduce everything to dollars. So here it is. We've just been talking about it. So this is a cost-effectiveness. So this is the cost per additional year of schooling induced. We're not linking it back to dollars we're measuring. We're just assuming that we want kids in school. Millennium Development Goals have it as a goal. we just think it's a good thing, whether or not it generates income. What we did is take all the randomized impact evaluations that had as an outcome getting more children in school and calculated the cost per additional year of schooling the resulted. So you see a very wide range of different things. Now conditional cash transfers turn out to be by far the most expensive way of getting an additional year of schooling. Now that's partly because mainly they're done in Latin America where enrollment rates are already very high. So it's often more expensive to get the last kid in school than the 50th percentile kid in school. And then the other thing, of course, in general, things cost more in Mexico than in Kenya, especially when you're talking about people. Teacher's wages or wages outside of school are more expensive. But the thing that was amazing was that providing children with deworming tablets was just unbelievably cost-effective. So $3.50 for an additional year of schooling induced. And putting it this way I think really brought out that difference. The other thing I should say in comparing this is, there were other benefits to these programs. So Progresa actually gave people cash as well. So it wasn't just about getting kids in school. So of course it was expensive, right? And we haven't calculated in those costs. In cost-benefit, if we reduced everything to dollars, it would look very different because you've got a value of all these other benefits. But again, deworming had other benefits. It had health benefits as well as education benefits. So we're just looking at one measure of outcomes here. AUDIENCE: Excuse me. RACHEL GLENNERSTER: Yeah? AUDIENCE: Are these being adjusted for Purchasing Power Parity, PPP? RACHEL GLENNERSTER: So this is not PPP. This is absolute. So again, we've sort of debated it backwards and forwards. So if you're a country, you care more about PPP. But if you're a donor and you're wondering whether to send a dollar or a pound to Mexico or Kenya, you don't care about PPP. You care about where your dollar is going to get most kids in school. So there's different ways of thinking about it. It sort of depends on the question you're asking and who's the person. For a donor, I think this is the relevant way. If you're a donor who only cares about getting kids in school, this is what you care about. We also can redo this taking out the transfers. There's this other benefit, the families, of getting the money. So this is the cost to a donor. So that's one way of presenting it. But you can present it in other ways too. AUDIENCE: Can you also sometimes do a cost-benefit of the evaluation itself? RACHEL GLENNERSTER: That's kind of hard to do because the benefits may come ten years later. The way to think about that is to think about who's going to use it, and only do it if you think it's going to actually have some benefits in terms of being used and not just maybe within the organization. But if it's expensive, is it going to be useful for other people? Is it answering a general question that lots of people will find useful? So often evaluations are expensive in the context of a particular program. But they're answering a question the lots of other people will benefit from. So the Progresa evaluation has spurred not just the expansion of Progresa in Mexico, but it has spurred it in many other countries as well because it did prove very effective. Although it's slightly less cost-effective in these terms. But it led to an awful lot of learning in many, many other countries. So, I think, in that sense, it was an extremely effective program evaluation. AUDIENCE: Excuse me, I just have a question on that very last item there. RACHEL GLENNERSTER: OK, so this one is even cheaper and it's a relatively new result. But it only works in certain circumstances. When people don't know the benefits of staying on in school, ie., how much higher wages they're going to get if they have a primary education, then telling them that information is very cheap. And both in the Dominican Republic and Madagascar-- so two completely different contexts, different rates of staying on in school, different continents, very different schooling systems-- in both cases it was extremely effective at increasing the number of kids staying on in school. But that only works if people are underestimating the returns of staying on in school. If they're overestimating them, then it would reduce staying on in school or if they know already, then it's not going to be effective. So this is something that I think is a very interesting thing ti do, and again, is worth doing. But you need to first go in and test whether people know what the benefits of staying on in school are. Basically they just told them what's the wage if you complete primary education versus what's the wage if you don't complete primary education. It's very cheap. So if it changes anything, it's incredibly effective. AUDIENCE: Is the issue of marginal returns a problem? Do you have to say that every program is only relevant to places where it's at same level of enrollment or admission? RACHEL GLENNERSTER: Well this is a sort of wider question of external validity. When we do a randomized evaluation, we look at what's the impact of a project in that situation. Now at least you know whether it worked in that situation, which is better than not really knowing whether it worked in that situation. Then you've got to make a decision about whether you think that is useful to another situation. A great way of doing that is to test it in a couple of different places. So again, this was tested in two very different situations. The deworming had very similar effects. In rural primary schools in Kenya, it works through reducing anemia. Reducing anemia in preschool urban India had almost identical effects. Getting rid of worms in a non-randomized evaluation to be true, but kind of a really nicely designed one in the south of the United States had almost exactly the same effect. So they got rid of hookworm in the 1900s. And again, it would increase school attendance, increase test scores, and actually increase wages just from getting rid of hookworm. And they reckoned a quite substantial percentage. This paper by Hoyt Bleakley at Chicago found that quite a substantial difference in the income of the North and the South of United States in 1900 was simply due to hookworm. So this is being tested. So ideally you test something in very different environments. But you also think about whether it makes sense that it replicates. So if I take the findings of the women's empowerment study in India where it works through local governance bodies that are quite active in India and have quite a lot of power, and tried to replicate in Bangladesh where there is no equivalent system, I would worry about it. Whereas worms cause anemia around the world. And anemia causes you to be tired. And being tired is likely to affect you going to school. That's something that seems like it would replicate. So you have to think through these things and ideally test them. If I'm doing microfinance, would I assume it has identical effects in Africa, or Asia, and Latin American? No. Because it's very dependent on what are the learning opportunities in those environments. And they're likely to be very different. So I'd want to test it in those different environments. We're falling a bit behind. So I promised to do when to do an impact evaluation. So there are important questions you need to know the answer to. So that might be because there's a program that you do in lots of places, and you have no idea whether it works. That would be a reason to do one. You're very uncertain about which strategy to use to solve a problem. Or there are key questions that underline a lot of your programs, for example, adding beneficiary control, having some participatory element to your program. It might be something that you do in lots of different programs when you don't know what's the best way to do it or whether it's being effective. An opportunity to do it is when you're rolling out a big new program. And you're going to invest an awful lot of money in this program, you want to know whether it works. This is a tricky one. You're developing a new program and you want to scale it up. At what point in that process should you do the impact evaluation? Well you don't want to do it once you've scaled it up for everywhere. Because then you find out it doesn't work, and you've just spend millions of dollars scaling it up. Well that's not a good idea. On the other hand, you don't want to do it when it's your very first designs. Because often it changes an awful lot in the first couple of years as your tweaking it, and developing it, and understanding how to make it work on the ground. So you want to wait until you've got the basic kinks ironed out. But you want to do it before you scale it up too far. We've done a lot of work with this NGO in India called Pratham. And we started doing some work for them. And by the time we finished doing an evaluation, their program had completely changed. So we kind of did another one. So we probably did that one a little bit too early. But on the other hand, now they're scaling up massively. And it would be silly to wait until they'd done the whole of India before we evaluated it. AUDIENCE: You said it may be more appropriate to do a process evaluation initially to get a program to the point where it can be fully implemented and all the kinks are worked out. RACHEL GLENNERSTER: Yeah, exactly. If we're going back to our textbook example again, you don't want to be doing it until you've got your delivery system for the textbooks worked out, and you've made sure you've got the right textbook. It's a bit of a waste of money until you've got those things. And exactly, a process evaluation can tell you whether you've got those things working. The other thing that makes it a good time or a good program to do an impact evaluation of is one that's representative and not gold-plated. Because if Millennium Development Villages, $1 million per village. If we find that that has an impact on people's lives, that's great. But what do we do with that? We can't give $1 million to every village in Africa. So it's not quite, what's the point? But it's less useful than testing something that you could replicate across the whole of Africa, that you have enough money to replicate in a big scale. So that's interesting because you can use it more. Because if you throw everything at a community, yes, you can probably change things. But what are you learning from it? So it takes time, and expertise, and money to do it right. So it's very important to think about when you're going to do it and designing the right evaluation to answer the right question that you're going to learn from. AUDIENCE: If a program hasn't been successful, have you found that the NGO's have abandoned that program? RACHEL GLENNERSTER: Yes, mainly. We worked with an NGO in Kenya that didn't work. They just moved on to something else. Pratham, we actually did two things, both of which worked, but one which was more cost-effective than the other. And they dumped the computer assisted learning even though it was like phenomenally successful. But the other one was even cheaper. So they really scaled that up. And they haven't really done computer assisted learning even though it had a very big effect on math test scores. And compared to anybody else doing education, it was very cost-effective But compared to their other approach, which was even more cost-effective, they were like, OK. We'll do the one that's most cost-effective. Now there are some organizations that kind of do one thing. And it's much harder for them to stop doing that one thing if you find it doesn't work. They tend to think, well, how can I adapt it? But these organizations that do many things are often very happy to, OK, that didn't work. We'll go this direction. So we want to develop an evaluation strategy to help us prioritize what evaluations to do when. So the first thing to do is step back and ask, what are the key questions for your organization? What are the things that I really, really need to know? What are the things that would make me be more successful, that I'm spending lots of money on but I don't know the answer, or some of these more fundamental questions, as they say, about how do I get beneficiary control across my different programs. The other key thing is you're not going to be able to answer all of them by your own impact evaluations. And as they say, it's expensive to do them. So the first thing to do is to go out and see if somebody else has done a really good impact evaluation that's relevant to you to answer your questions already. Or half answer or more gives you the hypotheses to look at. How many can I answer just from improved process? Because if my problems are about logistics, and getting things to people, and getting cooperation from people, then I can get that from process evaluation. So from that you can select your top priority questions for an impact evaluation and establish a plan for answering them. So then you've go to look for opportunities where you can develop an impact evaluation that will enable you to answer those questions. So am I rolling out a new program in a new area? And I can do an impact evaluation there. Or you might even want to say, I want to set up an experimental site. I don't really know whether to go this way or that way. So I'm just going to take a place and try different things. And it's not going to be really part of my general rollout But I'm going to focus in on the questions. Should I be charging for this or not? How much should I charge? Or how should I present this to people? And you can take a site and kind of try a bunch of different things against each other, figure out your design, really hone it down, and then roll that out. So those are two kinds of different options of thinking about how to do it. And then, when you've got those key questions of your impact, you can combine that with process evaluations to get your global impact. What do I mean by that? Let's go back to our textbook example. If you're giving out textbooks across many states or throughout the country, you've evaluated it carefully in one region. And you find that the impact on test scores is whatever it is. And then you know very carefully, and maybe you've tested it in two different locations in the country and you've got very similar results. So then you can say, well I know that every time I give a textbook, I get this impact on test schools. Then from the process evaluation, you know how many textbooks are getting in the hands of kids. Then you can combine the two, multiply up your impact numbers by the number of textbooks you give out. Malaria control with bed nets, if I hand out this many bed nets, then I'm saving this many lives. I've done that through a careful impact evaluation. And then all I need to do is just count the number of bed nets that are getting to people and I know my overall impact. So that's a way that you can combine the two. You don't have to do an impact evaluation for every single bed net you hand out. Because you've really got the underlying evaluation impact model, and you can extrapolate. AUDIENCE: Rachel? RACHEL GLENNERSTER: Yeah? AUDIENCE: Do you think in the beginning when you got a program that you're interested in, do you think that's the moment to think about the size of the impact that you're looking at that people expect? And also, as part of that, what's going to be the audience, the ultimate audience that you're trying to get to if you're successful with a scale-up. And those two things, I think, frequently come together. Because it's the scaling up process where people are going to start to look at those cost-effectiveness measures and cost-benefit. RACHEL GLENNERSTER: I mean, I would argue that you've always got to be thinking about your ultimate plans for scaling it up when you're designing the project. Because you design a project very differently if you're just trying to treat a small area than if you're thinking about, if I get this right, I want to do it on a much wider area. If you've always got that in mind, you're thinking a lot about is this scalable? Am I using a resource that is either money or expertise that is in very short supply, in which case there's no point in designing it this way because I won't able to scale it beyond this small study area. So if that's your ultimate objective, you need to be putting that into the impact evaluation from the moment. Because there's no point in doing the impact evaluation, very resource-intensive project, and say, well, that works. But I can't do that everywhere. Well then what have you learned? You want to be testing the thing that ultimately you're going to be able to bring everywhere. So in a lot of our cases, we're encouraging our partners to scale it back. Because you won't be able to do this on a big scale. So scale it back to what you would actually be doing if you're trying to do the whole of India or the whole of this state. Because that's what's useful to learn. And you want to be able to sell to someone to finance the scale-up. So I think having those ideas in your mind at the beginning is very important, and as they say, making it into a strategy, not a project by project evaluation, but thinking about where do I want to go as an organization. What's the evidence I need to get there, and then designing the impact evaluations to get you that evidence. And people often ask me about how do you make sure that people use the evidence from impact evaluations. And I think the main answer to that is ask the right question. Because it's not about browbeating people to make them read studies afterwards. If you find the answer to an interesting question, it'll take off like wildfire. It will be used. But if you answer a stupid question, then nobody is going to want to read your results. So we're learning from an impact evaluation, so learning from in a single study did the program work in this context? Should we expand it to a similar population? Learning from an accumulation of studies, which is what we want to get to eventually, is did the same program work in a range of different contexts, India, Kenya, south of the United States? And that's incredibly valuable because then your learning is much wider and you can take it to many more places. Did some variation in the same program work differently, ie., take one program and try different variants of it and test it out so that we know how to design it. Did this same mechanism seem to be present in different areas? So there's a lot of studies looking at the impact of user fees in education and health. You seem to get some very similar results. And again, that's even more useful. Because then you're not just talking about moving deworming to another country. You're talking about user fees. What have we learned about user fees across a lot of different sectors? There's some common understandings and learnings to take to even a sector that we may not have studied before. And then, as they say, putting these learnings in the place, in filling in an overall strategy of what were my gaps in knowledge? And am I slowly filling them in? So, I think that's it. So I'm sorry the last bit was a little bit rushed. The idea was to kind of motivate why we're doing all of this. Today you're going to be in your groups. The task for your groups today, as well as doing the case, is to decide on a question for an evaluation that you're going to design over the next five days. So hopefully that's made you think about what's an interesting question. What should we be testing? Because I think often an underlooked element of designing an evaluation is what's the question that we want to be answering with this evaluation? Is it a useful question? How am I going to use it? What's it going to tell me for making my program, my whole organization more effective in the future? So any questions? AUDIENCE: What would you say are some of the main limitations of randomization? So I assume one of them is extrapolate the populations that are different? Are there other main ones that you can think of? RACHEL GLENNERSTER: So it's important to distinguish when we talk about limitations. One is just general, what's the limitation to say, extrapolating beyond? But the other thing is to think of it in the context of what's the limitation versus other mechanisms? Because, for example, the extrapolating to other populations is not really a limitation of randomized evaluations compared to any other impact. Any impact evaluation is done on a particular population. And so there's always a question as to whether it generalizes to another population. And the way to deal with that is to design it in a way that you learn as much as you possibly can about the mechanisms, about the routes through which it worked. And then you can ask yourself when you bring it to another population, do those routes seem like they might be applicable, or is there an obvious gap? This worked through the local organization. But if that organization isn't there, is there another organization that it could work through there or not? If you think that deworming works through the mechanism of anemia, well it works between the mechanism of there being worms and of anemia, you can go out and test. Are there worms in the area? Is the population anemic because there may be worms and they're not anemic. So that's a way to design the evaluation to limit that limitation or reduce the problem of that limitation. But it's not like the very active flipping a coin and randomizing causes the problem of not being able to extend it. It's true of any impact evaluation. I think one limitation which you will find in your frustration as you want to try and answer every single question that you have, and you get into the mechanics of sample size and how much sample size do I need-- and again, that's not necessarily just of a randomized evaluation, but any quantitative evaluation-- you can test a limited number of hypotheses. And every hypothesis you want to test needs more sample. And so the number of questions you can answer very rigorously is limited. And I think that's the limitation that we often find very binding. Again, any rigorous quantitative evaluation will have that limitation. We'll talk a lot tomorrow about sometimes you just can't randomize. Freedom of the press is not something that you can randomize except by country. And then we'd need every country in the world. It's just not going to happen. So we'll look at a lot of new techniques or different techniques that you can use to bring randomization to areas where you think it would be impossible to bring it to. Compared to other quantitative evaluations, you sometimes have political constraints about where you can randomize. But as I say, quantitative versus qualitative, the qualitative isn't so limited by sample size constraints. And you're not so limited to answer very specific hypotheses. The flip side is you don't answer any specific hypotheses. And it's the same rigorous way. But it's much more open. So very often what we do is we combine a qualitative and quantitative, and spend a lot of time doing qualitative before to hone our hypotheses, and then use a randomized impact evaluation to test those specific. But if you sit in your office and design your hypotheses without any going out into the field, you will almost certainly waste your money because you won't have asked the right question. You won't have designed it. So you need some element of qualitative to make sure some needs assessment, some work on the ground to make sure that you are asking, you're designing your hypotheses correctly because you've only got a few shots. Yeah? AUDIENCE: I was wondering, do you know of some good evaluation or randomized impact evaluation on conservation programs? RACHEL GLENNERSTER: On conservation programs? I can't think of any, I'm afraid, but eminently doable. But we can talk about that if you can persuade your group to think about designing one. Anyone else think of a conservation program? Yes? AUDIENCE: I don't have an example. I wish I did. And you've mentioned this. I just need to really underline it for myself. A lot of the programs that my organization does are comprehensive in nature. So they have lots of different elements meant to in the end, collectively, [UNINTELLIGIBLE PHRASE]. What I'm understanding here is that you could do an impact evaluation of all of those collectively. But really it would be more useful to pull them out and look at the different interventions side by side or something. Because that way you'll get a more targeted-- RACHEL GLENNERSTER: It's true. The question was, if you have a big package of programs that does lots of things, you can do an evaluation of the whole package and see whether it works as a package. But in terms of learning about how you should design future programs, you would probably learn more by trying to tease out, take one away, or try them separately. Because there might be elements of the package that are very expensive but are not generating as much benefit as they are cost. And you would get more effect by doing a smaller package in more places. You don't know unless you take the package apart. Now then if you test each one individually, that's a very expensive process. Because it needs a lot of sample size to test each individually. There's also a very interesting hypothesis that's true in lots of different areas. People often feel, where there are lots of barriers, so we have to attack all of them. It only makes sense. You won't get any movement unless you do. There are lots of things stopping kids going to school. There's stopping, say, girls going to school. They're needed a home. There are attitudes. There is their own health. There's maybe they are sick a lot. So we have to address all of those if we're going to have an impact. We don't know, the answer is. And indeed, in that example where we're working with Save the Children in Bangladesh, they had this comprehensive approach. Where there are all these problems. So let's tackle them all. We convinced them to divide it up a bit, and test different things, and see some of their own worked, or whether you needed to do all of them together before you changed anything, which is a perfectly possible hypotheses and one that a lot of people have, but hasn't really been tested. The idea that you've got to get over a critical threshold. And you've got to build up to it. And only once you're over there do you see any movement. Well actually on girls going to school, it's quite interesting. Most of the evaluations that have looked at, just generally, improving attendance at school, have had their biggest impact on girls. I should say most of those were not done in the toughest environments for girls going to school. They're not in Afghanistan or somewhere where it's particularly difficult. But it is interesting. Just general things and approaches in Africa and India have had their biggest impacts on girls, which suggests that you've got a hit every possible thing is maybe not right. Yeah? AUDIENCE: [INAUDIBLE PHRASE] and the political constraints [INAUDIBLE PHRASE]. RACHEL GLENNERSTER: Right. So we'll talk actually tomorrow quite a lot about the politics of introducing an evaluation or at least the different ways that you can introduce randomization to make it more politically acceptable. That's slightly different from whether the senior political figures want to know whether the program works or are willing to fund an evaluation. I've actually been amazingly surprised. Obviously we find that some places. There are certain partners or people we've started talking with. And you can see the moment the penny drops that they're not going to have any control. Because you're going to do a treatment comparison. You're going to stand back. At the end of the day the results going to be that. There's no fiddling with it, which is one of the beauties of the design. But it will be what it will be, which is kind of why it's convincing. But there are certain groups who kind of figure that out. And they run for the exit because there's going to be an MIT stamp of approval evaluation potentially saying their program doesn't work. That's life. Some people don't want to know. The best thing I can say in that situation is test alternatives. It's much less threatening to test alternatives. Because there's always some alternative of this versus that, that people don't know. And then you're not raising the does it work. You're saying well, does this work better than that? And that is much less threatening. It doesn't tell you quite as much. But it's much less threatening. There's a report called When Will We Ever Learn, looking at the politics of why don't we have more impact evaluations, which was very pessimistic. But if you look at somewhere like the World Bank that just put a purse of money for doing randomized impact evaluations out there. And anybody in the bank could apply. And people were like why? There's no incentives for them to do it. Program office's, they've already got a lot on their plate. Why would they add doing this? It's going to find out that they're opening themselves to all these risks because maybe their program doesn't work. Massively oversubscribed, first year, six times more applicants then there was money. It just came out of the woodwork as soon as there was some money to do it. So I'm not saying every organization is like that. Obviously not everybody in their bank did that. But it was, to me, actually quite surprising how many people were willing to come forward. Now we have the luxury of working with the willing, which if you're working within an organization, you don't necessarily have that luxury. You will see as you get into the details of these things, that you need absolutely full cooperation and complete dedication on the part of the practitioners who were doing these evaluations alongside the evaluators. You can't do this with a partner who doesn't want to be evaluated. It just doesn't work. They are so able to throw monkey wrenches in there if they don't want to find out the answer. Then it's just not worth doing it because it's a partnership like that. It's not someone coming along afterwards and interviewing. It is the practitioners and the evaluators working hand in hand throughout the whole process. And therefore if the practitioners don't want to be evaluated, there's not a hope in hell of getting a result. We should wrap up. A lot of these things we're going to talk about. But I'll take one more. Yeah? AUDIENCE: How important, or how relevant is it, or how much skepticism can there be about a case where the evaluators and the practitioners work for the same people or are funded by the same people? RACHEL GLENNERSTER: Yeah, we've even got practitioners as co-authors on our studies. This is another place where I kind of part company from the classic evaluation guidelines, which say that it's very important to be independent. I'd argue if your methodology is independent, what you want is not independence. You want objectivity. And the methodology of a randomized evaluation can provide you the objectivity. And therefore you don't have to worry about independence. Now there's one caveat to that. The beauty of the design is you set it up, as I say. Well you don't stand back in the sense. You've got to manage all your threats and things. But you can't fiddle very much with it at the end. The one exception to that is that you can look at subgroups. So there was an evaluation in UK of a welfare program. And it was a randomized evaluation. And there was some complaining. Because at the end, they went through and looked at every ethnic minority. and then you know I can't remember whether it did work in general. But it didn't work for one minority, or it didn't work. But anyway, you can find one subgroup for whom the result was flipped. And that was the thing on the front page of the newspapers, rather than the overall effect. So there's a way to deal with that, which is increasingly being stressed by people who are kind of looking over the shoulder and making sure that what is done in randomized evaluations is done properly, which is to say that you need to set out in advance-- we'll talk about this a bit later on-- but you need to set out in advance what you're going to do. So if you want to look at a subgroup like does it affect the lowest performing kids in the school differently from the highest performing kids-- do I care most about the lowest performing kid-- if you want to do that, you need to say you're going to do that before you actually look at the numbers. Because even with a randomized evaluation, you can data mine to some extent. Well if I look at least ten kids, does it work for them? If I look at least ten kids, does it work for them? Statistically you will be able to find some subset of your sample for whom it does work. So you can't just keep trying 100 different subgroups. Because eventually it will work for one of them. So on the whole, you need to look at the main average effect. What's the average effect for the whole sample? If you are particularly interested in a special group within the whole sample, you need to say that before you start looking at the data. So that's the only way in which you get to fiddle with the results. And otherwise it provides an enormous amount of objectivity in the methodology. And therefore, you don't have to worry so much about a Chinese wall between the evaluators and the practitioners, which, I think, is incredibly important. Because we couldn't do the work that we do if we had that Chinese wall. It just wouldn't make sense, doing your theory of change, finding out how it's working, designing it so it asks the right questions. None of that would be possible if you had wall between you. So it just wouldn't be anything like as useful. So getting your objectivity from the methodology allows you to be very integrated with the evaluation, and practitioners to be very integrated.