Episode 82: Kirabo Jackson
Kirabo Jackson
Kirabo Jackson is the Abraham Harris Professor of Education and Social Policy at Northwestern University.
Date: November 8, 2022
A transcript of this episode is available here.
Episode Details:
In this episode, we discuss Prof. Jackson's work on the multidimensional aspects of school quality:
“What is a Good School, and Can Parents Tell? Evidence on the Multidimensionality of School Output” by Diether Beuerman, C. Kirabo Jackson, Laia Navarro-Sola, and Francisco Pardo.
OTHER RESEARCH WE DISCUSS IN THIS EPISODE:
“Do Parents Value School Effectiveness?” by Atila Abdulkadiroglu, Parag Pathak, Jonathan Schellenberg, and Christopher Walters.
“Why Do Households Leave School Value Added 'on the Table'? The Roles of Information and Preferences” by Robert Ainsworth, Rajeev Dehejia, Cristian Pop-Eleches, and Miguel Urquiola.
“Smart Matching Platforms and Heterogeneous Beliefs in Centralized School Choice” by Felipe Arteaga, Adam J Kapor, Christopher A Neilson, and Seth D Zimmerman.
TRANSCRIPT OF THIS EPISODE:
Jennifer [00:00:08] Hello and welcome to Probable Causation, a show about law, economics and crime. I'm your host, Jennifer Doleac of Texas A&M University, where I'm an economics professor and the director of the Justice Tech Lab. My guest this week is Kirabo Jackson. Bo is the Abraham Harris Professor of education and social policy at Northwestern University. Bo, welcome to the show.
Kirabo [00:00:28] Thank you so much and good to be here.
Jennifer [00:00:30] Today, we're going to talk about your research on school quality, including how schools affect students criminal behavior. But before we get into that, could you tell us about your research expertize and how you became interested in this topic?
Kirabo [00:00:42] So I am an economist by training, and most of my research is on the economics of education. I sort of became interested in this topic mainly because, you know, most of the things that I have been reading in the area, in the literature are looking at the effects of the various interventions, be they schools, school quality, teacher quality, you know, school readiness programs most of them, or say almost all of them used test scores as a measure of quality. So I sort of became interested in this idea that, well, maybe there are other things that we're not measuring that are actually important outputs of the education system also. So I got to thinking about things like non-cognitive skills, social skills, and also sort of behaviors that matter a lot too, such as teen pregnancy or criminal activity that are not necessarily well measured by how well someone performs on tests, but may nonetheless be important outputs of the educational system. And that's sort of what got me interested in this topic.
Jennifer [00:01:36] So your paper is titled "What Is a Good School? And Can Parents Tell? Evidence on the Multi Dimensionality of School Output." It's coauthored with Diether Beuermann, Laia Navarro-Sola and Francisco Pardo, and it's forthcoming in the review of Economic Studies. So in it, as you say, you consider various dimensions of school quality. So say a little bit more about what researchers usually mean when they try to identify good schools and why you think it's important to consider this broader set of outcomes.
Kirabo [00:02:04] So the typical way that researchers would identify a school quality would be to say some sort of outcome. So there are two ways to think about school quality. I think one is, I should say sort of like an inputs based way to think about school quality, which is sort of the way that we typically use it, do it before we could measure outcomes for students. And I think now in the policy world, we've moved away from sort of an inputs based measure which be looking at things like, oh, this, look at class sizes and see what teachers are doing towards an output based metric, which is to say, let's look at how students are performing. So within this sort of more output based way of looking at things. The typical approach is what we call value added, where you would say, look at outcomes at the beginning of the school year that measures kind of what students know at the beginning of the year and you look at outcomes at the end of the school year that measures kind of their stock of knowledge at the end of the school year look at the difference between those two things. And that's a measure of how much students have learned over the course of that year when they're exposed to a particular school or particular teacher. Let's say that we're interested in, in schools, you might say, well, a so a school that is high quality would be one that raises test scores a lot from the beginning to the end of the year compared to a school that's not very good to be, one that doesn't raise test scores very much.
Kirabo [00:03:15] So we do something somewhat more sophisticated in that we account for a lot of other characteristics that may differ across schools. You might account for the composition of the student in terms of their demographics, the socioeconomic background, other things that may differ across students. And the idea is that once you account for all these different factors, you can sort of compare different schools that have different impacts on students test score growth over time and the way that people would typically define school quality would be a school that has the largest sort of adjusted school test score growth over time. So for many people, school quality and test score or impacts would be synonymous.
Jennifer [00:03:55] And so you're going to look at this broader set of outcomes. And so what does that focus on test scores miss? Why might outcomes like test scores and dropout and arrests and employment not be highly correlated? Do you have examples in mind of why some schools might be better at affecting some of those outcomes but not others?
Kirabo [00:04:17] Yeah, this is a great question. So like I said before, the sort of the narrow view that impacts on test scores is analogous to school quality. I think misses potentially misses a lot of important things that schools may be doing. So without getting into specifics at the moment, you might think that certain schools may be very good at reducing certain kinds of behaviors, such as criminal behaviors among the students that attend the schools or certain schools may be very good at getting kids really, really motivated to learn, but they may not necessarily be doing so well in terms of getting them to do well on these standardized achievement tests. So there are a lot of things that we might care about as a society that are not well measured by test scores.
Kirabo [00:04:55] Same things goes in terms of labor market attachment, you may be schools that are preparing students, giving them the skills that are going to be helpful for them to hold down a job, but that may not necessarily be the same set of skills that are measured by standardized tests. So you might wonder, well, why would it be the case that there are some schools that are raising test scores but are not having an outsized effect on, say, criminal behavior or reducing teen pregnancy or promoting labor market participation? And there are few things you can imagine. So one example could be there may be some some kinds of schools. And there's anecdotal evidence that this does happen to some extent. There may be some schools that are oriented towards academics. So there may be schools that admit students and their main goal is to make sure the kids at that school go on to higher education and go to elite higher educational institutions.
Kirabo [00:05:41] You might imagine that those are exactly the kinds of schools where a kid who may have some underlying predisposition towards acting out or bad behavior may be ignored and may not be taken well taken care of. So you can imagine a setting where a school that is really oriented towards getting kids towards the sort of high, high achievement track and making sure they're doing well academically may actually not be paying as much attention to the kids who are not as strong on the academics, but could really benefit a lot from a lot of attention towards their social emotional skills and looking at their behaviors and promoting better behaviors. So you can imagine a scenario where there are some schools are doing very well in terms of pushing kids on the academic stuff, but not doing very well for the kids who are perhaps less well academically prepared, but who may need a little bit of a different kind of intervention to make sure they're not engaging in risky behaviors and doing bad outcomes.
Kirabo [00:06:29] And you can imagine the converse would be you can imagine schools that have a large set of students who are coming from backgrounds that need a little bit more extra support to make sure the kids are behaving well, they know how to act in accordance with the standard norms of society. You can imagine a school that really focuses on those kinds of aspects of the education and somewhat less of an emphasis on the academics. And you can imagine those schools doing really, really well in terms of reducing kids teen pregnancy or reducing kids crime activity versus the other schools that are more academically focused, having strong effects on test scores. So part of it could just be that the schools themselves may be oriented towards different kinds of outcomes, and that could be what we're seeing.
Jennifer [00:07:10] The other piece of the study is understanding parents preferences over schools. Why was your team interested in this?
Kirabo [00:07:18] So the we're interested in what parents we're just not understanding how parents choose schools because understanding parental preferences is actually really important for understanding what the benefits of school choice policies may be. So the people often think that school choice is a good thing and we think it's a good thing because having more competition among schools and by school choice, I mean having more competition among schools, having more charter schools, having voucher systems where schools have to compete for the parents to some extent. And that idea would be, you know, when the money follows the kid, if a school is doing what it should be, should be doing to attract parents, they're going to enroll their kids to the school that brings in more money. As long as the money follows the kid, then schools have an incentive to attract parents from other schools, and that generates competition among schools for the parents in the system.
Kirabo [00:08:07] And in some ways, in some sense, the idea behind these models of why this is beneficial for everyone is that if parents value things that are important for society, then the schools have an incentive to create those things. So parents value schools that are say raising test scores and the parents value that. The schools are going to do what they can to raise test scores as much as possible, and that's going to generate positive benefits for everyone. And that you could think of competition as being the tie that raises all boats. So by understanding what it is that parents actually value in schools, that gives us an insight into the kinds of things that schools are likely to be competing to sort of generate on their own to sort of attract the parents attention.
Kirabo [00:08:45] And the reason why this is sort of interesting from a resource perspective is that there are some research papers out there that have cast doubt on the idea that parents are even able to observe school quality. So if it were the case that parents are unable to determine whether a school does good or bad, then you think about the model I just described. If parents can't tell what school is good and what school is bad, then the schools are not necessarily competing to improve outcomes for kids are just competing to do something that is not actually productive for society and that's sort of that's sort of like a bad thing and would imply that the benefits of school competition might actually be zero or potentially potentially negative if the schools are competing for things that are not productive. So being able to document that parents actually value things that are productive is actually really important for demonstrating that there may be some benefits to school competition, even if it's not something that we see as measured by standardized tests.
Kirabo [00:09:35] Maybe we're seeing it in terms of the schools we're seeing in terms of reduced criminality, reduced teen pregnancy, and increasing students ability to hold on a job. Those are all things that are valuable, all things that parents may value and if the educational system is promoting those things, that is a good thing, even if they're not actually having the incentive to promote higher test scores per se.
Jennifer [00:09:55] Got it. And so tell us more about about these these other studies. What have we previously known about school quality and parents ability to tell which schools would help their kids succeed?
Kirabo [00:10:06] Right. So it's well documented in existing work that schools do vary in their impacts on test scores. So using this sort of approach I described before, this sort of value added type thinking, looking at test scores before versus after, it's been pretty well-documented that schools vary in their ability to raise student test scores and there's a little bit of research sort of documenting. There's not a lot. There's one well-known study that looks at New York City, and they do they basically find that schools ability to raise test scores are somewhat related to parent preferences. So parents do seem to prefer schools that raise test scores, but once you account for the peer composition of the schools, it doesn't seem to matter that much, which is to say parents do seem to choose schools that have higher achieving peers, more affluent peers. They're choosing schools based on the characteristics of the student composition. But once you account for the cap for the student composition, they are no more likely to choose a school that raises test scores versus one that does not. So that's from research out of New York City that basically documents this.
Kirabo [00:11:07] There's also some older research that looks at just sort of attendance boundaries. If you look at attendance boundaries in general, you'd expect housing prices to be higher in areas that have better schools. The idea is that parents are willing to pay more for a house that is zoned for a better school. So if you look at the boundary, you should be able to see that the housing prices are higher for the homes that are zoned for the better school and the school that's better able to raise test scores. The evidence on this suggests that to some extent that's true, but again, once you account for the composition of the students, there's no relationship between housing prices and the ability of schools to raise test scores. So these two papers sort of casts a lot of doubt on the idea that parents are really, really value school effectiveness per se. And maybe the only thing parents care about is the composition of the peers that their their children are going to be exposed to.
Kirabo [00:11:54] In our work, what we show is that schools do absolutely vary in their ability to raise test scores, but they also vary in their ability to do all these other things that we care about a lot. Schools vary a lot in their ability to raise to reduce criminality among the boys mainly. They vary a lot in their ability to reduce teen pregnancy among the girls and the interesting finding there is that the schools that raise test scores are not necessarily the ones that reduce criminality and vice versa. So they seem to be largely a different set of schools that are perhaps doing different sets of things. So from that perspective, the fact that these other studies have found very little evidence that parents value schools ability to improve outcomes.
Kirabo [00:12:33] Once you account for student composition, there are two reasons why this could be happening. One, it could be that parents just can't figure out what a good school is, which has been kind of the conclusion based on the previous work. Or it could be that they just value different things. And what we're finding out more is that the parents are actually valuing these other things. Parents are more likely to choose schools that reduce criminality among boys or reduce teen pregnancy among their girls. Even after you account for the composition of the peers suggesting that parents do value things about schools, it's just not necessarily the things that the researchers have been looking at for the past decade or so.
Jennifer [00:13:10] It's not just all about tests.
Kirabo [00:13:12] That's correct.
Jennifer [00:13:14] So okay. So we've got, you know, a couple other studies out there. Why has this been so difficult to figure out? Are the hurdles here mostly getting the right data? Or is it about finding useful experiments? What are the challenges here?
Kirabo [00:13:26] So the you you hit the nail on the head or the both nails on the head. Yeah. So to do this, one would need to be able to first credibly identify the effect of schools on outcomes. So for that, you need some kind of natural experiment. So that's a challenge to to get that in any setting. So we used data from Trinidad and Tobago where there students are assigned to schools using an algorithm. So the algorithm basically makes it so that there are certain sets of students who are basically the same, but some are assigned to one school and some are assigned to another for reasons outside of their control. And that basically allows us to identify, to separate the effect of the school from the effect of a student characteristics that typically might bias us, that make wrong inferences about the benefit of the school versus not.
Kirabo [00:14:13] So that's that's one thing that we were able to do because we have a data for transition. And that's one challenge is just figuring out what schools are having effects on outcomes. The second challenge, as you say, is having the data. So you have to be able to have data not only on academic outcomes, but you also have to have data that links the schools that student, the students attend to their criminal records and also to their labor market outcomes and also to their teen pregnancy outcomes. So we used data from Trinidad Tobago. This is joint work with the IADB, where we basically merged a bunch of databases to emerge all the administrative school records that we got rumors that end with the basically the vital statistics records from basically the Ministry of Vital Statistics. So they have all the births. And we also got information from the Defense Ministry on all the arrests. And then we also got some information from the Labor Department, which basically allowed us to merge in these four administrative sets, allowing us to basically identify the effect of individual schools on a variety of outcomes, not just those that are measured by standardized tests, which is typically the data that people have available. So we were able to do this mainly because we were able to compile a variety of data sets that typically are a challenge to get together in one place.
Jennifer [00:15:26] Okay, so let's take each of these pieces in turn and dig in a little bit more. So as you said, you're going to use school assignments in Trinidad and Tobago as a natural experiment to answer the questions you're interested in. So tell us more about the process of how students are assigned to schools there.
Kirabo [00:15:43] Sure. So everyone basically goes to school so that in the traditional public school setting, most students are in the traditional public schools. Only about 4% of students are and are in private schools. So basically when they get to the end of fifth grade, everyone takes an examination takes a test, it's called secondary interest examination. So they take a test and at the same time that they take this test, they submit a list of four secondary school choices that they want to attend. So at that time, at the end of primary school, basically in a primary school, they're going to decide on, they're going to put a list of four schools that they want to attend and they're going to take the test. The tests are scored centrally along with the choices, and this goes into an algorithm that basically assigns students to schools.
Kirabo [00:16:30] So students are assigned to schools based on what is called the deferred acceptance algorithm. I can describe it very, very briefly and describe how we're going to use it. So the way that it works essentially is that the first, the highest scoring kid in the country gets assigned to their top choice school. And then the second then then it goes out to the second, the second kid, the second highest scoring child in the nation. And they are going to be assigned to their top choice schools. And you keep on going and going and going until that top choice was filled. And then that school was taken out of the mix. And then the next high scoring kid will be assigned to their top choice school. And if that top choice school happens to already be filled, then they're going to be assigned to the second choice school and so on and so forth.
Kirabo [00:17:07] And you just keep on going down the list by the ordering of individuals, test scores and individuals are assigned to school. So that's basically how the assignment mechanism works. There's a few tweaks here and there that makes it not exactly like that, but that's basically how it functions. So the key aspect of that assignment mechanism that we use is the fact that if you go down the scores and you fill all the schools and at some point there's going to be some test score above which an applicant is going to be admitted to a school and the lowest they are not. There's going to be some kid who's a less kid admitted to any particular school and that basically is a kid that defines the test score cutoff or threshold above which anyone who was applied to that school would have gotten in and anyone who applied to that school who scored below that would not have gotten in and gone to a different school.
Kirabo [00:17:51] So that basically is amenable to what is called a regression discontinuity design, where essentially what you can do is, let's say we have the most elite school in the nation and it has 100 plus. And the kid who is the highest scoring kid, the 100 high school kid, maybe it is, scores 95. And then the kid right below that, their score is 94 for the kid who scored 95 got into the top school this kid who scored 94 did not belong to a different school, but you could imagine that the differences in ability and family background between a kid who scored 94 and 95 are basically minuscule. The differences could have been generated by the fact that one person had a cold that day or something like that and just scored.
Kirabo [00:18:28] They scored a little bit less than they would have. So you can imagine those two individuals being essentially the same, but one basically got into a morally school and the other and we can ask the question if we look at the outcomes among this relatively narrow sets of kids who scored very similarly, but one was above the threshold, one was below. Can we identify the effect of going to one school versus other? Do we see an appreciable jump in outcomes for individuals who scored just above the threshold and are therefore assigned to a more a more elite school compared to those who scored just below and then go to a less elite school, or I should say a less desirable.
Jennifer [00:19:01] Yeah. And especially when you know all the rankings to use, you're comparing people who they all wanted. This was their everyone's top choice or would have been the school that they both wanted to get into. So you're not as you're also kind of holding fixed with their preferences are right?
Kirabo [00:19:15] Exactly. That's exactly right. So in our research design, we have we're basically going to say among individuals who had exactly the same rank ordering of schools, school, A, B, C and D, we have two individuals. We have a whole set of individuals who had basically those same choices. There's going to be some of them that score above the threshold for a school A, which is a most elite school and some of whom who scored a little bit below that, who got into school B essentially. So by comparing those who have very similar scores and had exactly the same choices, because we observed the choices, we can basically see, okay, those that got into school A versus B, your test scores are very, very, very similar maybe one point apart, they made the same choices. So any large differences in outcomes that we observe between those who got into school A versus those who get to go to school B we're going to attribute to the benefits of attending school A versus B.
Jennifer [00:20:06] Great. And then tell us more about this amazing data you were able to get and also how you got it. How did this I imagine there's a story here about how you pulled this off.
Kirabo [00:20:17] Right. So it's one of these things where, you know, having having longstanding relationships is valuable. So for actually the first time, I'd use the data from Trinidad and Tobago was for my dissertation, which now seems like eons ago. So I got my PhD in 2007. So it was, it was probably had this data in 2005. So it's quite a while back. So at that point the Ministry of Education issued in Trinidad and Tobago, had they I think they had just started thinking about digitizing their records and they had one year of data digitized for the outcomes. And they just were in the process of getting the data for the secondary and the basically admissions exam that was taken. So my it turns out my aunt at the time worked at the Ministry of Education and she pulled some strings to get me access to the data. So I was able to basically merge in one cohort where I could basically look at their assignment scores and their choices and use exploit this exact algorithm to identify the effects of of basically going to a more elite school. And I had a measure of some of their outcomes later on. So that was in 2005. I got access to those dat basically because I knew somebody. A few years later, the whole system became a little bit more systematic and there was an official process through which one apply for data, and that's how I was able to get data later on that I use for products later on using the same data.
Kirabo [00:21:36] Fast forward another ten years and the IADB where Diether was working, they had a few projects that was also looking at the educational data and they had basically connections with basically basically the lay people and labor battle statistics and crime and they were able to facilitate pulling in all these various data sets together and using the same approach that I had sort of used, you know, several years ago, were able to pull this this project together.
Jennifer [00:22:04] Awesome. Okay. And then so what are the outcome measures you're most interested in?
Kirabo [00:22:09] Well, I mean, the ones that we focus on are sort of against sort of the crime, the ones we have, the academic outcomes, obviously, which are how well students perform on these standardized tests. They have high stakes. These are very relatively high stakes examinations that are you can actually for one of them, you can go straight. You can basically graduate high school with this. Another one is kind of like an an associate degree, which is doing well. And that would be a prerequisite to going on to a university setting, basically. So you can you can go to a nonselective U.S. institution with what is called the CSEC, which is kind of like the O-levels exam, which is one of the outcomes we look at. And then you could get into you can actually get some advanced credits if you do well on what's called the CAPE examination, which is analogous to the English A-level examinations. So those are the those are the academic outcomes we look at. In terms of the non academic outcomes, we look at having a teen birth for the girls and then for the boys. We look at having a teen arrests, have an arrest at the age of 18 or younger are the outcomes we look at and then we have whether an individual was involved in the formal labor market. So those are the main outcomes that we look at in the paper. I guess we also we also have dropout as well.
Jennifer [00:23:14] Yeah. Somewhere in between academic and academic.
Kirabo [00:23:18] That's right.
Jennifer [00:23:19] Okay. All right. So let's talk about what you find. So do schools affect test scores in this setting?
Kirabo [00:23:26] Yes. So schools have large effects on test scores and the schools definitely vary in their ability to raise individual's achievement on these tests in very meaningful ways. So are the effects that we documents are, I think, in line with some of the existing work that we've seen in the United States. Maybe a slightly larger, but we were finding very meaningful effects on performance on these on these examinations. So, you know, going from a school that say at the average to one that is at the 85th percentile. So a school that sort of average versus one that's, you know, top 20, you're talking about increasing the likelihood of completing a secondary school by probably about six or seven percentage points, which is pretty large.
Jennifer [00:24:05] And then how much do those test score impacts predict effects on all these other outcomes you're interested in?
Kirabo [00:24:11] Surprisingly little. So it's surprising in the sense that if you so, you know, if you look at the data, it's absolutely the case that if you look at just individuals who tend to do well on examinations, tend to be very unlikely to commit crimes and very unlikely to drop out of school and very unlikely to have a teen birth and we see that in our data as well. I've seen this in data in the United States. I've done similar work in Chicago. And generally speaking, if you look at the individual level, that's the relationship. So I think there's a tendency to imagine that, well, you know, you know, people who do well academically do well on all these other and all these other measures.
Kirabo [00:24:48] So we would expect that schools that raise one also improve the others. But that tends to at least in our in our data, that's not the case. And, you know, in other settings that I've looked at, it's actually not the case either. So for some of the I would say for some of the outcomes, it's close to zero, which was actually remarkable. There's very, very little relationship. So I would say the relationships all go in the direction you'd expect. So it is definitely not the case that the schools that are good at raising test scores are increasing criminality or increasing. So I just want to make that clear. So it's not the case that you have to sort of make a tradeoff between one or the other, but it is the case that the schools that raise test scores are having very small and sometimes basically no effect at all on these non test score outcomes that we value.
Jennifer [00:25:30] And then presumably there are other schools that are having really big impacts on arrests and very little impact on test scores. Is that right?
Kirabo [00:25:37] Exactly. Exactly, exactly. So to be clear, we're not saying that no schools are good at both. But in general. So, you know, there are some schools that just by chance may be doing well on both or badly in all dimensions, but in general, if you were to sort of choose a school that was doing well in terms of test scores, on average, those sets of schools are not doing particularly strongly in terms of improving these other outcomes, maybe a little bit, but not much. And the same goes for if you look at the schools are doing really, really well at reducing criminality, you might say, oh yeah, they're going to be improving test scores a lot in general. We're not seeing much evidence of that. So again, the associations are positive. So as a general rule, if you find a school that's doing well in one dimension, they're probably doing above average in the other. But it's not going to be a very strong relationship.
Jennifer [00:26:22] Okay. And then let's turn to what the parents know. So how do parents preferences across schools correlate with these various dimensions of school quality that year you're measuring?
Kirabo [00:26:32] So what we're finding is so I just I'll just explain a little bit how we figure that out. So, you know, we like I said before, the data has the choices that the students have to submit. So the choices are basically what the parents and the students decide to put on these lists. So because we know the schools that were ranked first versus second, second versus third, third versus fourth, you could infer that you can make the assumption that, well, they preferred school one to school two and school to the school three. And there's a whole literature on whether the assigning mechanism basically encourages people to be truthful about the prefaces. And the answer is yes. So we can basically use that information to infer what schools parents value. And then once you figure that out, you can sort of make some assumptions about what it is about those schools that they value. So what we find is that the choices of parents are absolutely strongly related to the effect of schools on test scores. So consistent with other with what other people have found, parents are more likely to choose schools that are close to their homes. This makes sense. They're more likely to choose schools that have higher achieving peers, which is consistent with existing research as well.
Kirabo [00:27:38] But where we're also finding that parents are likely to choose schools that raise test scores as well. So that is actually kind of a new finding in light of what I said before. Other papers have found that there is relatively little relationship between parent preferences and school impacts on test scores. Once you account for peer composition in our setting, we find even when we account for peer composition, we're still finding that parents do choose schools that raise test scores. That is true, but we're also finding that parents do also choose schools that reduce criminality and reduce teen pregnancy and increase labor market participation, but I should say that that results is something that is true on average. But there's a lot of heterogeneity in these patterns, which is probably your next question going to go.
Jennifer [00:28:20] Yeah, so. Exactly. So what are the differences here in these results across types of students and types of parents?
Kirabo [00:28:28] So basically, there are very large differences in these in the choice patterns between parents of children, of low achieving kids versus those of high achieving kids. So we basically find is that parents of high achieving kids, their choices are very strongly related to the school's impacts on test scores on these high achieving these high stakes examinations. So the high achieving kids, their parents are choosing schools that are very, very strong on the academics and their choices are relatively weakly related to the school's impacts on these non test score dimensions such as crime, teen pregnancy, labor market participation. And we find the exact opposite among parents from parents of children who are low achieving. So parents of children who are low achievers upon entry to secondary school, their choices are very strong and related to the school's impacts on crime and their impacts on teen pregnancy, and also impacts on labor market participation and very weakly related to their impacts on these sort of high stakes examinations.
Kirabo [00:29:28] So it seems as though the different kinds of parents or I should say the different kinds of kids their parents are making different kinds of choices, perhaps privileging the schools impacts on those dimensions that matter most to them and it sort of makes sense low achieving kids are also the kids who are most likely to perhaps be susceptible to criminal behavior, are more likely to be engaged in risky behaviors that may manifest themselves in being arrested, having a teen pregnancy, not being able to hold on a job. So parents of those kids value schools that make sure those bad outcomes don't happen. The higher achievement kids, they are less likely to be involved in those kinds of activities, less likely to have those problems. They're perhaps more focused on making sure their kids can go to university. They want to go to a fancy school. So they're focused on the schools that have large positive effects on these high stakes examinations. And that's basically exactly what we see borne out in the data.
Jennifer [00:30:18] So what are the policy implications of all of this? What should policymakers and practitioners who are listening take away from these findings?
Kirabo [00:30:26] One is that I think it means that I think we should take a broader view of what it is that we mean when we talk about school quality. So, you know, it's important to say that, you know, not only do we find that schools have important impacts on these other outcomes that we value, but we also find that parents are able to see it. So it's not just some weird thing that we're seeing in the data. It's something that parents are able to recognize and respond to. So that suggests a few things. One suggests that when we're thinking about evaluating schools in a way that parents should care about, we should not just use test score measures, the outcomes as a way, as a measure of quality, we should be using broader measures.
Kirabo [00:31:02] It also means that when we evaluate the potential efficacy of school choice, we should not look at whether school choice is improving test scores, which is the way that has been done in the past. We should be looking at whether school choice improves a broader set of outcomes, not just test scores. So that means, you know, maybe going back to some data sets and figuring out whether school choice is, leading to better outcomes in terms of going to college, reduce criminality, etc. So I think it's asking, I think more of the research literature who is engaged in this policy research to sort of broaden the scope of things that we that we examine and also brought our definitions of what school quality is.
Kirabo [00:31:37] I think also from a policy perspective, I think the results are speaks to the potential role of information. One nice thing which is encouraging is that we're finding that parents are making choices that seem to be consistent with them choosing schools that are going to make outcomes better for their kids. So I think what that means is if we can also provide them with information about schools, provide them with information about schools that are improving outcomes versus not, we could actually help them make better choices because they appear to be behaving in ways consistent with them wanting to make good choices, which is not surprising to say that parents want to make good choices for their kids. But the fact that we're showing the evidence consistent with that suggests that if we were able able to provide them with better information, they could perhaps make better choices, which we just improve the general system in broadly speaking.
Jennifer [00:32:22] Yeah, this does seem like very good news for that general theory of, you know, competition leading to better outcomes overall. Right.
Kirabo [00:32:29] I think that's right.
Jennifer [00:32:31] Yeah. So if any other papers come out since you first started working on the study that are related to this topic.
Kirabo [00:32:37] Yeah, there there is one paper that is a recent paper by a research team, Ainsworth, the teacher, a center and they're looking at data in Romania. And they actually did a very cool experiment where they basically try to elicit information about household's, beliefs about schools, and then saw the extent to which their beliefs predicted their choices for schools and found consistently you think that, you know, when if schools were good, they were more likely to put them at the more highly.
Kirabo [00:33:07] But then they also did like an experiment where they provided information to parents about which schools were better in terms of raising test scores. Again, for the focus on this papers on test scores, but it was a nice experiment. They actually gave the parents information and then observed changes in the choices that they made. And what they found was that providing information for parents did improve the parents choices, but they did find this only led to changes among the higher achieving children's parents, consistent with the view that perhaps these are the parents that value information about school in schools, impacts on academics. But it does suggest that there is absolutely scope for providing information to parents about the impacts of schools on outcomes. I would love to see some kind of intervention, either policy wise or even just in an experimental setting where we provided information to parents about schools impacts on these non test score dimensions such as crime. And I suspect especially in areas where criminality is a problem, there are many settings where low income areas, criminality is a problem.
Kirabo [00:34:06] I could imagine this having a really positive impact on the choices that parents are making and it can allow them to sort of not leave effects on the table to some extent. You know, if there are schools that are having large benefits and reducing criminality among the students that are attending there, and it's hard for us to know that they could be making suboptimal choices. So by providing better information to them on these non test score dimensions, I think we may actually be able to improve outcomes for these parents as well and also generate better competition on these dimensions. So this other paper that it's showing the effect of information on parents choices, I think suggests that from a policy perspective, there's a lot more scope for the kinds of information we could provide to parents and the potential for improvements on these outcomes, particularly for these, you know, these more vulnerable populations, for these outcomes that better luck for society.
Jennifer [00:34:52] Yeah. And so you've alluded to this already got a couple of of new research questions you're throwing out there. But what's the research frontier? What other big questions in this area will you be thinking about and should others be thinking about going forward?
Kirabo [00:35:06] Yeah, so I think I think the research frontier, there are two areas where I think there's a lot of where research is needed. So one I think is just trying to figure out whether we can change parents perceptions and provide them with good information in a way that actually leads them to make better choices for their children. So, you know, the paper that I just mentioned with Eric call on that team. And there's also some very exciting work that's being done in Chile by Seth Zimmerman and others as well, which I think is also very exciting that are these are experimental studies where you're providing randomly providing certain parents information about school quality and hopefully facilitating them making better choices.
Kirabo [00:35:42] So I think that's one dimension where I think more research is is what is happening right now. I would like to see more of this work focused not just on academic achievement, but also on these not ten square dimensions, but I think that's kind of the frontier for that dimension. The other part I think, relates to the question that you asked me earlier, which is what our why would it be the case that some schools are going to be promoting academic outcomes, but the other is what some schools are going to be having large benefits make outcomes, but not so much for the behavior stuff. And other schools are doing really, really good for the behaviors, but not so good for the academic stuff. I think understanding that is also going to be very important. So getting inside the black box of the, you know, the production function of these skills, you know, I've been beating this drum about the multi dimensionality of educational output for a while now.
Kirabo [00:36:25] And, you know, it's pretty clear that schools have important impacts on a variety of competencies and skills and dimensions that matter only, some of which is measured by test scores. And it's pretty clear that there are schools that are very good at promoting things that are measured by test scores. And there are other schools that are very good at promoting other things that matter a lot, such as criminality, labor force, participation, etc.. And we don't really have a good handle on what it is that they're doing to generate those differences. And I think that's kind of where the frontier is now trying to figure that out because only, you know, to some extent. And it's also gets this will be effect on policy only when we start to understand why different schools may be having effects on different kinds of outcomes, can we start to sort of think about, okay, how do we use this information to create schools or promote schools is going to reduce criminality versus academics and vice versa. So I think that's the other area where I think research is needed.
Jennifer [00:37:19] Yeah. And I guess ideally, you know, we'd like to have schools that can affect all of these dimensions in a positive direction. And serve all students rather than having because it sounds like right now what parents are doing is trying is like sorting their kids based on their particular needs and which schools can serve those needs and maybe that's great maybe that's the kind of optimal outcome. But it seems at least possible that we could design schools that can serve everybody.
Kirabo [00:37:48] I think that's exactly right.
Jennifer [00:37:49] My guest today is and Bo Jackson from Northwestern University. Bo, thank you so much for talking with me.
Kirabo [00:37:55] And thank you so much for having me. It's a pleasure.
Jennifer [00:38:02] You can find links to all the research we discussed today on our website ProbableCausation.com. You can also subscribe to the show there or wherever you get your podcasts. To make sure you don't miss a single episode. Big thanks to Emergent Ventures for supporting the show and thanks also to our patrons, subscribers and other contributors. Probable causation is produced by Doleac Initiatives, a 501(c)3. So all contributions are tax deductible. If you enjoy the podcast, please consider supporting us via Patreon or with a one time donation on our website. Please also consider leaving us a rating and review on Apple Podcasts. This helps others find the show, which we very much appreciate. Our sound engineer is Jon Keur with production assistance from Nefertari Elshiekh. Our music is by Werner and our logo was designed by Carrie Throckmorton. Thanks for listening and I'll talk to you in two weeks.