Lynne:	Hi everyone my name is Lynne Steuerle Schofield and I'm a member of the Department of Mathematics and Statistics. I'm here to talk to you today about our common wealth high stakes testing system. What I want to do is use a project that I've been working on that's looking at education policy as a way to think about what is statistics, why is it important, how does it broadly cover lots of different disciplines. I'm going to use this particular example as a way to answer some questions about what a statisticians do, what does we like, what do we find fun as a way to do this. 
	I'll start essentially by first acknowledging a couple of other people who have been working on this project with me. Frank Grosling and I have a paper that's under review right now about this work and we've worked with a couple of students who've since graduated, Emily Deena and Symone.
	I'll start by telling you a little bit about the particular policy that I'm going to talk about and then we'll get into how I've used statistics to study this particular thing. 
	Those of you that have children who are high school age will know that in January 2010 the common wealth board of education approved changes to the states graduation requirements. The big changes that they made required students to pass six what they call end of course exams. These are replacing what were previously known as the PSSA's which students had to pass. The PSSA's were a test that were done in mathematics, reading, writing and a general science test. These new end of course exams are specific to a course that you take. If I am a senior and I take a biology test, then I take an end of course exam in biology not the general science PSSA test now.
	Part of Pennsylvania law stats that whenever a state agency changes any of the rules and regulations, they have to garner public opinion about these changes that they make and then they have to make all of the public opinion that they garner, public. It makes it an interesting thing that those of us that are interesting in this can actually study.
	This change in Pennsylvania is one example of a larger shift in educational policy that's occurring where our educational policy is being more centralized at the state level. Previously local districts could make decisions, they could make policies that they want and then they would go forth and do whatever they wanted. Now more and more particularly over the last decade or two, states are stepping in and making statewide policies. 
	There are two schools of thought about this centralization of educational policy. Some see this as a huge benefit because they believe that what this does is equalize and increase equity across educational policy and educational inputs, outputs and outcomes. An easy example of a place where there was a huge benefit to centralization was in educational finance. 
	If we compare the revenue sources between Wallingford-Swarthmore school district where most of us live and Chester Upland right down the road where some of us live as well, you'll see that the revenue for Wallingford-Swarthmore almost 85% of it comes from local sources. We have a strong tax base here in Wallingford-Swarthmore. We're able to afford to give lots of local taxes that goes to support our educational tools. Whereas in Chester Upland they have a much smaller tax base. In fact, your state taxes, if you live here in Wallingford-Swarthmore, do go in some part to support the Wallingford-Swarthmore schools but they go in even larger parts to support Chester, Philadelphia, Pittsburgh, Allentown, some of the places. The idea here is that by equalizing this, we've now equalized the revenue sources into the districts meaning that now when we look at things as opposed to say two decades ago, we have approximately the same amount of expenditures for students in poorer urban and poorer rural districts versus the richer suburban districts. Here's one place where centralization would be considered a particularly good thing. It's equalized what we've got here.
	There are some though who claim that there's a huge disadvantage to centralization. That the idea here is that if we have a one size fits all policy across how we do educational work that in fact that leads to less experimentation and less diversity. Certainly those of us who are faculty here in the room probably would argue that we don't want anyone making a broad educational policy to tell us what we should do in our classrooms. The curricular decisions are a particular place where people argued against some of the centralization and saying that in fact we would want to be able to take in local customs, local cultures whatever sort of the expertise of particular teachers are to be able to do this. 
	What you see with centralization is that you have this balancing system that you're trying to do where you're trying to both equalize things to increase equity but at the same time not decrease efficiency and experimentation too much by centralizing [inaudible 00:05:19].
	The question that we, my colleagues and I have been asking is what centralization might mean for political participation. We could argue that it's pretty like that by centralizing decisions we're decreasing efficiency in people being able to voluntarily participate in the decision making that occurs. 
	Here's a really easy example. [inaudible 00:05:44] who is sitting here was recently on the Wallingford-Swarthmore school board. Carl lives three houses away from me so if there was ever a time when the school board was making a decision and I didn't like it, I could literally walk down the street, knock on his door and say, "Carl what are you doing? This is a ridiculous idea." Or, "Hey Carl can you give me a little bit more information about this?" Now that these decisions are being centralized, even if my state representative lives on my street, which he or she does not, he or she also has to take into account the fact these congressional districts actually cross many school districts. Instead of just being able to take into account me, they also have to be able to take into account all of the other school districts that they represent.
	We can see that this centralization in the sense of political participation clearly does decrease, you could argue, efficiency. We went and asked the question is it actually increasing equity when we do this at the same way as political participation. 
	Why do we even care whether or not we have equity across political participation? Political scientist, I don't know if there's any in this room. Political scientists have studied this and what they've noticed is that individuals participate quite differentially across almost any demographic variable that you can come up with. The most obvious place where people participate differentially is in voting. What we see is that those who on average tend to be white, male, high socioeconomic status and high education levels are much more likely to vote then those who are not. That's not to say that that's ... That's on average that you see. 
	These differentials become even larger when you look at things like voluntary participation. Things like signing petitions, giving money to local political action groups, participating in town hall meetings, you see it in larger differentials. What you also see is that if you dig much deeper that if you look across groups, even where groups agree, but they both disagree with a particular policy, they actually have very different reasons between different groups who are participate ... who are being against a particular policy. Men might be against one policy for very different reasons than women might be against a particular policy.
	Why do we care about this? Because participation actually matters in both the design and the implementation of our policies. What we know is that policy makers actually listen differentially as well to individuals depending on what groups and Bartell's in particular noted that policy makers tend to listen more to their constituents who are from higher socioeconomic statuses than those which are lower socioeconomic statuses. If in fact we have this differential participation, we have people who have different opinions about the modifications and changes to policies that we might have and we have policy makers listening differentially. We want to understand are people actually participating differentially? Are we making it easier for people to participate or are we setting up more barriers as a result?
	What is our data is order to try to answer this broader question of who's participating and what do they think. As I mentioned, Pennsylvania law states that every single solitary one of our public opinion had to be made public, we have every single solitary piece of material that was stated responding to these proposed graduation policy changes. That means we have every testimony, every email, every letter, every petition that was done. Giving us a little over 1,500 documents and by documents I mean a petition, a email, a testimony and over 3,000 signatures. 
	Why do we have differences between the documents and the signatures? There are some documents that are multi signature documents. A petition could be one document but could account for 250 signatures. Whereas a letter might be one document and also be one signature that we have in particular. The characteristics or what statisticians would call the variable that we've noted for each of the documents is the position that the individual stated for the policy either I am for this policy, I am against this policy. We've coded the reasons that the are either for or against in order to look to see whether or not individuals from different groups have different policy changes they would like to make, the number of actors who sign the source and because individuals when they sign such things tend to give addresses, we were also able to match almost everybody to a school district that they lived or they worked in and then the actor type, so whether or not they self identified as a school board member, a superintendent, an employee other than the superintendent, a parent and if they didn't self identify we called them a citizen.
	Our research question or at least the one that I'm going to focus on for today because we don't have enough time to do all of the research questions are who takes the public stand for or against these state policies and how representation might differ within and between school districts and these [inaudible 00:10:50]. A lot of you are probably saying wait, I thought this was suppose to be the second Tuesday science café. This seems like a political science question and not necessarily a science question. You might also say why does a statistician care about something like that. To answer that question we have to stop step back for a second and ask ourselves what is the discipline of statistics really alternately about. I see the discipline of statistics as essentially being the study of data. Statisticians what we want to know about is how individuals collect, how we organize, how we analyze, how we interpret, and ultimately how we present data and how the collection organizational analysis and presentation effects our interpretation of the results.
	We study very broadly data which means I might argue, some people might disagree, that the discipline of statistics is the most interdisciplinary discipline that you could possibly have because any single solitary discipline that involves data means that on some level you're involving some statistical guidelines or policies or theorems that are involved in that. 
	Why do we study data? Essentially because what we're trying to do is be able to explain variation that exists. In this case I'm trying to explain the variation of who participated, what different people thought about how they participated. In the case of Amy's talk last month, she's studying the variation in if two people are exposed to the same virus, why does one get sick and the other doesn't but underneath, her study is some level of statistical analysis that she had to do in order to do this. In some sense one could argue that as a statistician who's interested in policy issues, I come by this very honestly because the etymology of the word statistics. The stat part of statistics actually comes from a political german science or german political scientist who first defined statistics as being a science of dealing with data about the position of a state or community. That stat comes from the state, from the word state. We've broaden this definition of statistics to now essentially being the science of dealing with data, not just about political states but about biological states, about physical states, about chemical states, about sociological states. There's a broader definition.
	What I'd like to do for the remainder of the talk is think about how we've collected our data, how we've organized our data, how we've analyzed this data. If that does affect the interpretation of the data and in so doing provide a couple very broad guidelines. These are not all of the guidelines one needs to think about as a statistician but a couple of those and use this particular ed policy as a example, sort of a point.
	The first one we want to look at is how we collect data. What are the kinds of questions that statisticians have to ask in the collection of data and how does the collection of data actually affect how we might interpret our results. 
	The first question that most statisticians will ask when they're thinking about collecting data is what population does my sample belong to? We could argue in this case that the population that the sample of this data, this data is a bunch of petitions, it's a bunch of letters, it's a bunch of documents, it's a bunch of signatures, so it belongs to the larger population of all of the possible documents and all of the possible signatures that could have been written by citizens of the common wealth of Pennsylvania. That what we have is just one sample of the ones that were written.
	The next question we might want to ask is, is our sample representative of all the citizens in the common wealth of Pennsylvania? What do you guys think? No, who are we missing? 
Audience:	People who don't care.
Lynne:	People who don't care right? What we have in this case, whenever our sample is not representative, we're going to have some form of bias in the collection of our data. That can then bias the interpretation of our results. In this particular case as Eric pointed out, we have something that statisticians call a self selection bias. This is when individuals select themselves into a particular group and that that selection going into the group can cause a bias sample. 
	If we have two different things, what we would want would be everyone of those docs might be a random citizen or a random signature or a random document out of the entire population of Pennsylvania. We want that to be a random group of people so that we can be able to say something broadly about what the citizens in Pennsylvania believe about this new graduation requirement. What we have is more of a black hole. Of all of those who chose to write in and we know nothing about all of these citizens here in the blue. When it matters to have this self selection bias is when the choice to self select into a certain group also effects the outcome that we're looking at. That's when it matters. That's why statisticians like to have randomness. If we have randomness then we know that the choice of you being in that sample is not at all effected by any characteristic that might also affect your possible outcome is what you'd think.
	In this case, we believe that those who wrote in probably care an awful lot so what we have is the two extremes. Those who really hated this new policy and those who really loved it. We don't know anything about the people in the middle. What we also probably don't know very much about, what we probably also have are a lot more citizens who don't feel disenfranchised, can feel as though their voice is something that politicians and policy makers are going to want to listen to. We have within those who wrote in, those who think my voice actually matters. I should write in and I have a certain amount of political and economical capital that I can use to write in and make my voice known. Meaning that among these in this group as well are those who may have a very strong opinion about the policy but have neither the economic nor the political capital to chose to write in as well.
	In other words, because we've collected the data in the way that we have, because we've used the data that's available to us of those who chose to write in, the interpretation of our results we have to be very careful then to say that any results that we come up with can only be about those who chose to write in. We don't know anything about these blue people. All we know stuff about are the black. How we collect the data matters for how we interpret the results of things. I point this out just for those people who are in the room. The thing that statisticians hate more than anything is when people have already done the whole experiment and then they come to us and they say here's my data help me analyze it and we say but wait a second. As a statistician we needed to be involved at the very beginning. How did you collect the data, how did you chose your sample, who's in the sample, what individuals are there and what do we need to know about those individuals.
	The second part, the second broad issue is how do we organize this data? We've got a lot of data. You have all these petitions and it's pretty hard to interpret them if all I'm looking at is every petition and every email, all of these things individually. One question that we often have to ask ourselves is what should the unit of analysis be? In other words, what should be the basis about which I'm doing this. Should I be looking at this analysis at a per signature level or should I be looking at this analysis at a per document level? In fact, whether I do things at a per document or a per signature level can matter. Let's think broadly why this matters and then I'll give you a specific example.
	If I'm looking at things at a signature level, then ultimately the questions that I'm asking are about who participated and I'm counting whether you signed a petition or whether you wrote an individual letter as being equal. That's the unit part of unit of analysis. Everything counts as one. If you signed a petition and you wrote a letter and I'm saying those two things are equal, policy makers might not agree with that. Policy makers might say I'm going to count this letter as more because it takes more effort for you to sit down and write a particular letter by yourself. I should count that letter more than I should count some random person who got stopped at the local ACME and signed something just to get the person off their back. 
	If I count things at the document level though, then I'm saying that every document counts the same. I'm saying that then a policy maker is saying I believe you've got a petition that has 500 signatures should count the same as a petition that has 50 signatures. That might not also be true. It could be that as a policy maker I'm saying wow 500 people think this is a really bad idea or think this is a great idea. That's a whole lot more than thinking about 50 people thinking it versus 50 people I mean 500 versus 50. This is a very different kind of level about where we do. 
	Now the question of whether the unit of analysis should be signature or document is actually not one that I as a statistician can particularly answer in this case. I don't actually know how policy makers count things. I don't know and I would imagine that different policy makers count things very differently depending on what policy they want to put forward or not. What is important is to recognize that I might get different results depending on whether I do things at a signature or a document unit of analysis.
	In some sense the choice that we've made is to analyze everything both by signature and by document. Look to see where are things the same in the results that we get and where are they different in order to understand how the unit of analysis can effect things.
	Let me show you one particular case where how we count things can matter quite a bit. What we have here is bar charts which is showing the percentage of signatures and/or documents by the different actor types, whether they're superintendents, school board, employees, parents or citizens. The gray, the dark red and the dark green are the signature ones and the white, the pink and the light green are the document one. The red and the pink are the people who are against, they're saying stop the policy that's why they're red. The green ones are woo hoo this is a great idea let's go forward with it, so we're going forward. What you see is if you look here at the total we see that regardless of whether I count things by signatures or documents, about 85-90% of signatures and documents were against the policy and only about 10-15% of individuals wrote in in favor of it. That probably makes sense, very rarely do we write our local representatives and say, "Way to go, you did a really good job." We usually say, "Oh my gosh, you're doing horribly."
	Where this is interesting though is to look at the comparison of signatures and documents is to look at the comparison between school boards and look at parents. If we look at the signatures, these are the dark ones, what we notice is that about 30% of signatures are from parents and only about 15% or so are from school board members. What I'm trying to talk about is who participated. If I look at the signature level analysis what I'm saying is parents participate more than school board members do. I see the opposite actually happening if I look at the document level analysis. There I only have about 18% or so of parents participating and I have about 22, 23, 25% of school board members participating. I'm sorry not of school board members but 25% of the documents are from school board members. There I might say wow, school board members are participating more than parents. Anybody know why we see these differences?
	School board members and parents both do have vested interests but why am I seem a difference between the signature level analysis and the document level analysis. Parents are a lot more likely to sign petitions so what we end up seeing is that parents, out of all of these different groups are much more likely to be on what we call multiple signature documents. They're much more likely to sign petitions or get together with their husband or wife and say we both disagree with this. They're much more likely to get together with their PTO and say we the undersigned of the PTO think this is a horrible idea. School board members are much more likely to write in and say I'm a school board member, I know what I'm talking about and I either think this is a good or this is a bad policy. Whether or not we analyze this side of two things, gives us very different interpretations but doing the analysis at both levels actually gets us to this subtle difference in how people are participating. Not just whether they're participating. This is cool and interesting.
	We can similarly look at the differences by district of how people participated. Here what we have are box plots. What these are is the bottom line is the first quartile or the 25th percentile. The middle line right here is the median and then the top line here of the box is the third quartile or the 75th percentile. All of these are outliers, groups that are a little bit different. What this is looking at is the number of signatures per district of who was for and who was against the policy. Overall if you look at the total again you see that on average districts sent in about two signatures total. Although we have a couple of districts who sent in 682 and 418. These are districts that really got together and did some major petitions as they went through things. 
	What we look at though is, if we look here what we see is that on average high performing districts, those are the districts that were in the top 50 of those PSSA's the previous state exam that I talked about, participate much more so than do the mid performing districts which are those that did well enough to pass but aren't in the top 50 and those that are called no AYP or didn't make adequate yearly progress which is the way the federal government defines by no child left behind, you're failing or not failing based on these state level tests.
	You can think of these as being the kids who did really well on the test, these are the kids that did decently well and these are the school districts that did particularly poorly on these tests. What we see is that high performing districts participate a lot. It looks like they're pretty much against the policy quite a bit so. If we look though, we actually see that those districts that did not make AYP actually participate more than the districts that are in the middle, which is interesting because that's not what political scientists would have said. They said that people participate differentially and that those who are more disenfranchised tend to participate less. We see something a little different. Those groups that are some districts where you see a lot more disenfranchised students, you tend to see actually slightly more participation then you do in the mid performing groups. 
	Similarly we see that those districts that have high groups of URM which are under represented minorities versus low, we see that we don't see that big of a difference between these two. This is if we look at things by signature. 
	We see something a little bit different when we look at things by documents again. What we notice is again, yes we see that high performing districts participate much more so than anybody else but what you'll notice is that the only district type that has one of these green boxes are those that are not AYP. That didn't make adequate yearly progress. 
	What this is telling us is that if the districts are high performing, mid performing have high under represented minorities or low represented minorities, they're pretty much against the policy but that those that didn't make AYP, they're giving some documents that are against the policy and some documents that are for the policy. High performing districts are uniformly pretty much against it. You're seeing almost every document is saying this is awful. Whereas in the non AYP you're seeing that some of those districts are saying we think this is good, some are saying we think this is bad and some are sending in documents that are saying both. They're saying some of the documents say I think this is a bad idea and some of the documents say I think this is a good idea. 
	It's only by analyzing this both by document and by signature that we're able to see these differences that occur. If we think back to the statistical issues that we have here, the unit of analysis gives us a very different interpretation of the data that we have depending on what's going on. I could make these box plots and these bar charts a little bit more formal by doing something which is called a logistic regression. 
	I want to explain this and then I'll show some pictures of what this is. What's this thing over here on the side? The P of [kahn I 00:28:33] over P of pro I is essentially saying the probability that a particular signature is against the policy over the probability that the particular signature is for the policy. Another way to think about that is to say these are the odds. The odds that a signature is for or against. The same kind of odds when you play the lottery. You have a one in four change so one time you're going to win four times you're going to lose if you play five times.
	We take the log for mathematical reasons which I'm not going to go into but ultimately what I'm looking at there is trying to predict the odds. I'm trying to predict that in this case based on whether the individual's a superintendent, a school board member, an employee or a parent. What statisticians do is we try to estimate what these betas are going to be, the beta zero, beta one, beta two, beta three, beta four. What that's going to tell me is whether changing from being a non superintendent to a superintendent changes my odds. 
	Let's look at a picture of what this might look like. If I look at a particular signature, it can fall into one of four categories. It could be from a non superintendent and against the policy. It could be from a non superintendent and for the policy. It could be a superintendent against or a superintendent for the policy. These are the four different things. What you'll see is if you look at the columns, those that are non superintendent are weighted a lot higher at being against the policy. 
	If I take a big bowl and I put all of the signatures that are from non superintendents in that bowl, the odds that I'm going to pick something against the policy are much higher cause I've got a lot more con signatures than I do pro signatures. Whereas if I take that big bowl and instead put all the signatures that are superintendents, what I see is that they're weighted equally. It's almost like a 50/50 chance as to whether or not they're for or whether or not they're against the policy. What you see here is this downward slope. This downward sloping curve and the coefficient, that data that I showed you that I was trying to estimate before, here is negative. What that is saying is that as I move from being a non superintendent to a superintendent I reduce the odds that I'm against the policy. A negative number for those betas is saying I'm reducing the odds. The larger the number is the more I'm reducing the odds.
	We could look at a similar picture and look at parents. Here again I have four categories. I could be a parent who is against the policy, a parent who's for the policy, a parent who's not against the policy or who's for the policy and a parent that's against the policy. What we notice is that there's a lot of parents that are against the policy still. We also notice that there's quite a few of non parents who are for the policy. What we notice here is that if you are a parent there are only four parent signatures of for the policy. This is across every school district. This means whether you're a parent from a high performing school district, a low performing school district, a school district that has high kids that are in poverty, a school district that has high kids or low kids in poverty, you are against the policy. This is very different then what we saw with regard to the superintendents. 
	The waiting here is that if I put all of those parent signatures in a bowl, what are you going to guess? You're going to guess that that signature is going to be against the policy. We end up seeing a positive sloping line here. Our coefficient here is positive and it's very strongly so cause it's saying the odds once I move from not being a parent to being a parent are incredible strongly against it.
	I could go through and I could do a logistic regression on all of the different variables I have here. I could think about who wrote in and I could also think about what kind of districts they're all from. 
	The final issue I have to think about when I'm getting to this more formal analysis is where is the variation that I have within this. What's very interesting about this particular problem that we have is that we have variation at two levels. We have variation at whether or not you're a parent, a citizen, a school board member, a superintendent. We also have variation about where all of those individuals live. Did they live in school districts that did make AYP or did not make AYP? Did they live in school districts with high poverty or low poverty? What's interesting is that all of the individuals lived in a school district so there's a hierarchy. You live and you are a particular, you have a particular relationship with the educational institution and you live in a particular educational institution itself. 
	What I want to study is not just how the variation is different between and across all of these different school districts, what low AYP districts say in comparison to what high performing districts say but I might also want to examine more closely what's going on with in school districts or is there a difference if I look within the no AYP district as to how the actor type affects whether or not I'm for or against the policy so because of that-
Audience:	Can you remind me what the policy is [inaudible 00:34:07]
Lynne:	Yes it's the new graduation changes. It's changing from the test, from doing the PSSA to these end of course exams. You have to pass every one ... Your student has to pass every one in order to graduate.
Audience:	Most parents are against that?
Lynne:	Yes.
Audience:	Thank you.
Lynne:	Sure. Okay so we performed what is known as a hierarchical linear model. Remember I said that we had this hierarchy, people live within districts. That's why it's called a hierarchical linear model. What you'll notice is that the individual level thing up here, this is that same model that I showed you before, that logistic regression. All of the variables that are involved in this one are all variables that are about who the individual is. That's a variable to say I'm a school board member, I'm a superintendent, I'm this. 
	The second level, are all variables about the district itself. Did the district make AYP? Is it a high performing district? Is it a small district as defined by enrollment? Does it have high numbers of under represented minorities? Just as before, I was trying to estimate those betas to see how that changes the odds. Now I'm not just going to estimate the betas, I'm also going to estimate these alphas to see how changing from a district that made AYP to not making AYP affects the odds of whether or not I am for or against the policy.
	Here's what we see is we have results and remember a negative beta or a negative alpha means that I'm more likely to be in favor of the policy and a positive one means I'm more likely to be against the policy. We have things both at the signature and at the document level because remember we have to analyze things at two units of analysis here. 
	Here's what we noticed. High performing districts, regardless of whether it's a signature level or a document level analysis are pretty much against the policy. This probably makes some sense. They're in a system right now that's working for them. They're performing well. There students are graduating, they don't want to change the policy [inaudible 00:36:20]. However districts where we have high numbers of under represented minorities and districts that did not make AYP are much more likely to be against the policy. What's very interesting is that the only ... Oh I'm sorry for the policy, yes thank you. The only group of people who are for the policy in any way at all are superintendents.
	Now if we went back to this picture right here. If I had colored these according to whether or not the superintendent was in a high performing district or low performing district, what this hierarchical linear model will tell me is that all of the superintendents who are con are all from high performing districts. All the superintendents that are pro are all from low performing districts. Only by doing this hierarchical linear modeling can I determine that. All this tells me is that superintendents are 50/50. It doesn't tell me anything about how superintendents living within particular districts do anything for me. 
	Once again the analysis, the choice of what analysis I do as a statistician affects the interpretations of the results that I have. Some of the emerging themes that we've seen at the level of the political participations that high performing districts are far more likely to participate and there seems to be a huge alignment across it. They're all going in together, superintendents, parents, school board members, employees, they're all saying this is a bad idea, this new policy. 
	Low performing districts however are much less likely to participate but of those that do participate, we see a non alignment in the views across the actor types. Superintendents are a lot more likely to write in and say this is a good idea, this gives more money to me that I can spend that I can do, it equalizes how everybody has to do across the state, whereas non superintendents, the parents that did write in from low performing districts are also against the policy. The employees, the school board members, they're all saying we don't think this is a good idea. If you look at the reasons why, you see some interesting reasons that are different also across the parents which I can talk about if we have time for questions afterwards.
	It asks an interesting question or at least to me it raises a really interesting question, who's speaking on behalf of low performing districts? Are we listening to the superintendents or are we listening to the parents, the school board members and the employees? It's very clear it doesn't matter who we listen to from the high performing districts they're all saying the same thing. Who are we listening to from the low performing districts actually matters with regards to how we implement the policy. 
	The final conclusion I guess I would say as a statistician, so what I think is really clear and I think this gives us an interesting example is that how I collected, how we organize, how we analyze our data matters to the interpretations of the results. There's certain results that came out only because of how we collected, how we organized, and how we analyzed this data that otherwise we would see very differently. That's my final comment. There's actually a quote from John Tukey sort of a reminder of saying I think that being a statistician is the coolest because we get to see most interdisciplinary ... John Tukey said it best and he said the best thing about being a statistician is that you get to play in everybody's backyard. Thank you.
	Yes.
Audience:	[inaudible 00:39:51]
Lynne:	We actually do. We include whether or not the district is a small district by enrollment or whether or not the district is a large district by enrollment. We do actually include it in the more formal linear analysis. You're right I don't do anything about it in the pictures but I do have ... You can see we actually do look if you look at who takes a stand, I did look at how they do that by being either small or large. We see that there's slightly more participation from large districts which is probably not all that surprising. There's more people within those districts so they're more likely to participate. They're not huge differences that we see either by signature or by document in fact. Any other questions? 
	I don't think you have no idea. I think that you have to read the fine print well. Ultimately they should have some notes in their plots and their charts and things that tell you something about what the unit of analysis was. That tell you something about how they collected their data, where it all went and more and more, at least academic journals, they're forcing individuals to do more of that and actually to put their data up and make it available to anybody so that anybody could say I'm going to analyze it in a different way how do I get something different. It's true, I want to sell a magazine so I want to make the most compelling case for the case I'm trying to make. You need to be a little bit more careful about thinking about how they collect and how they organize and how they analyze the data to understand them.
	Clark.
Clark:	[inaudible 00:41:35] why they voted if everyone hated it so much, that policy.
Lynne:	It is interesting. The next step would be that there actually were changes made to the policy from the first iteration to the second. I think the next analysis that ought to be done, though this would be more difficult for a statistician to do, is to look at and say what changes were made, why were those decisions made for those changes so were the only changes made so that the high performing districts [inaudible 00:42:10], the low performing districts sort of how is it that modifications and changes the policies are made and how much of that is influence by political participation and citizen input.
	Yes.
Audience:	Does the data show or perhaps I should say was the data collected in a way that it could show any variation between parents of high performing students in low performing districts versus low performing students in high performing districts?
Lynne:	That's a great question. We don't actually know that. In some sense part of the reason that we have the data the way we do is that all we know from this data, cause it's just letters that individuals wrote in, is what they chose to self report. The reason that we actually connected every individual to a district was in some sense to use that as a proxy for telling us something about, you're from a high performing district so that means you're more likely to be white, you're more likely to have high socioeconomic status, you're more likely, based on things that we know are correlated with certain things. There's no way to know anything about that. It's a very interesting question and one I'd love to study more.
	Yes.
Audience:	[inaudible 00:43:24]
Lynne:	These are the reasons that we coded for the oppositions. You can see that they're across different things. The C3, penalizing students for systematic problems is better [inaudible 00:43:39] earlier in their career, that was the reason or that reason was most often given by those in low performing districts. They're saying we're not actually opposed to some sort of state wide policies coming in, we just wish that you wouldn't do it once they're seniors. Why don't you actually do something to help support them so they're not low performing from the very beginning. Use those resources to invest in preschools or invest in other things. Those from high performing districts, a lot of them made the argument that there's a lot of local control. They made this efficiency argument. They made this thing to say what a second, we're doing well already so why are you coming in and telling us what to do. We're making good choices so you shouldn't be trying to take that away from us. You see that there's big differences depending on which one of these things [inaudible 00:44:30].
	Tim.
Tim:	[inaudible 00:44:32] 
Lynne:	I think that's a really interesting question. I think on some level you could say that. I think what I find interesting is that as a nation in some level we talk about high stake testing and we use it as a noun. I would argue in some sense, it's a verb like depending on which state or local place I live, high stake testing means something very different. Even in for instance this case, the first iteration of the policy was that if they fail they don't graduate period and then as I said a bunch of the high performing parents came in and said whoa, no way, my kid's getting an A in this class and then they don't pass the test and now they don't graduate and that means they can't go on to college or whatever in the workplace. They changed the policy so that now it's actually worth 1/3 of their grade. It's still possible to fail this exam and graduate and pass. 
	On the one hand yes, I broadly yes I agree. It says something about governance but on the other hand I think we need to be a little bit more careful about looking more specifically at what the details and characteristics of the policies are and what particularly people are for and against within that policy. That's why even though I ... This starts to give us that. I didn't have time to do that today obviously but that's exactly what these kinds of things start to tell us about. Looking at things like why people are against what sort of differences they had. 
	What you see is superintendents said they were against the policy for the most part because they believe it harms particular groups. They might argue well let's just make tests that provide certain accommodations for groups [inaudible 00:46:25] thing. That's very different then the parents who aren't saying it's harming some group but are saying whoa, I'm losing all my local control, I want to be able to do this.
Speaker 5:	We're past the hour. I want to thank Lynne for-