Jersey Jazzman

2 UPDATES BELOW

As we enter the final days of the election, the debates have intensified -- and not just over whether we should give the nuclear codes to an unqualified, misogynistic, racist maniac...

In Massachusetts, a battle is being waged over Question 2, a proposal to lift a statewide cap on the expansion of charter schools. Like many of the debates over charter expansion and educational "choice," much of the rhetoric revolves around the ostensible "gains" that charter students make compared to students who are in public district schools.

But the Q2 debate is unique: a group of academics, many affiliated with MIT, have been following the expansion of charter schools in and around Boston for years. I can't think of another region where charter schools have been studied so closely using econometric methods. This work has been at the heart of the policy discussion of Q2; it's cited repeatedly in reports about the ballot initiative.

And the economists who conducted this research have been happy to help make the case that their studies support lifting the cap:

As policies are debated, we often have to rely on research that is ill-suited to the task. Its methodology is frequently too weak to form a firm foundation for policy. Or, the population, design, and setting of the research study are so different from the policy in question that the findings cannot be easily extrapolated.

This is not one of those times. We have exactly the research we need to judge whether charter schools should be permitted to expand in Massachusetts. This research exploits random assignment and student-level, longitudinal data to examine the effect of charter schools in Massachusetts. [emphasis]

Do we? Do these studies tell us exactly what we need to know about whether Massachusetts should lift its cap on charter expansion?

Because simply showing that charter school students in Boston get better test scores than similar students in the Boston Public Schools is not, I'm afraid, nearly enough evidence to support lifting the cap. In fact, the more time I spend looking at this research, the more questions I have about whether Massachusetts can reasonably expect charter expansion to improve its schools:

- Are the students who enter charter lotteries equivalent to the students who don't? This is a critical limitation of these studies that is often ignored by those who cite them. The plain fact is that the very act of entering your child into a charter school lottery marks you as different from the rest of the population; you are taking an affirmative step the majority of public school parents are not taking in an attempt to improve your child's education. There's a real likelihood your family is not equivalent to a family that doesn't enter the lottery.

Here's a diagram from a report I coauthored with University of Wisconsin professor Julie Mead:

If the target population is all Boston students, we have to acknowledge the samples from the lottery studies are missing many of them. And it's not just students who don't enter the lottery; what about students who enter, don't earn a seat, and subsequently move to, say, a private school?*

Here's another look at the issue. This is quick and dirty but it makes my point:


It's not always clear how to calculate the overall target population in these studies; I used the Ns that made the most sense to me.** But even if we're not quite sure about the exact numbers, the scope of the issue is clear: the study sample is only a fraction of the total population. Which would be fine -- if the sample was randomly drawn from the target population.

But clearly, that's not the case: The sample is self-selected, because families have to choose to enter the lottery. Which means the results of the study can only be generalized to that population, because there may be characteristics of the students in the sample that are different from the entire Boston population and affect test scores.

Now, there is one thing researchers can do to mitigate this problem: compare the measurable student characteristics of the study sample and the target population. But there's an issue with doing this, because the metrics used to measure things like economic disadvantage aren't really up to the task.

Ironically, we know this thanks to the work of Sue Dynarski, one of the authors of the lottery studies and the coauthor of the paragraph above. According to Dynarski, crude dichotomous measures like free lunch-eligibility mask significant differences in socio-economic status. So we can't truly determine whether the family characteristics of students in charters are equivalent to those in BPS.

But even if we could, I would still say it wouldn't tell us what we really need to know. It's the unobserved characteristics that probably count here, especially parental involvement and support. Charters often require things like family contracts and time commitments that not all parents can adhere to. The only practical way to account for this is random assignment to treatment: not after entering the lottery, but before.

- Are the effects of charter schools due to their "charteriness"? All of the above said, I still think it's clear that something is going on in Boston's charters. If, however, they are going to be scaled up, it's important to ask not just who gets the gains, but why.

Many of the MIT lottery studies cite Thernstrom and Thernstrom's No Excuses: Closing the Racial Gap in Learning for a qualitative description of the inner workings of charters. The truth, I'm sorry to say, is that this book is not a serious piece of research; it's a political document whose methods would be questionable if they were actually documented.

In the most recent lottery study (Cohodes, Setren & Walters, 2016), the authors describe Boston's charters this way:

The potential for sustained success at scale is of particular interest for “No Excuses” charter schools, a recent educational innovation that has demonstrated promise for low-income urban students. These schools share a set of practices that includes high expectations, strict discipline, frequent teacher feedback, high-intensity tutoring, and data-driven instruction.

First of all, how do we know the charters are any different than the traditional public schools regarding these school practices? Angrist, Pathak & Walters (2012) surveyed charters for their practices, which is fine... except we don't really know how they compare to the public district schools. If we're going to ascribe effects to these practices, we should know how they differ across our treatment and control schools.

We can say, however, that the charters have longer school days and years. This is probably a significant contributor to any effects the charters show. But is it necessary to expand charters to lengthen instructional time? Can't Boston just do that in its district schools?

Probably not easily, because the charters have another big difference with BPS schools:

From Cohodes, Setren & Walters (2016). The teachers at the charters studied have far less experience than BPS teachers. In fact, the charters don't have even one teacher over the age of 49. As I've noted in my studies of New Jersey charters, a staff with less experience means less expense; that, in turn, means you can pay staff more to have more instructional time, but still keep your overall costs low.

There's a good argument to be made that it's unrealistic to expect public schools, subject to collective bargaining agreements, to deploy this staffing strategy. Which brings up my third question...

- Can the strategies that make charters "successful" be brought up to scale? Here's a graph based on the table above.


The study only looks at middle schools, and the BPS number above appears to be for all grades within the district. Still, the scope of the issue is apparent: as the charters take more market share, they are going to have to hire a lot more teachers. Will they still be able to attract young, educated people to work in their schools? And will they retain their current teachers, or will they churn-and-burn their staffs? If there's going to be a lot of turnover, the demand for new staff will be even greater than expansion by itself would require.

Boston is renowned for its colleges, so there are plenty of well-educated young people in its workforce. But can the city really continue to supply what the charters need for their staffs if the cap is lifted?

Furthermore, is this good for the teaching profession, and students in urban schools, in the long run? Does Boston really want to be known as a place that hires many inexperienced teachers that leave after a few years? We know teachers gain in effectiveness as they gain more experience; is churn-and-burn really a good model that deserves replication on a wide scale?

In addition: as I pointed out before, there are patterns of significant cohort attrition within Boston's charter high schools:


There are two possibilities for this pattern. First, the charters may be shedding students, which calls into question the viability of expanding their enrollment. However, as a reader of mine recently argued, maybe the charters are retaining students, giving them an extra year of enrollment to catch them up. OK...

Is Boston ready to spend much more on its schools so it can retain even more students? How much more will this cost? Has anyone looked into this?

This last question is yet another reason I can't agree with Dynarski when she says we have exactly the research we need to be confident Q2 will lead to better education for all of Boston's students. Yes, I agree that the charters are working for many, if not most, of the students they enroll -- at least so far as we can measure based on test scores and other quantifiable outcomes. But there are serious questions as to whether lifting the cap will bring gains that are worth the costs -- questions that cannot be answered by lottery studies.

And, yes, there are costs. As this clever model developed by a couple of public school parents shows, districts can't easily absorb the costs of charter expansion, which is why the state offers extra funds. Unfortunately, the state has not fully funded this program in recent years; if they can't find the money now, how will they find even more funding in the future?

We know that charters place fiscal burdens on hosting districts, largely because they educate students who would otherwise go to private school and they replicate administrative and other costs by creating multiple systems of school governance. We know that charters are not held to the same standards of transparency and accountability as public district schools, because they are not state actors. This has created major problems in other states, incentivizing behaviors that are not in the public's interest.

Is it not possible, given all this, that Boston's charters are getting good results because of the cap? That limiting their expansion has increased quality and stopped the abuses that have plagued states like Michigan, Ohio, and Florida, which have let charter expansion run wild?

I've seen plenty of folks on social media cite Cohodes, Setren and Walters (2016) as proof that charter expansion will work in Massachusetts. I'd point out that the study is only measuring effects in middle schools, and, again, the enrollments are only a fraction of what they will be if the cap is removed.


This chart uses data from the same year as the last year of their study. The high school market share is still quite small, and a significant portion is in Horace Mann charters, which are controlled by the hosting district. It may well be that the cap has kept Boston from reaching a tipping point where the damages wrought by rampant charter expansion can no longer be contained.

I want to be clear about something: I think these lottery studies are well worth considering. I have great respect for the economists who have been doing this research. Josh Angrist's books sit on my shelf (well, OK, my Kindle).

But the work they've done is limited, so no -- we don't have exactly the research we need to state confidently that lifting the cap will be worth the costs. You wouldn't know that to read the popular press accounts of these studies, but it's true nonetheless.

I understand Boston's children can't wait any longer for real improvement in their schools and their lives. But lifting the cap largely on the basis of these limited studies is not, in my opinion, smart public policy. The good people of Massachusetts have every right to question whether voting yes on Q2 is in the best interests of students both in and out of charters, and to consider the limits of the evidence presented to them as they make their decision.



ADDING: I can always count on a group of reformy types to criticize my stuff without apparently reading it. Just to be clear:

According to the NCES Private School Survey, there are 4,558 students enrolled in private schools with a Boston address. It is, of course, impossible to know how many of those had applied to charter schools and then, after not being offered a seat, enrolled in a private school. But, as Bifulco and Reback (2012) note (p.2):

Charter schools can generate excess costs for a number of reasons. First, charter schools can be expected to attract some number of students from private schools (Buddin, 2012; Toma, Zimmer, & Jones, 2006; Ludner, 2007). The additional resources that charter schools use to educate these students are not necessarily new resources from the point of view of society. Nonetheless, transfers from private to charter schools do shift educational costs from the private schools and their parents to the public sector and taxpayers, and thus, create fiscal impacts for public education systems. [emphasis mine]

You can, of course, dispute the scale of this. But there is a sizable population of private school students in Boston. It's hardly unreasonable to bring this up.

UPDATE 1:

I see I've kicked up quite a bees nest among economists and others who are inclined to support Q2. Let me add a few thoughts from some other folks on the utility of charter school lottery studies.

McEwan and Olsen (2010), from Taking measure of charter schools: Better assessments, better policymaking, better schools. R&L Education. (p.103).



Here's Bruce Baker:

The other type of study is often referred to as meeting the gold standard – as being a randomized study – or lottery-based study. It is assumed, since these studies are declared golden, that they therefore necessarily resolve both above concerns. And it is possible, that if these studies truly were randomized (or even could be) that they could resolve the above concerns. But they don’t (resolve these concerns), because they aren’t (really randomized).

First, what would a randomized study look like? Well, it would have to look something like this – where we randomly take a group of kids – with consent or even against their will – and assign them to either the charter or traditional school option. The mix of kids in each group is truly random and checked to ensure that the two groups are statistically representative (using better than the usual measures) of the population.  Then, we have to make sure that all other “non-treatment” factors are equivalent, including access to facilities, resources, etc. That is, anything that we don’t consider to be a feature of the treatment itself. This is especially important if we want to know whether expanding elements of the treatment are likely to work for a representative population.  This is a randomized, controlled trial.

So then, what’s randomized in a randomized charter school study? Or lottery-based study?  One might sketch out a lottery-based study as follows:

Here, the study is really only randomized at one point in a long complicated sequence – the lottery itself. Students and families have to decide they want to enter the lottery – that they are interested in attending a charter school, which will ultimately affect the composition of the charter school enrollments. Then, among those selecting into the pool, students are randomly chosen to attend the charters along side others randomly chosen to attend (from a non-random pool of lottery participants), and the others randomly selected, to go, well, somewhere else… with a group of peers non-randomly chosen to end up in that same somewhere else.

So, while the studies compare the achievement of kids randomly chosen to those randomly un-chosen (thus comparing only those who tried to get a charter slot), the kids are shuffled into settings that are anything but randomly assigned, containing potentially vastly different peer groups and a variety of other differences in setting. Add to this the likelihood of non-random student attrition, further altering peer group over time.

As such, I very much prefer these studies to be referred to as “lottery-based” rather than randomized or experimental. These studies are randomized at only one step in this process, potentially conflating setting/peer effects with treatment effects, thus substantially compromising policy implications.

As with those matching studies, the types of variables used to check and/or correct for peer composition and non-randomness of attrition are often too imprecise to be useful.

UPDATE 2:

Diane Ravitch posted about this post, as she often does with my stuff. Dave Leonhard of the NY Times, who I link to above, tweets back:

David LeonhardtVerified account‏@DLeonhardt

David Leonhardt Retweeted Diane Ravitch

The entire start of this blog post is based on a falsehood: The study compares lottery winners to lottery losers, not to full population.

Yes, that is exactly what I say above, and that's the problem: you can't generalize to the full population when you have a self-selected sample, because self-selection may correlate with an unobserved variable that biases your estimates. The best you can do is compare the sample to the population on observed variables -- but if those are crude or don't correlate with the unobserved variable, they aren't going to be up to the job.


* There's also the matter of compliance with treatment and the use of instrumental variables in these studies' regressions. It's a complex issue that I'll have more to say about later.

** The studies will give "BPS students" as a number for comparison, but it isn't always clear whether charter enrollments are included. I tried to get this as accurate as I could, but, as always, caveat regressor.

Oh, my:



Kills me we don't have video of Cannonball playing this. The slideshow on this video is good, though.

I played this today at a Jazz Vespers series I'm doing in Morristown. Stop by on Sundays this month if you feel the spirit.

The debate over Massachusetts's Question 2 -- a referendum on lifting the cap on the number of charter schools in the commonwealth -- rages on. When I last weighed in, I pointed out that the "successes" of Boston's charter sector could not fairly be compared to the "failures" of the public schools because the two sectors were educating fundamentally different students.

One indicator of this is the cohort attrition rate: the shrinkage in the size of a cohort that occurs because students leave a school, but are not replaced with new students entering.

Every once in a while -- if you listen carefully -- you can hear charter school supporters give away their game...

Take, for example, Shavar Jeffries, president of Democrats for Education Reform, one of the most influential pro-charter lobbying outfits in the country. Jeffries was put out by yesterday's resolution from the NAACP calling for a moratorium on charter school expansion. It's clear Jeffries sees the NAACP's position is a betrayal of its core mission -- but notice who he evokes to make his case:

New York, NY — Following today’s vote by the NAACP to approve a proposed moratorium on all charter schools, DFER President Shavar Jeffries released the following statement:

“W.E.B. DuBois is rolling in his grave.  The NAACP, a proud organization with a historic legacy of expanding opportunity for communities of color, now itself stands in the schoolhouse door, seeking to deny life-changing educational opportunities to millions of children whose parents and families desperately seek alternatives to schools that have failed them for too long.  Public charters schools throughout the country are creating new pathways to college and career that were previously unavailable.  The idea that the NAACP would support a blanket moratorium that would apply across-the-board to all charters, including schools like Urban Prep that send 100% of its graduates to college, is a tragic contradiction of what the NAACP has traditionally stood for. The NAACP faces a choice: cling to policies of the past that have failed Black children for decades, or embrace the future and the innovative practices that will create hope and opportunity in places where neither is present.  We will continue to stand with Black parents and families throughout the country, as well as leaders of color, including President Obama, in supporting any policy in any form that creates equity and opportunity for children of color. The NAACP has chosen a different path, and history will record its choice.” [emphasis mine]

Let's talk a little about DuBois and education:

I've actually been doing a bit of reading on the debates surrounding the education of African Americans that took place in the wake of the Reconstruction Era. James Anderson wrote a brilliant book, The Education of Blacks in the South, 1860-1935, that digs deep into this history.

The standard trope about this era is that there was a debate between DuBois and Booker T. Washington that pitted DuBois' desire for blacks to pursue a classical liberal education against Washington's call for vocational training.

Anderson shows this is far too simplistic: in fact, very few in the black intelligentsia sided with Washington. But as DuBois makes quite clear in his famous essay from 1903, the elite education he favored was just that: elite. DuBois clearly and unapologetically was calling for a particular sort of education that would develop the best and the brightest -- the "talented tenth." (all emphases mine)

The Negro race, like all races, is going to be saved by its exceptional men. The problem of education, then, among Negroes must first of all deal with the Talented Tenth; it is the problem of developing the Best of this race that they may guide the Mass away from the contamination and death of the Worst, in their own and other races. Now the training of men is a difficult and intricate task. Its technique is a matter for educational experts, but its object is for the vision of seers. If we make money the object of man-training, we shall develop money-makers but not necessarily men; if we make technical skill the object of education, we may possess artisans but not, in nature, men. Men we shall have only as we make manhood the object of the work of the schools–intelligence, broad sympathy, knowledge of the world that was and is, and of the relation of men to it–this is the curriculum of that Higher Education which must underlie true life. On this foundation we may build bread winning, skill of hand and quickness of brain, with never a fear lest the child and man mistake the means of living for the object of life. 

If this be true–and who can deny it–three tasks lay before me; first to show from the past that the Talented Tenth as they have risen among American Negroes have been worthy of leadership; secondly to show how these men may be educated and developed; and thirdly to show their relation to the Negro problem.

 DuBois continues:

Can the masses of the Negro people be in any possible way more quickly raised than by the effort and example of this aristocracy of talent and character? Was there ever a nation on God’s fair earth civilized from the bottom upward? Never; it is, ever was and ever will be from the top downward that culture filters. The Talented Tenth rises and pulls all that are worth the saving up to their vantage ground. This is the history of human progress; and the two historic mistakes which have hindered that progress were the thinking first that no more could ever rise save the few already risen; or second, that it would better the uprisen to pull the risen down. 

All men cannot go to college but some men must; every isolated group or nation must have its yeast, must have for the talented few centers of training where men are not so mystified and befuddled by the hard and necessary toil of earning a living, as to have no aims higher than their bellies, and no God greater than Gold.

Further:

I am an earnest advocate of manual training and trade teaching for black boys, and for white boys, too. I believe that next to the founding of Negro colleges the most valuable addition to Negro education since the war, has been industrial training for black boys. Nevertheless, I insist that the object of all true education is not to make men carpenters, it is to make carpenters men; there are two means of making the carpenter a man, each equally important: the first is to give the group and community in which he works, liberally trained teachers and leaders to teach him and his family what life means; the second is to give him sufficient intelligence and technical skill to make him an efficient workman; the first object demands the Negro college and college-bred men–not a quantity of such colleges, but a few of excellent quality; not too many college-bred men, but enough to leaven the lump, to inspire the masses, to raise the Talented Tenth to leadership; the second object demands a good system of common schools, well-taught, conveniently located and properly equipped.

Now, it is certainly true that DuBois, no doubt feeling he had to respond to his critics, adjusted his ideas about the "talented tenth" later in life. In 1948, he specifically addressed how his thinking about the "talented tenth" had evolved:

Turn now to that complex of social problems, which surrounds and conditions our life, and which we call more or less vaguely, The Negro Problem. It is clear that in 1900, American Negroes were an interior caste, were frequently lynched and mobbed, widely disfranchised, and usually segregated in the main areas of life. As student and worker at that time, I looked upon them and saw salvation through intelligent leadership; as I said, through a "Talented Tenth." And for this intelligence, I argued, we needed college-trained men. Therefore, I stressed college and higher training. For these men with their college training, there would be needed thorough understanding of the mass of Negroes and their problems; and, therefore, I emphasized scientific study. Willingness to work and make personal sacrifice for solving these problems was of course, the first prerequisite and Sine Qua Non. I did not stress this, I assumed it.

SACRIFICE

I assumed that with knowledge, sacrifice would automatically follow. In my youth and idealism, I did not realize that selfishness is even more natural than sacrifice. I made the assumption of its wide availability because of the spirit of sacrifice learned in my mission school training.

In my reading, DuBois didn't call for the removal of an African American aristocracy; he just wanted a better one:

This screened young membership must be far greater in number than it is now. Baltimore for instance has more than 166,000 Negroes and only 23 in its Boule, representing less than 100 persons. Surely there must be at least 23 other persons in Baltimore worthy of fellowship. It is inconceivable that we should even for a moment dream that with a membership of 440 we have scratched even the tip of the top of the surface of a group representative of potential Negro leadership in America. Nothing but congenital laziness should keep us from a membership of 3,000 by the next biennium without any lowering of quality; and a membership of 30,000 by 1960. This would be an actual numerical one hundredth of our race: a body large enough really to represent all. Yet small enough to insure exceptional quality; if screened for intelligent and disinterested planning.

So DuBois, even in his later years, believed in the education of a "screened" group of "exceptional" black students whose mission would be to become leaders: "The Negro race, like all other races, is going to be saved by its exceptional men."

Frustrated that the traditional public schools aren’t willing to prioritize their children’s needs, many low-income strivers have turned to high-quality charter schools instead. But now those are under attack, too. Both the PBS Newshour and the New York Times have recently presented highly critical coverage of Success Academies, charter schools in New York City that have shown excellent results in improving student performance. The reports focused on the academies' suspending students aggressively and pushing out chronic disrupters. There were similar controversies over the relatively high rates of suspensions and expulsions at charters in Chicago and Washington in recent years.  

The casual observer might wonder: What’s wrong with that approach? Why not ensure that schools are safe places to be? If the Success Academies and schools like them didn’t exist, many of those hard-working, high-achieving students would be in chaotic, low-performing public schools. Don’t their needs count? [emphasis mine]

This is a "talented tenth" argument if I ever heard one (although maybe Petrilli would argue about the actual percentage). I've given Petrilli credit before for daring to go where most in the charter industry won't: the biggest selling point that urban charter schools make to parents isn't "innovation" and "freedom" -- it's peer effects.

We see this in the high suspension rates of Uncommon's charter schools. We see it in the classroom practices and discipline policies of Success Academy. We see it in the "no excuses" philosophy of KIPP (to their credit, some of the KIPP schools seem to be rethinking this). Despite all their protests to the contrary, these "successful" urban charter schools are attempts to foster elitism. They exist not to educate everyone; they exist to educate the "talented tenth."

Now, there's a legitimate critique to be leveled against white guys like me living in the suburbs who express our reservations about all this: who am I to tell people of color how to run their schools? Who am I to tell a family that wants to do right by their children that they shouldn't have the opportunity to send their child to a school with positive peer effects? Why should my kids get the advantages of being surrounded by lots of college-bound peers while other kids aren't?

It's a fair point, and it's why I never fault a parent for sending their child to a charter school; everyone has to do what's best for their kids.

But if charter cheerleaders like Jeffries are going to promote "choice" by evoking DuBois, I think it's more than fair to start asking a few tough questions:

- If charters are serving the "strivers," why pretend they are doing the same job as the public schools? Why pretend that they've found some sort of secret sauce that leads to improved outcomes when the real secret may be unmeasured -- but real -- differences between students?

- If the primary goal of urban education going forward is to serve the "talented tenth," what happens to the other 90 percent? What sort of schooling to they deserve? Why don't they get the resource advantages of "successful" charters? Why should their public schools suffer just to make sure charters get a bigger slice of the pie?

- Why do the charters need to be privately run? If their "success" is due to different student populations (and, to some extent, resource advantages), why allow them to operate under a set of rules that actually incentivize behaviors that aren't always in the public interest?

- Why is it OK to segregate the strivers from everyone else in the urban schools, but not in the suburbs? As I've said before, there are plenty of kids in the leafy 'burbs who have emotional, cognitive, and other learning impairments. But unless they are extreme cases, suburban schools go out of their way to integrate these children in with their peers. Why should this be the case in some schools but not others?

- If charters are serving the "strivers," why are so many of them engaging in pedagogical and disciplinary practices that do little to foster student agency and social justice? Why do so many "successful" charters have high attrition rates? And high suspension rates? And a curriculum that is narrowly focused on test scores and disturbingly teacher-centered? Is this really the best way to develop leaders?

- Finally: yes there are some "successful" charters. But they are not the norm.


If you look at the total charter sector, the results are underwhelming, and profit-taking runs rampant. How does it serve anyone -- in or out of the "talented tenth" -- to enroll students in the many charter schools that put profits before the needs of students?

I'm glad Shavar Jeffries decided to quote W.E.B. DuBois -- his words are well worth considering as we continue to ponder the growth of the charter school sector.

ADDING MORE: Julian Vasquez Heilig reminds us that DuBois was not a big fan of the privatization of governmental roles:

A week ago, I wrote a post about attrition rates in Massachusetts charter schools. The issue is important as the state will vote soon on Question #2, which would lift the cap on charter school proliferation. If charters lose students at inordinately high rates, there's a good chance allowing them to expand won't improve student outcomes, and could instead cause harm to public schools.

Diane Ravitch often comments on my stuff and provides a link, and did so with this post. Her readers are more inclined to leave comments than mine, and a discussion about attrition in MA's charters ensued. Stephen B. Ronan, a frequent commenter at Diane's site, left a particularly long one; he also posted here.

With his permission, I am reposting his comment below. I'm doing so because he raises many of the same points I've read in the larger debate over Question #2, and I thought it would be useful to have them all in one place so I can answer them in subsequent posts.

Leave your comments below; mine will follow over the next few weeks, with the exception of the following:

- I didn't provide a link to the MCPSA document in my original post. That was an oversight; regular readers know I post primary source links regularly. I've added the link to the original post.

- Phoenix Charter Academy is, indeed, in Chelsea and not Boston. Again, I corrected that in the original post, and apologize for the error. That said, excluding Phoenix does not substantially alter my original analysis.

It looks to me, JJ, as if you may have some misunderstandings that may be worth correcting.

Without providing a link, you quote the Massachusetts Charter Public School Association.
I think this is the full document for those who may be interested:
http://www.masscharterschools.org/media/news/just-facts-public-charter-schools-massachusetts

The segment you quote from states:

"The attrition rate at Boston charters (9.3%) is significantly lower than in BPS (14.2%).
In Gateway Cities, charter attrition rates (6.2%) are lower than Gateway districts (11.4%).
"From 2012-2014, an average of just 82 students left charters and returned to Boston Public Schools, according to BPS numbers – one-tenth of one percent of BPS total enrollment of 57,000."

And you responds to that:
"Yeah, uh... no. Not really."

You claim, properly, that our department of education's "attrition" rate is inadequate to provide a full perspective insofar as it addresses students leaving during the summer. What you may not understand is that the sentence that starts "From 2012-2014" is supplementing the summer attrition figures by citing the number of students who leave charter schools to return to BPS schools not during the summer but rather during the school year. I think the Association's point there is that there's not much such movement during the year relative to summer attrition which is when most of the mobility between sectors occurs.

Our state department of education makes very easily available both summer attrition rates and also measures that show the rates at which students leave during the year. For example, to view the summer attrition and the school year stability rates of one superb charter school and one superb traditional public school, see;

MATCH attrition: http://profiles.doe.mass.edu/attrition/default.aspx?orgcode=04690000&fycode=2015&orgtypecode=5&
MATCH stability: http://profiles.doe.mass.edu/mobility/default.aspx?orgcode=04690000&fycode=2015&orgtypecode=5&
Nathan Hale attrition: http://profiles.doe.mass.edu/attrition/default.aspx?orgcode=00350243&fycode=2015&orgtypecode=6&
Nathan Hale stability: http://profiles.doe.mass.edu/mobility/default.aspx?orgcode=00350243&fycode=2015&orgtypecode=6&

You write: "If charter schools shed kids year after year -- especially if those kids are low-performing -- then their vaunted performance advantages are in question, particularly when compared to public district schools that aren't losing students."

Yes, but in fact the evidence demonstrates that charter schools in this area shed considerably fewer kids than public district schools.

I guess you recognize that may well be the case so move swiftly on to change the subject:
"What we really need to consider is whether the students moving out of charter schools are being replaced at rates equal to the replacement rates for students moving out of public district schools."

Your conclusion there is correct: there's far less backfill in the charter sector.

You write: "Every independent charter school in Boston had a higher cohort attrition rate in 2014 than BPS as a whole."

Is this a specifically New Jersey use of the phrase "cohort attrition rate"? I find it difficult to locate a definition in the academic research literature that combines what is normally considered to be attrition (loss of students) with backfill (gain of students) and calls the merged product "attrition". Though I do see it used that way by both you and Bruce Baker and in New Jersey.

In respect to what may constitutes a "cohort" in respect to individual schools in Boston, it appears that you blend members of multiple entering classes, and of multiple cities. For example, you write:

"In the case of City on a Hill and Phoenix, their 2011 freshman class shrank by more than half by the time they were seniors."

Phoenix doesn't have a school located in Boston. Perhaps you are thinking of the Phoenix charter school not too far away in the city of Chelsea. But given the nature of the school's specialization it makes it doubly hard to justify comparing it with a traditional public school in Boston:

"The school she founded, Phoenix Charter Academy, was conceived as an alternative school for students who had given up on traditional high school, or who felt that their schools had given up on them. As it celebrates its 10-year anniversary this week, it remains a haven for students who thought they might never finish high school."

https://www.bostonglobe.com/metro/2016/03/31/chelsea-charter-program-that-reached-struggling-kids-reaches-milestone/KYZKbHYHdsqCVD6XM8xxeI/story.html

As for City on a Hill, you omit the implications of this, from its annual report: "There is no social promotion at City on a Hill. 100% of students promoted to the next level in each subject demonstrated mastery of the school’s common-core aligned college prep curriculum by earning 70% or above on written and oral proficiencies."

http://www.cityonahill.org/wp-content/uploads/2015/08/City-on-a-Hill-2014-15-Annual-Report-Final.pdf

Presumably as a consequence of that policy City on a Hill has extremely front-loaded grade level retention. The 9th grade class enrollments you examine merge students who are newly starting out and others who are repeating the grade. That distorts any attempt to compare 9th grade and 12th grade enrollment figures to measure attrition.

Massachusetts grade-level retention reports can be found here:

http://www.doe.mass.edu/infoservices/reports/retention/

Looking at Appendix B for 2015-2016, one finds for City on a hill the following grade-level retention rates:
9th 10th 11th 12th
27.8 3.3 3.2 7.4

In the past, in the several messages here, you and I discussed whether such a front-loaded grade-level retention pattern skews the results you and your NJ colleagues try to portray:

https://dianeravitch.net/2016/07/14/jersey-jazzman-takes-charter-cheerleaders-to-the-woodshed-for-destroying-public-schools/#comment-2582329

I continue to believe it does.

The one clearly valid point you make, not a novel one, is that there's less backfill at charter schools here in Massachusets charter schools than at traditional public schools. I'd be interested to see you debate with Michael Petrilli on whether or not that's helpful:

https://edexcellence.net/articles/backfilling-charter-seats-a-backhanded-way-to-kill-school-autonomy

You write: "Is it really worth expanding charters and risking further injury to BPS when the charter sector appears, at least at the high school level, to rely so heavily on cohort attrition?"

Further injury? Could you explain what you meant by that? This has what has been transpiring at BPS in parallel with the growth of charter schools:

http://www.bostonpublicschools.org/___domain/238

* * *

ADDING: Ronan also left this comment at Ravitch's site:

To save us some back and forth, I would suggest that in your response you also consider the material referenced here in respect to attempts to ascribe high test scores at charter schools to attrition:
https://dianeravitch.net/2016/09/20/george-joseph-the-racism-of-charter-school-discipline-policies/#comment-2597509
And both of these documents in respect to hypothetical financial injury to BPS schools:
http://bmrb.org/wp-content/uploads/2016/04/SR16-2Charter.pdf
http://manhattan-institute.org/html/massachusetts-charter-school-cap-pro-con-9278.html
The latter, which is brand new, states: “In 2016, charter enrollment had the effect of increasing per-pupil spending in district schools by approximately $85 million statewide.”
It has both strengths and weaknesses and you may find it, more generally, worth addressing together with the Brookings study.
You may find this useful: http://www.doe.mass.edu/research/reports/2016/02CharterReport.pdf
Finally, if you have time and space, you may wish to consider how passage or failure of our Q2 may impact likelihood of passage by the voters in 2018 of our millionaire’s tax proposal: http://www.bostonmagazine.com/news/blog/2016/05/17/what-is-millionaire-tax-massachusetts/
Best wishes.