Declaring Level 4's: "Where everything's made up and the points don't matter"

Title courtesy, of course, of Whose Line Is It, Anyway?
Since we're talking about possibly expanding which schools the state gets to make decisions around, let's take a look at how we're determining which schools the state gets to intervene in now, shall we?

So, let's back up a step: all schools each year get a CPI (Composite Performance Index) for the entire school and for "high need students." If you'd like to know more about how this is calculated, you can go through the state's presentation here. The short version is that it takes four years of data (if we have them) on MCAS scores and (where applicable) dropout and graduation rates, crunches the numbers, and scales them against where schools and districts "should" be. This new system came in with our No Child Left Behind waiver; if it makes your head spin, you're not alone.
Once there is a CPI for every school in the state, the state can make a list of all of the schools in the state, like this. That gives you not only each school's "level" but also the CPI for both the school and the high needs group and the school's percentile. That same information is on this spreadsheet, and, because it's a spreadsheet, one can sort by column.
If 95% of your kids are tested, and you're hitting 75 or higher on your number crunching, you're called a Level 1. Just about everyone else is Level 2.

Now, according to the 2010 Act Relative to the Achievement Gap, Level 3 schools are the lowest 20% of schools (either by school or by subgroup) in the state; the regulations also include not improving on graduation rate or having less than 90% of kids participating in your testing. So, when you line up all of the schools in the state in rank order, and draw a line at the bottom 1/5, that's your Level 3's.
But what makes a Level 4?
Beyond their having to be in Level 3 status, the only other restriction the state law makes on Level 4 and 5 schools is that they can make up no more than 4% of the schools in the state at any one time (per section 3).
So which schools fall from 3 to 4?
The state, in their explanation of the classification of Level 3 schools (that's a PDF) says this:

The state’s lowest achieving, least improving Level 3 schools are candidates for classification into Level 4 at the discretion of the Commissioner of Elementary and Secondary Education. A Level 4 school may be classified into Level 5 by the Commissioner on behalf of the Board of Elementary and Secondary Education if it fails to improve; or if district conditions make it unlikely that the school will make significant improvement without a Level 5 designation.

(emphasis added) 
"...at the discretion of the Commissioner of Elementary and Secondary Education."

Now, as someone who's been following this all since it was actually just in a bill being rushed through the Legislature to get Race to the Top money, I had the notion that Level 4's would of course be the lowest 4% of schools in the state.
When you take the list of schools and sort them by their percentage in the state*, though, Level 4's aren't the lowest in the state. Nor, at this point, are the lowest all Level 5's. Right now, there are 13 schools that are the bottom 1% within their school type--note that the state rounds to the nearest whole number--and that includes one Level 5, one Level 3, and the rest Level 4's.
It gets even weirder if you move up. There is a Level 5 as high as the 15th percentile within its school type; a Level 4 as high as the 10th percentile.

It would be easy to dismiss this as this year's report, but those scores are calculated off of four years of information: it is by its nature a report that is designed not to be terribly volatile. And while there are some assumptions one can make--not wanting to overwhelm any one community, for example--there certainly are choices that were made on which schools would be declared Level 4 and which would not, and those decisions were not mathematical ones.

I say this not to criticize the Commissioner. An enormous degree of importance is placed on these standings, however, and the most critical ones, the ones that do the most damage to a school's image, the ones that cause the most angst in and about districts are the ones that are not based simply on a calculation (however flawed those might be). They're chosen.

So when we're making assumptions and drawing conclusions about what the levels mean, let's keep that in mind.

*That's this chart sorted by column U.