2015 Accountability Levels: everything's made up, but the points matter

It's time once again for the release of the Massachusetts statewide accountability levels!
The big news for Worcester yesterday is that Burncoat Prep, in its first year that it was eligible to do so, left Level 4 status. It should be noted that both of our previous Level 4 schools, Chandler Elementary and Union Hill, are Level 1 this year for closing achievement gaps. Elm Park remains Level 4 (it was always going to, as turnaround plans are for three years).
Statewide, the big news was that Madison Park Vocational in Boston has been declared Level 4; that, despite having a turnaround plan already.

You can find all of the school levels on this list from DESE. Schools have to have a PPI) of 75 or higher for both their entire student population and their subgroups in order to be Level 1. (You don't have to have all high scores to be Level 1, just break 75 on both.) Slip on either, or on any subgroups, and you're Level 2.
This gets a little weird when the state starts trumpeting districts that are all Level 1 and 2, as it did yesterday. A subgroup has to have 25 students to "count." Thus a school that has (for example) 23 ELL students doesn't have an ELL subgroup. Likewise with any such subgroup. Beyond the myriad of other ways in which this system is stacked, this is another one in which a few kids in your student population can change all the results.
This gets even more odd when it comes to Level 3 this year. Level 3 schools are the bottom 20% of schools in the state. Because there will always be a bottom 20%, for every school that goes up a level, one must come down a level. However, schools that took the PARCC this past year were held harmless--they could not slip a level down. Thus the only schools that newly fell into Level 3 were schools that took the MCAS, and their chances of doing so (since they weren't--I hate using this verb but it is sadly accurate--competing against all schools, but only against those taking MCAS) were proportionally that much greater.
And note that this will be even more the case this coming year, assuming that more schools switch to PARCC: only MCAS schools can fall, so proportionally more MCAS schools will fall.

Last year, I ran through the way in which Level 4 schools are declared, pointing out that this isn't based on lowest test scores, lowest PPI, or any such thing, but solely at the Commissioner's discretion. It took some digging, since they've moved these charts over to the Executive Office of Education's website* (which is kind of a mess, to be kind), but I found the spreadsheet that lays this all out again. If you remember from last year, this is the data that rank orders schools based on their combined PPI's (the full student body and the subgroups). This year, the column with the percentile rankings is column O. Thus the following is based off sorting by column O.
First, there are 309 schools that have "insufficient data" to have a percentile ranking. This includes all of the early childhood centers (if you only teach kids in grade 2 and younger, you don't have test data), but it also includes a good number of full elementary schools and high schools, and it has a mix of schools that took MCAS and took PARCC. I don't know what that means.
But again, the schools that are lowest in percentages are not the ones that are, or are being declared, Level 4. There are schools that are at 1%; Madison Park is at 2%. And again, that isn't a one-year thing; PPI is based on multiple years of data.
No doubt we'll hear on Tuesday that the Commissioner has been "very concerned" and probably "for some time" on Madison Park; note that Southbridge is also on the agenda, and the lowest percentage any Southbridge school has is a 3 (there are 28 schools that are 1's and 2's).

All of which is to say, again: this isn't a clean, statistical system. That was true of Level 4's and 5's already; it's now true even of Level 3's.
And that makes it pretty important that under the Every Student Succeeds Act that President Obama signed today, states are only required to do something (and what is left fairly vague) for their bottom 5% of schools.
It's time for Massachusetts to change the law.


________________________
*to find it: EoE website>>Departments and Boards>>Board of Elementary and Secondary Education>>A-Z Programs and Topics>>Accountability>>Reports>>Accountability Lists, Materials, and Tools>>Lists (which is the 2015 Excel spreadsheet)
Really. You don't want to know how long it took me to find that.

CPPAC notes on testing

Dave Perda and Amanda Kershaw from the Data and Accountability 
Perda: this is "perhaps a transition year" on testing

PARCC could be a replacement for the MCAS; Board votes in the spring

"it can be helpful to take a look back"

retrospective: in 1994, state's requirement for graduation was one year of U.S. History and four years of gym (there were local requirements)

MEAP provided no results at school or student level, only at district level; were not connected to standards, as there were none

prior to MCAS, development of curriculum frameworks

first administration of MCAS was in 1998; class of 2003 was first to need to pass MCAS to graduate

"so here we are twenty years later and we're still talking about the same thing"

parent/guardian report from schools on MCAS results for each student taking the test, along with school level data

also information on student growth: student growth percentile 

comparison to your child's school, district, and state
item analysis on testing; breakdown of item by item on how your child did
Q: do they compare students and schools to where they should be?
Perda: MCAS is a standards-based test, thus compared to what each kid should know/be able to do
not a norm-referenced test, comparing student to population of test-takers
information about how schools are doing, as well: not just MCAS, graduation rates and dropout rates as well
"we all know that there are lots of other things that are condusive to a child education"
(test scores) "you gotta kind of take it with a grain of salt"
can be difficulty of interpretation when things are not standardized, so the test are at least that
School and district accountability system
"they have this complicated formula that look at all the different parts of MCAS in the school" computes the PPI: Progress and Performance Index
factors in the trends over time; also whether they met their target over the year
percentile for schools: if you're 20 or below, you're considered a Level 3 school (within your grade range)
"tell the community some crude numbers around the schools"
Q: these levels are a bit misleading...if I were a school and I were a B student, and I stayed a B student three years in a row, I could be a Level 2; a D student going up to a C- could be a Level 1. "too much weight put on these numbers"
"That's an important observation...new calculation includes student growth"
"The guidance documents for this are an inch think"
"If you're into it, look beyond this crude leveling system...look beyond the levels."
"the problem with any system is they just aren't perfect"
"You can get a better feel in your heart and in your gut by going into your child's school than you will by any of this."
Q: Is a better measurement SAT or MCAS?
Perda: "Or neither"
Doctors take lots of measurement when you go to the doctor; you need more information, this is just like taking your blood pressure.
More information on MCAS and released information on doe.mass.edu here
Q: trying to figure out how to get my son ready for PARCC
"They're kind of leaning that way"
"It's supposed to be an improvement on the MCAS, not everyone agrees. It's out there getting smacked around in the policy arena now."
Some of the thinking was that every state is doing their own thing, more or less, possibly not such a great idea; trying to get more kids taking the same assessment
Q: PARCC was originally supposed to be an online test. Access to technology limited. Online question doesn't translate to a paper question
Going to computers for kids with a lot of different issues, lot to ask a kid "who doesn't even know how to type yet" Is anyone seeing these issues?
Perda: yes, eventually will all be online.
Kershaw:  state needs to make sure the information is worth the time away from instruction
Q: will John & Abigail Adams scholarships still be tied to test scores if test changes?
You know what? We have no idea. I haven't even heard this question asked. Let's find out.
resources on PARCC 
making sure teachers know which standards are crucial
making sure teachers know which habits of thinking are stressed
Performance-based assessment in ELA assesses ability to analyze text and write effectively. In math, multi-step problems using abstract reasoning and strategy
End of year assessment in ELA will focus on reading comprehension. In math, ability to demonstate "further understanding of the major content and additional and supporting content of the grade level/course and demonstrate fluency (in applicable grades)."
"some students have trouble with all of these difference tools and all of these different options"
"they have to refine all of these different elements of the tests"
Q: "we do way too much testing...is some of this going to go away?"
"Worcester, A City that Reads never reads a novel in school."
"we need to look at the district's system of tests...and what do we use them for, and what are the trade-offs"