June Board of Ed: student data trend analysis

 Thomas Kane from Harvard University
he's an economist
"in the education policy debate, I have observed that there is one common storyline...that the folks on the reform side...will marshal statistics that things are terrible...that the status quo is unsustainable...those on the other side...are often arguing similar points...and will point to prior...policy for why"
"over time it's corrosive"

provide some history
nationally, achievement as measured by NAEP rose from 1990-2013
"have made quite a bit of progress"
I am not going to try to reproduce numbers here
notes decline in 2019-22, which started pre-pandemic
in Massachusetts increase was larger, but recent losses are larger
"a year's worth of growth is about 10.4 points"
(this always feel very made up to me)
quotes Mann on education being "the balance wheel of the social machinery"
"but the balance wheel has been under strain"
Increased economic stratification 
main countervailing force has been policy: school accountability and school finance
"merged each of the schools back to neighborhood" in NAEP results
He is presuming that the schools draw from their neighborhood in this analysis--not all do
and then looked at neighborhood income
Larger increases for lower income families, by this analysis
reading improved for low and middle income families
in MA, high poverty districts improved the most
looked at states that saw big increases in the NAEP, then looked at income over time of those born in those states; states with large NAEP growth were also places were those born those had larger income growth
["What do changes in state test scores imply for later life outcomes" Dec. 2022 NBER working paper]
"I just can't help myself; I'm an economist." on next set of data
what has the increase in achievement been worth?
for one cohort of 8th graders in 2013, he calculates at $6.4B
"Education reform did not fail."

Current challenges: respond to pandemic losses; resume progress in MA education
"would encourage us to be bold" in current challenges

"pandemic losses increased inequality in MA"
IMPORTANT POINT: THIS WAS NOT THE CONCLUSION DRAWN BY DESE IN THEIR OWN PRESENTATIONS. If you recall, during the goal setting conversation, the Department was quite clear that it was not the traditionally lowest performing districts in which those losses were concentrated
Would recommend policy changes that focus on these traditionally underperforming districts

"what are some of the processes that we could create that would help accelerate improvement?"
process by which improvement could be scaled up: MA state innovation fund

Characterize this as a pep talk
"can make progress, have made progress...we currently have some major challenges"
consider what is possible in the coming years

Craven: some people voted against 1993 education
"people argue that nothing but money matters"
you said policy matters; what do you mean by that?
Kane: "there is research...looked at what happened to NAEP scores" in schools like MA pre-NCLB accountability
which is also when we put money in, so this is not holding money aside
They found large positive effects, especially in math
"more narrowing in schools that did school finance reform in the states that didn't"
"I don't think the data enables us to show which was more important, but that both were important"
Moriarty: steps one through three; I think steps 4 and 5 are extremely problematic
(this is on his slide of "resuming improvement")
Kane: I don't think any state does 4 and 5 well (that's "share" and "scale)
thinks the best way to convince people that their peer districts had compelling outcomes
"as a persuasion scaling exercise"
Hills: when you see declining gaps reverse themselves, would be curious what items you want to highlight
Kane: very few things we understand in education research, but the turnaround in math "was related to school closures"
high poverty, high minority districts "stayed closed longer"
Newton students had more resources outside of school
both Kane and Hills live in Newton
Hills: ten years ago to five years ago, how did trends reverse themselves?
Kane: resource, incentives, but also persuasion; "my hypothesis--I can't prove it!--is we'd start to see improvement again"
Plankey: was there a tapering off in other states over this time, too? "Does ed reform just sort of expire?"
Kane: "pattern of rising and then flattening is pretty universal"
Plankey: are there any states that implemented a change and reversed?
Kane: "I hope that Massachusetts would be the first."
Have to acknowledge large losses in places like Lynn
"we'll learn a lot when MCAS results come back this summer"
hopes that Massachusetts has turned it around, but don't expect it
Tutwiler: what do you mean by spread?
Kane: districts voluntarily adopting effective practices, ideally
"but state also has levels" that it can pull 
"imagine if the state held like an annual meeting" on what is going to be tested this year and what will be tested this next year
"some of the things that get tried won't work...I wouldn't be surprised if even most of them don't work"
"by identifying that 20% that worked and spreading them"
Tutwiler: appreciate naming the differences between different communities
lived in Andover, worked in Lynn and saw those differences
there are elements that exclude the control factor "there could be wisdom in those control factors..in things for us to think about"
schools that had long-term subs, or positions that they never filled, or chronic absenteeism rates that are much higher
Kane: have research on what works in chronic absenteeism, for example
start systematically sharing what works
Stewart: strikes me that when Commissioner was new was a conference sharing what works
districts to share their successes
can target a few initiatives, but don't think the state needs to share the work and be excited about it
Kane: the role the state could play is set up the things to try, and then let the folks who did it be the ones to try and then share
"and for the state to invest substantially in it...even setting aside 1% of that" for experimentation
Mohammed: thinking about experience in health care: preventable medical errors
"it wasn't until hospitals really started to work on this using many of the processes you're describing"
might be things to learn from different industries, storytelling, measurement of outcomes was key to changing trajectory of things

Plankey: catalogue of learning supports
will catalogue be updated yearly as things change? (Riley nods)