Fundamental question about PBE – where’s the research?

This is long. But I swear there’s a point. And, I hope, a clear and decent analogy to illustrate it.

So I, and I assume many others, keep wondering and asking — is there any evidence-based, unbiased research out there that definitively shows that standards-based, competency-based, proficiency-based education (PBE) leads to better outcomes for our kids?

The response is often the sound of crickets or a quick deflection and redirection of the conversation to discuss just how much the traditional system is failing. I’m not one to doubt that things can be improved in education. I believe that there’s room for improvement in everything. What I hate is when faulty logic is used or when someone jumps to the cause and provides a solution before doing the research to back it up. Or when you ask a question about any aspect of it, you’re dismissively told that you’re just confused, and then they start to explain, but the explanation doesn’t answer the actual question. And the icing on the cake is when the purported solution magically creates an industry that can profit from it — and kids, teachers and parents essentially become guinea pigs without input.

In doing more research, I’ve run across an old meeting for RSU5’s board where some members of the administration try to explain why there was/is a push to PBE. It highlights my exact concern.

The meeting in question (though I’m sure there are others that talk about PBE implementation — before and after this one), shows how people can seem like they’re answering your questions and concerns, and present a valid solution based on research… but they’re not. And this is a pattern. I’ve asked in an email to the board, and asked in board meetings (as others have), for people to point to ANY evidence-based research that is unbiased about PBE. Anything that can justify overhauling a whole system that affects everyone and every aspect of teaching, and I’ve gotten no response.

In this RSU5 board meeting that took place on January 8, 2014, at 1:11:11 mark, Joe Makley (the curriculum director for RSU5 at that time) shows a slide in his presentation that illustrates my concern about what and how proponents of PBE sell their solution. The presentation was about PBE and grading, and its current status and why it’s being done (here’s a link to the minutes from that meeting: http://www.rsu5.org/userfiles/-12/My%20Files/Minutes%20Archive/2014-01-08.pdf%3Fid=1161).

Here’s a screenshot of that slide:

PBE “research”

Looking at this slide and listening what he says while it’s up (video is linked at the bottom of this post), it comes across as thought PBE has all sorts of validated research. He leaves it to the audience to infer this, because he never actually says this. And it’s presented as a big joke (lots of laughter) — that education has essentially been researched to death. So of course there needs to be no more research — we have all the answers. But if you listen, he never says PBE methods or implementation have been researched. At all. He talks about what people say the deficiencies are in the existing methods of education. And he really only highlights a couple concerns. And then he jumps right to PBE being the solution. This is called an assumption. First — that the deficiencies he highlights really are key issues that need to be addressed (and ignores that there may be any other possible causes or concerns). This is not research. And if all this research has been done on this, and he just fails to mention it, that’s still a red flag. What is presented is a hypothesis — and to me, it’s a pretty vague one (“Our education system is totally flawed and our students are not being educated in the best way possible.”). And from that, they expect us to draw a definitive conclusion — that PBE methods can solve, or be a big chunk of a solution to, our problem.

It’s been bugging me for months that this sort of fundamental flaw in reasoning is never explicitly addressed. That we’ve claimed we have an issue (and even though I think most would agree that education has issues everywhere — the degree to which we do and where and what aspects are flawed is all up for debate) — and that we have a valid, all encompassing solution.* We’re dismissed with lots of “you’re just parents and you don’t understand education.” Or the one that I love — that we’re just not open to change. Personally, I love change. I’ve moved about 20 times in my life. I’ve changed jobs at least a dozen times — and even whole career tracks a few times. I don’t run from change. But I RESEARCH all my changes. I can’t guarantee I won’t make decisions that I’ll question a bit later — but if I did my due diligence, I’ll shake my head and know I did everything I could have beforehand to be sure I wanted to make the change. And I won’t regret that I made it (because I did the research and didn’t make a hasty decision) — just that I wasn’t able to predict everything and end up with the specific outcome I wanted.

What I see happening here, and throughout Maine (and possibly other locations), is a jumping to a conclusion based on not doing or reviewing enough unbiased research. And it triggered a memory for me. I’ve seen this happen in a similar way, but the subject matter was a bit different. It was still about what was best for the public, though. Follow me through to the end of my analogy and you’ll start to see parallels.

In Tucson, AZ, there were lots of car crashes at major intersections. It was a concern for everyone. No one wants to be in an accident, or stuck in traffic when someone else crashes. No one wants to deal with insurance companies or be physically injured or killed. So, out of concern for public safety, Tucson really wanted to fix this problem. The problem was easy to identify — crashes at intersections. And magically — a solution was proposed. Red-light cameras! We would catch the culprits running red lights and fine them and that would change their behavior. And others would stop running red lights out of fear they’d be caught and have to pay fines. And the number of collisions would drop. And, lucky for us — there was a company that was ready to provide the solution! They told us this would solve the problem and collisions would decrease. And fewer collisions would mean public safety would increase.

Sound familiar? We’ve got a problem. We assume a cause — it’s obvious, it’s the red-light runners. There’s no need to do the research. We find a solution provider that’s conveniently waiting in the wings. We can ponder why they were ready from the moment a proposed solution was put forth and draw our own conclusions. But, lucky for us, the provider of the solution is ready to “help” us. For a fee. Of course. But they’re doing this for the greater good — not to benefit themselves.

So, Tucson contracted with a private company to install red-light cameras at intersections with high incidences of collisions. Now, when someone ran a red light, they got a ticket. Lots of tickets got issued. Traffic schools raked in money (first offenses in a 2 year period could be erased if you passed a class), though they claim they saw no additional funds for this type of offense (though, they likely had a lot more students). The people who installed the cameras raked in money (they were catching offenders and got paid for their good work). But, you know what happened? We still had collisions. And Tucson ended up removing the cameras — and the collision rate didn’t change much at all during any of this. So, while they may have seen a decrease in red-light runners, the goal and purported benefit of installing the cameras, to reduce the number of collisions, was never actually achieved. They jumped to a conclusion (i.e., red-light running is the major cause of collisions), a solution (install cameras that will scare people into not running red lights), and they assumed it would have our desired outcome — there would be fewer collisions.

Who benefited? Not the public. We still had collisions at roughly the same rate (though, it was somewhat disputed, but highlights the fact that data validity is so dependent upon who’s doing the research, how it’s gathered and who’s interpreting it). And we now had more fines. Many people initially just ignored tickets. There was a loophole in the law — if the ticket came by mail and wasn’t served to you by a representative of the court, it could be legally ignored. Research before implementing red-light cameras and how they would assess fines might have highlighted that problem. But that wasn’t done before half the city just ignored and tore up tickets that came by mail. Then there were people who fought their tickets (claiming some cameras had bad timing triggers — and in some instances, that was actually true) — but that costs everyone money with associated court costs, legal fees, time lost at work, etc. Again, the general public is still not benefiting. Collision rates haven’t markedly improved.

The problem was no, or poorly done, research. Maybe it was the fact that the arrow lights at intersections turned green after everyone in your direction had stopped (in most places in the country, people get a green arrow at the beginning — in Tucson, it was after). Maybe it was because these intersections were huge (3 straight lanes and 2 left-turning lanes — in all directions). Maybe it was because many of these roads allow you to approach a light going 40+ mph and people often went 50+ mph leading up to them. Maybe it was red-light runners — but it’s only a certain subgroup of red-light runners — people who don’t care if they get a ticket and continue to run red lights, even with the cameras. The point is, no one knew. Because no one did sufficient research. They saw a problem, drew a conclusion, and some group that saw a way to make a buck had the solution.

See a similarity? What I keep seeing in RSU5 is that people claim they’ve done the research. But it’s not necessarily the right research, or good research. We’re paying consultants to lead discussions of 20 parents — where they ask the same muddled questions that have gone out in a survey. And they claim that the survey questions are good and that’s how they get “good data.” Re-asking the same question in an different way can be good. Asking the same open-ended question with zero specificity, 5 different ways and have those be the only questions, is NOT good research. If we’re going to pay to try to get valid input, go to someone who was NEVER part of the system (try to remove bias), and have them conduct real research. Poor data collection methods lead to pointless data. Getting data just to say you have data does not instill confidence in anyone who understands real research methodology. Don’t do a survey with parents who haven’t really been told about PBE and ask “what could be improved?” and expect us to believe that because 90% of the parents don’t specifically cite PBE as a concern, that that means that 90% of parents therefore must approve of PBE and that only 10% of parents have concerns. Don’t ask 11-14 year-old kids to put their name on a survey and answer only yes or no to questions about how much they like or don’t like something about their school or education — and then draw the conclusion that something is overwhelmingly a success. First, they don’t know any other system. Second, they may not answer honestly — with an administrator or teacher hovering around and seeing their answer next to their name. Third… really, is anything in this world that cut and dry that you can feel like your thoughts and feelings are clearly represented when you’re answering someone else’s questions — with only yes or no being the accepted answers? Sure, a few questions — force a yes or no — but ALL the questions? And we wonder why half the country isn’t speaking to the other half — we force everyone to choose one side or the other and there’s zero middle ground or room for discussion. Someone’s always right or wrong or left or right. There is absolutely no way you’ll convince me that data coming from that kind of survey does anything other than illustrate the beliefs of the person who constructed it.

One thing I did find in my quest for real research on PBE… it appears the state of Michigan wants to explore standards-based education. You know what they did? They asked a journalist, with no skin in the game, to do some research and write a report as to what he found. And Maine is cited throughout it.

COMPETENCY-BASED EDUCATION: An Overview for Michigan’s Superintendents (Source: http://piedmontpen.com/CBE_Final.pdf)

You know what Michigan did with that report? Michigan has since decided to fund 3-year pilot programs… https://www.michigan.gov/mde/0,4615,7-140-28753_65803-322532–,00.html  So, they did a little research and decided they needed to do MORE research before completely overhauling their whole state’s educational system.

That’s the sort of approach I wish we’d take. Or like Yarmouth did. Measured. We’re certainly taking our time implementing it. But, what I’m not convinced of — that we took any real time researching it to know IF it’s the right direction.

Below is the board meeting video from January 8th, 2014. PBE discussions start about midway through, with a lot of the key discussions starting around 1:09.

Also note that in the discussion about PBE, Joe references Derek Pierce (Casco Bay High School’s principal) and says something to the effect that Derek explains things about PBE in a way that makes sense. I’d like to think that proponents of PBE who are administrators are unbiased and truly behind this because they believe that PBE really is an improvement (even if they don’t feel the need to demand the type of unbiased research I want before supporting it). But when I did some quick research, I find that Derek Pierce won money for his school ($100k) from Nellie Mae (the same group that funds the other groups that we pay for help in implementing PBE in our schools) just 8 months after this RSU5 meeting where Joe talks about David as though David can explain PBE in such a way that we’d all believe PBE is the solution. I just find it not that surprising that someone who sings the praises and sells the benefits of a program that someone is selling… wins an award and their school gets a much needed $100k grant. From one of the major funders of one of the organizations our schools are paying to help implement PBE.

*Final note: One of the things I’ve seen repeatedly in the extensive research on education, is that there is at least one uncontested correlation when it comes to educational outcomes and kids. It is a combination of family income and poverty and how that can also limit access to a decent educational system (i.e., schools often have fewer resources when in “poor” neighborhoods because schools are predominantly funded through taxes from their area). If a school has no textbooks, kids can’t take them home to study. If a kid is hungry, he can’t focus. If a kid doesn’t have a parent who can watch him or help with homework, or encourage him… then that’s an educational hurdle that can be too large to overcome. And it’s not about “good parenting.” If you’re making minimum wage and working 2 jobs just to try to keep a roof over your family’s heads, you simply can’t be there for your kid’s homework. You can’t pay for them to go to programs that will help them with their homework. The only consistent thing that appears in almost all education research is that kids in more affluent schools, with parents with more income, do better. It’s by no means the sole factor, or difference, in levels of educational success — but it’s a huge one. And PBE does nothing to address this. It can actually make things worse — because PBE relies on kids to push themselves. Or kids to use computers or other methods to demonstrate proficiency. If your family doesn’t have a computer, or doesn’t have internet access or doesn’t have the means to send you to a college class or you don’t have access to a car so you can’t go out and do research for a project or… your means of demonstrating your proficiency are limited compared to other students before you even start. Your buddy may have 5 different ways to do something and you’ve only got 2. PBE doesn’t even the playing field for kids by giving them more options — it shows them even more ways that not everyone starts at the same level. And it unfairly asks them to all try to achieve the same goal in the end, even though some kids end up having many more ways of doing it than others. And somehow, teachers are supposed to keep things “fair” and be objective, even though the design of the system itself, and its inherent and touted flexibility, basically gives advantages to some kids over others.

Please help get the word out to others:

2 Comments

  1. I am an emerita professor of education at USM, where I taught research courses for over twenty years and recently completed a research textbook. As I explain to my students, the only way to demonstrated the effects ticeness of a program or practice is to conduct a control group study in which a sample of students is divided into two subgroups, one of which receives the new approach and one of which receives business as usual. The outcomes of the two groups are compared on the same measure and a statisrical procedure determines whether and to what degree there are differences. In
    My review of published research, I could find no evidence of any such study having been conducted of spmeyhing that goes by the name of PBL or PBE. Anecdotal studies do not meet this standard of valid research.

  2. Yep… this follows what I studied in undergrad and grad school (sociological research). But, at this point, I’d be happy to entertain any research that might not have strict control groups — as long as those funding and running the research explained the limitations (instead of trying to pretend there aren’t any deficiencies in the research). Specifically… it’s likely impossible to get a true control group because this is public education and being able to treat a whole class as an experiment and another as a control group has all sorts inherent issues (parental approval, ensuring/controlling for comparable student populations by abilities and incomes, the obvious knowledge of participants about who is in what group, etc.). Can you imagine if they did this right and had to go through the IRB process??? If I could just see some research that is conducted by a group with no link back to someone or some company that has a financial stake in the game, I’d be less cynical about the value of its results. And I’d be fine if it’s about other flavors of PBL/PBE — standards or competency-based education. The concepts don’t seem to be that different and it’s certainly not like we have consistency in Maine with how PBL/PBE is being rolled out. Every district can pretty much do what they want and define “proficiency” however they choose. So, even assuming any research that’s been done on PBL/PBE would be relevant/applicable across the board in this one state would be crazy. I’m open to any real research that even touches the concepts in any of these flavors of education reform. And I’m still having a tough time finding anything, even when I lower the bar.

Comments are closed.