online learning Archives - The Hechinger Report https://hechingerreport.org/tags/online-learning/ Covering Innovation & Inequality in Education Wed, 18 Sep 2024 19:42:29 +0000 en-US hourly 1 https://hechingerreport.org/wp-content/uploads/2018/06/cropped-favicon-32x32.jpg online learning Archives - The Hechinger Report https://hechingerreport.org/tags/online-learning/ 32 32 138677242 An AI tutor helped Harvard students learn more physics in less time https://hechingerreport.org/proof-points-ai-tutor-harvard-physics/ Mon, 16 Sep 2024 10:00:00 +0000 https://hechingerreport.org/?p=103689

We are still in the early days of understanding the promise and peril of using generative AI in education. Very few researchers have evaluated whether students are benefiting, and one well-designed study showed that using ChatGPT for math actually harmed student achievement.  The first scientific proof I’ve seen that ChatGPT can actually help students learn […]

The post An AI tutor helped Harvard students learn more physics in less time appeared first on The Hechinger Report.

]]>
A student’s view of PS2 Pal, the AI tutor used in a learning experiment inside Harvard’s physics department. (Screenshot courtesy of Gregory Kestin)

We are still in the early days of understanding the promise and peril of using generative AI in education. Very few researchers have evaluated whether students are benefiting, and one well-designed study showed that using ChatGPT for math actually harmed student achievement

The first scientific proof I’ve seen that ChatGPT can actually help students learn more was posted online earlier this year. It’s a small experiment, involving fewer than 200 undergraduates.  All were Harvard students taking an introductory physics class in the fall of 2023, so the findings may not be widely applicable. But students learned more than twice as much in less time when they used an AI tutor in their dorm compared with attending their usual physics class in person. Students also reported that they felt more engaged and motivated. They learned more and they liked it. 

A paper about the experiment has not yet been published in a peer-reviewed journal, but other physicists at Harvard University praised it as a well-designed experiment. Students were randomly assigned to learn a topic as usual in class, or stay “home” in their dorm and learn it through an AI tutor powered by ChatGPT. Students took brief tests at the beginning and the end of class, or their AI sessions, to measure how much they learned. The following week, the in-class students learned the next topic through the AI tutor in their dorms, and the AI-tutored students went back to class. Each student learned both ways, and for both lessons – one on surface tension and one on fluid flow –  the AI-tutored students learned a lot more. 

To avoid AI “hallucinations,” the tendency of chatbots to make up stuff that isn’t true, the AI tutor was given all the correct solutions. But other developers of AI tutors have also supplied their bots with answer keys. Gregory Kestin, a physics lecturer at Harvard and developer of the AI tutor used in this study, argues that his effort succeeded while others have failed because he and his colleagues fine-tuned it with pedagogical best practices. For example, the Harvard scientists instructed this AI tutor to be brief, using no more than a few sentences, to avoid cognitive overload. Otherwise, he explained, ChatGPT has a tendency to be “long-winded.”

The tutor, which Kestin calls “PS2 Pal,” after the Physical Sciences 2 class he teaches, was told to only give away one step at a time and not to divulge the full solution in a single message. PS2 Pal was also instructed to encourage students to think and give it a try themselves before revealing the answer. 

Unguided use of ChatGPT, the Harvard scientists argue, lets students complete assignments without engaging in critical thinking. 

Kestin doesn’t deliver traditional lectures. Like many physicists at Harvard, he teaches through a method called “active learning,” where students first work with peers on in-class problem sets as the lecturer gives feedback. Direct explanations or mini-lectures come after a bit of trial, error and struggle. Kestin sought to reproduce aspects of this teaching style with the AI tutor. Students toiled on the same set of activities and Kestin fed the AI tutor the same feedback notes that he planned to deliver in class.

Kestin provocatively titled his paper about the experiment, “AI Tutoring Outperforms Active Learning,” but in an interview he told me that he doesn’t mean to suggest that AI should replace professors or traditional in-person classes. 

“I don’t think that this is an argument for replacing any human interaction,” said Kestin. “This allows for the human interaction to be much richer.”

Kestin says he intends to continue teaching through in-person classes, and he remains convinced that students learn a lot from each other by discussing how to solve problems in groups. He believes the best use of this AI tutor would be to introduce a new topic ahead of class – much like professors assign reading in advance. That way students with less background knowledge won’t be as behind and can participate more fully in class activities. Kestin hopes his AI tutor will allow him to spend less time on vocabulary and basics and devote more time to creative activities and advanced problems during class.

Of course, the benefits of an AI tutor depend on students actually using it. In other efforts, students often didn’t want to use earlier versions of education technology and computerized tutors. In this experiment, the “at-home” sessions with PS2 Pal were scheduled and proctored over Zoom. It’s not clear that even highly motivated Harvard students will find it engaging enough to use regularly on their own initiative. Cute emojis – another element that the Harvard scientists prompted their AI tutor to use – may not be enough to sustain long-term interest. 

Kestin’s next step is to test the tutor bot for an entire semester. He’s also been testing PS2 Pal as a study assistant with homework. Kestin said he’s seeing promising signs that it’s helpful for basic but not advanced problems. 

The irony is that AI tutors may not be that effective at what we generally think of as tutoring. Kestin doesn’t think that current AI technology is good at anything that requires knowing a lot about a person, such as what the student already learned in class or what kind of explanatory metaphor might work.

“Humans have a lot of context that you can use along with your judgment in order to guide a student better than an AI can,” he said. In contrast, AI is good at introducing students to new material because you only need “limited context” about someone and “minimal judgment” for how best to teach it. 

Contact staff writer Jill Barshay at (212) 678-3595 or barshay@hechingerreport.org.

This story about an AI tutor was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post An AI tutor helped Harvard students learn more physics in less time appeared first on The Hechinger Report.

]]>
103689
OPINION: It’s finally time to put pandemic excuses behind us and hold students to higher standards https://hechingerreport.org/opinion-its-finally-time-to-put-pandemic-excuses-behind-us-and-hold-students-to-higher-standards/ https://hechingerreport.org/opinion-its-finally-time-to-put-pandemic-excuses-behind-us-and-hold-students-to-higher-standards/#comments Tue, 10 Sep 2024 05:00:00 +0000 https://hechingerreport.org/?p=103515

The pandemic disrupted education in previously unimaginable ways. It limited testing and pushed schools toward remote learning and easier assignments, along with softer grading and a more relaxed attitude around attendance. These accommodations were supposed to be short-term, but most are still with us and are having a negative impact on students. This needs to […]

The post OPINION: It’s finally time to put pandemic excuses behind us and hold students to higher standards appeared first on The Hechinger Report.

]]>

The pandemic disrupted education in previously unimaginable ways. It limited testing and pushed schools toward remote learning and easier assignments, along with softer grading and a more relaxed attitude around attendance.

These accommodations were supposed to be short-term, but most are still with us and are having a negative impact on students. This needs to change.

That’s why, as parents nationwide help their children settle into school this fall, they may want to ask questions about whether their kids are ready to dive into grade-level work — and, if not, find out what is being done to address that.

Four and a half years after the start of the pandemic, it’s time to raise the bar and stop making excuses for sagging achievement. Newly released data show that student growth in 2023-24 lagged behind pre-pandemic achievement levels in nearly every grade. That data follows the big declines in reading and math scores on the most recent Nation’s Report Card and the release of a study showing that high-needs districts have been recovering from the pandemic more slowly than their wealthier counterparts, worsening long-standing achievement gaps.

The pandemic also led to an explosion in chronic absenteeism, and we’ve seen only modest improvements. A recent study by USC researchers found a lack of concern about the issue among parents. School leaders also aren’t as worried as you’d expect, with only 15 percent saying they were “extremely concerned” about student absences in a survey released by the National Center for Education Statistics.

Related: Become a lifelong learner. Subscribe to our free weekly newsletter to receive our comprehensive reporting directly in your inbox. 

At the same time, we can see clear evidence of grade inflation in rising GPAs coupled with falling or flat test scores. And while I know that teachers are trying to be supportive, lowering expectations is harmful; recent research shows that students learn more from teachers who have higher grading standards.

However, the need to raise standards doesn’t just rest on the shoulders of teachers. Education leaders and policymakers are also making things too easy. After decades of raising the bar for what’s considered grade-level performance, several states have lowered their “cut scores,” or what it means to be deemed proficient on end-of-year achievement tests.

Many states are also cutting back on K-12 assessments and graduation requirements, despite the fact that they are critical to holding education systems accountable.

Even students don’t like the go-easy-on-them approach. In an op-ed for the Baltimore Sun, recent high school graduate Benjamin Handelman notes that what is more helpful is for teachers to show enthusiasm for the subjects they teach and offer rigorous and engaging learning opportunities.

That’s important for all students, but especially for those from historically marginalized groups, who are least likely to get interesting, high-level learning opportunities.

Related: PROOF POINTS: Why are kids still struggling in school four years after the pandemic?

Keeping the bar low is going to make our kids less competitive when they leave school. It shocks me every time I hear people say, “Well, if everyone is behind, then no one is really behind.”

Eventually, young people will compete for jobs that aren’t going to have lower standards. In fact, employers will likely have higher expectations than a decade ago given advances in generative AI, the impact of technological advances on the world of work and a growing demand for employees with strong analytical, problem-solving and interpersonal skills.

Progress over time is central to our lives. When I was growing up, my competitive swimming coach was a former world record holder and Olympian. The time she had needed to be the fastest in the world in the 200-meter butterfly in 1963 was just barely fast enough for her daughter to qualify for the U.S. Olympic trials 30 years later.

We cannot be complacent about the fact that math achievement for 13-year-olds has fallen to levels not seen since the 1990s. That’s why I’m glad there are states and systems holding kids to high expectations. We can learn from them.

In Maryland, Superintendent of Schools Carey Wright has pledged to raise rigor, much like she did in Mississippi, which made major achievement gains under her stewardship. Her strategy, emulated by others, centers around raising standards and implementing evidence-based instructional strategies, most notably in reading. Mississippi is among three states, along with Illinois and Louisiana, where research shows that students have returned to pre-pandemic achievement levels in reading. Additional strategies adopted by Illinois and Louisiana include tutoring and interventions for struggling learners and professional development for educators.

These states show us that all students can succeed when challenged and supported with high expectations and opportunities to learn. That must be what we strive for to help all kids finally put the pandemic behind them.

Lesley Muldoon is the executive director of the National Assessment Governing Board, which oversees the Nation’s Report Card. She previously served as chief operating officer of the Partnership for Assessment of Readiness for College and Careers.

This story about post-pandemic grade-level work was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s weekly newsletter.

The post OPINION: It’s finally time to put pandemic excuses behind us and hold students to higher standards appeared first on The Hechinger Report.

]]>
https://hechingerreport.org/opinion-its-finally-time-to-put-pandemic-excuses-behind-us-and-hold-students-to-higher-standards/feed/ 2 103515
Students aren’t benefiting much from tutoring, one new study shows https://hechingerreport.org/proof-points-tutoring-research-nashville/ https://hechingerreport.org/proof-points-tutoring-research-nashville/#comments Mon, 09 Sep 2024 10:00:00 +0000 https://hechingerreport.org/?p=103555

Matthew Kraft, an associate professor of education and economics at Brown University, was an early proponent of giving tutors — ordinarily a luxury for the rich — to the masses after the pandemic. The research evidence was strong; more than a hundred studies had shown remarkable academic gains for students who were frequently tutored every […]

The post Students aren’t benefiting much from tutoring, one new study shows appeared first on The Hechinger Report.

]]>

Matthew Kraft, an associate professor of education and economics at Brown University, was an early proponent of giving tutors — ordinarily a luxury for the rich — to the masses after the pandemic. The research evidence was strong; more than a hundred studies had shown remarkable academic gains for students who were frequently tutored every week at school. Sometimes, they caught up two grade levels in a single year. 

After Covid shuttered schools in the spring of 2020, Kraft along with a small group of academics lobbied the Biden administration to urge schools to invest in this kind of intensive tutoring across the nation to help students catch up from pandemic learning losses. Many schools did — or tried to do so. Now, in a moment of scholarly honesty and reflection, Kraft has produced a study showing that tutoring the masses isn’t so easy — even with billions of dollars from Uncle Sam. 

The study, which was posted online in late August 2024, tracked almost 7,000 students who were tutored in Nashville, Tennessee, and calculated how much of their academic progress could be attributed to the sessions of tutoring they received at school between 2021 and 2023. Kraft and his research team found that tutoring produced only a small boost to reading test scores, on average, and no improvement in math. Tutoring failed to lift course grades in either subject.

“These results are not as large as many in the education sector had hoped,” said Kraft in an interview. That’s something of an academic understatement. The one and only positive result for students was a tiny fraction of what earlier tutoring studies had found.

“I was and continue to be incredibly impressed with the rigorous and wide body of evidence that exists for tutoring and the large average effects that those studies produced,” said Kraft. “I don’t think I paid as much attention to whether those tutoring programs were as applicable to post-Covid era tutoring at scale.”

Going forward, Kraft said he and other researchers need to “recalibrate” or adjust expectations around the “eye-popping” or very large impacts that previous small-scale tutoring programs have achieved.

Kraft described the Nashville program as “multiple orders of magnitude” larger than the pre-Covid tutoring studies. Those were often less than 50 students, while some involved a few hundred. Only a handful included over 1,000 students. Nashville’s tutoring program reached almost 7,000 students, roughly 10 percent of the district’s student population. 

Tennessee was a trailblazer in tutoring after the pandemic. State lawmakers appropriated extra funding to schools to launch large tutoring programs, even before the Biden administration urged schools around the nation to do the same with their federal Covid recovery funds. Nashville partnered with researchers, including Kraft, to study its ramp up and outcomes for students to help advise on improvements along the way. 

As with the launching of any big new program, Nashville hit a series of snags. Early administrators were overwhelmed with “14 bazillion emails,” as educators described them to researchers in the study, before they hired enough staff to coordinate the tutoring program. They first tried online tutoring. But too much time and effort was wasted setting kids up on computers, coping with software problems, and searching for missing headphones. Some children had to sit in the hallway with their tablets and headphones; it was hard to concentrate. 

Meanwhile, remote tutors were frustrated by not being able to talk with teachers regularly. Often there was redundancy with tutors being told to teach topics identical to what the students were learning in class. 

The content of the tutoring lessons was in turmoil, too. The city scrapped its math curriculum midway. Different grades required different reading curricula. For each of them, Nashville educators needed to create tutor guides and student workbooks from scratch.

Eventually the city switched course and replaced its remote tutors, who were college student volunteers, with teachers at the school who could tutor in-person. That eliminated the headaches of troublesome technology. Also, teachers could adjust the tutoring lessons to avoid repeating exactly what they had taught in class. 

But school teachers were fewer in number and couldn’t serve as many students as an army of remote volunteers. Instead of one tutor for each student, teachers worked with three or four students at a time. Even after tripling and quadrupling up, there weren’t enough teachers to tutor everyone during school hours. Half the students had their tutoring sessions scheduled immediately before or right after school.

In interviews, teachers said they enjoyed the stronger relationships they were building with their students. But there were tradeoffs. The extra tutoring work raised concerns about teacher burnout.

Despite the flux, some things improved as the tutoring program evolved. The average number of tutoring sessions that students attended increased from 16 sessions in the earlier semesters to 24 sessions per semester by spring of 2023. 

Why the academic gains for students weren’t stronger is unclear. One of Kraft’s theories is that Nashville asked tutors to teach grade-level skills and topics, similar to what the children were also learning in their classrooms and what the state tests would assess. But many students were months, even years behind grade level, and may have needed to learn rudimentary skills before being able to grasp more advanced topics. (This problem surprised me because I thought the whole purpose of tutoring was to fill in missing skills and knowledge!) In the data, average students in the middle of the achievement distribution showed the greatest gains from Nashville’s tutoring program. Students at the bottom and top didn’t progress much, or at all. (See the graph below.)

“What’s most important is that we figure out what tutoring programs and design features work best for which students,” Kraft said. 

Average students in the middle of the achievement distribution gained the most from Nashville’s tutoring program, while students who were the most behind did not catch up much

Source: Kraft, Matthew A., Danielle Sanderson Edwards, and Marisa Cannata. (2024). The Scaling Dynamics and Causal Effects of a District-Operated Tutoring Program.

Another reason for the disappointing academic gains from tutoring may be related to the individualized attention that many students were also receiving at Nashville’s schools. Tutoring often took place during frequently scheduled periods of “Personalized Learning Time” for students, and even students not selected for tutoring received other instruction during this period, such as small-group work with a teacher or individual services for children with special needs. Another set of students was assigned independent practice work using advanced educational software that adapts to a student’s level. To demonstrate positive results in this study, tutoring would have had to outperform all these other interventions. It’s possible that these other interventions are as powerful as tutoring. Earlier pre-Covid studies of tutoring generally compared the gains against those of students who had nothing more than traditional whole class instruction. That’s a starker comparison. (To be sure, one would still have hoped to see stronger results for tutoring as the Nashville program migrated outside of school hours; students who received both tutoring and personalized learning time should have meaningfully outperformed students who had only the personalized learning time.)

Other post-pandemic tutoring research has been rosier. A smaller study of frequent in-school tutoring in Chicago and Atlanta, released in March 2024, found giant gains for students in math, enough to totally undo learning losses for the average student. However, those results were achieved by only three-quarters of the roughly 800 students who had been assigned to receive tutoring and actually attended sessions.*

Kraft argued that schools should not abandon tutoring just because it’s not a silver bullet for academic recovery after Covid. “I worry,” he said, “that we may excuse ourselves from the hard work of iterative experimentation and continuous improvement by saying that we didn’t get the eye-popping results that we had hoped for right out of the gate, and therefore it’s not the solution that we should continue to invest in.”

Iteratively is how the business world innovates too. I’m a former business reporter, and this rocky effort to bring tutoring to schools reminds me of how Levi’s introduced custom-made jeans for the masses in the 1990s. These “personal pairs” didn’t cost much more than traditional mass-produced jeans, but it was time consuming for clerks to take measurements, often the jeans didn’t fit and reorders were a hassle. Levi’s pulled the plug in 2003. Eventually it brought back custom jeans — truly bespoke ones made by a master tailor at $750 or more a pop. For the masses? Maybe not. 

I wonder if customized instruction can be accomplished at scale at an affordable price. To really help students who are behind, tutors will need to diagnose each student’s learning gaps, and then develop a customized learning plan for each student. That’s pricey, and maybe impossible to do for millions of students all over the country. 

*Correction: An earlier version of this story incorrectly stated how many students were assigned to receive tutoring in the Chicago and Atlanta experiment. Only 784 students were to be tutored out of 1,540 students in the study. About three-quarters of those 784 students received tutoring. The sentence was also revised to clarify which students’ math outcomes drove the results.

This story about tutoring research was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Students aren’t benefiting much from tutoring, one new study shows appeared first on The Hechinger Report.

]]>
https://hechingerreport.org/proof-points-tutoring-research-nashville/feed/ 2 103555
Kids who use ChatGPT as a study assistant do worse on tests https://hechingerreport.org/kids-chatgpt-worse-on-tests/ https://hechingerreport.org/kids-chatgpt-worse-on-tests/#comments Mon, 02 Sep 2024 10:00:00 +0000 https://hechingerreport.org/?p=103317

Does AI actually help students learn? A recent experiment in a high school provides a cautionary tale.  Researchers at the University of Pennsylvania found that Turkish high school students who had access to ChatGPT while doing practice math problems did worse on a math test compared with students who didn’t have access to ChatGPT. Those […]

The post Kids who use ChatGPT as a study assistant do worse on tests appeared first on The Hechinger Report.

]]>

Does AI actually help students learn? A recent experiment in a high school provides a cautionary tale. 

Researchers at the University of Pennsylvania found that Turkish high school students who had access to ChatGPT while doing practice math problems did worse on a math test compared with students who didn’t have access to ChatGPT. Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning. 

A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better. Students who just did their practice problems the old fashioned way — on their own — matched their test scores.

The researchers titled their paper, “Generative AI Can Harm Learning,” to make clear to parents and educators that the current crop of freely available AI chatbots can “substantially inhibit learning.” Even a fine-tuned version of ChatGPT designed to mimic a tutor doesn’t necessarily help.

The researchers believe the problem is that students are using the chatbot as a “crutch.” When they analyzed the questions that students typed into ChatGPT, students often simply asked for the answer. Students were not building the skills that come from solving the problems themselves. 

ChatGPT’s errors also may have been a contributing factor. The chatbot only answered the math problems correctly half of the time. Its arithmetic computations were wrong 8 percent of the time, but the bigger problem was that its step-by-step approach for how to solve a problem was wrong 42 percent of the time. The tutoring version of ChatGPT was directly fed the correct solutions and these errors were minimized.

A draft paper about the experiment was posted on the website of SSRN, formerly known as the Social Science Research Network, in July 2024. The paper has not yet been published in a peer-reviewed journal and could still be revised. 

This is just one experiment in another country, and more studies will be needed to confirm its findings. But this experiment was a large one, involving nearly a thousand students in grades nine through 11 during the fall of 2023. Teachers first reviewed a previously taught lesson with the whole classroom, and then their classrooms were randomly assigned to practice the math in one of three ways: with access to ChatGPT, with access to an AI tutor powered by ChatGPT or with no high-tech aids at all. Students in each grade were assigned the same practice problems with or without AI. Afterwards, they took a test to see how well they learned the concept. Researchers conducted four cycles of this, giving students four 90-minute sessions of practice time in four different math topics to understand whether AI tends to help, harm or do nothing.

ChatGPT also seems to produce overconfidence. In surveys that accompanied the experiment, students said they did not think that ChatGPT caused them to learn less even though they had. Students with the AI tutor thought they had done significantly better on the test even though they did not. (It’s also another good reminder to all of us that our perceptions of how much we’ve learned are often wrong.)

The authors likened the problem of learning with ChatGPT to autopilot. They recounted how an overreliance on autopilot led the Federal Aviation Administration to recommend that pilots minimize their use of this technology. Regulators wanted to make sure that pilots still know how to fly when autopilot fails to function correctly. 

ChatGPT is not the first technology to present a tradeoff in education. Typewriters and computers reduce the need for handwriting. Calculators reduce the need for arithmetic. When students have access to ChatGPT, they might answer more problems correctly, but learn less. Getting the right result to one problem won’t help them with the next one.

This story about using ChatGPT to practice math was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Kids who use ChatGPT as a study assistant do worse on tests appeared first on The Hechinger Report.

]]>
https://hechingerreport.org/kids-chatgpt-worse-on-tests/feed/ 2 103317
Researchers combat AI hallucinations in math https://hechingerreport.org/proof-points-combat-ai-hallucinations-math/ Mon, 26 Aug 2024 10:00:00 +0000 https://hechingerreport.org/?p=103071

One of the biggest problems with using AI in education is that the technology hallucinates. That’s the word the artificial intelligence community uses to describe how its newest large language models make up stuff that doesn’t exist or isn’t true. Math is a particular land of make-believe for AI chatbots. Several months ago, I tested […]

The post Researchers combat AI hallucinations in math appeared first on The Hechinger Report.

]]>
Two University of California, Berkeley, researchers documented how they tamed AI hallucinations in math by asking ChatGPT to solve the same problem 10 times. Credit: Eugene Mymrin/ Moment via Getty Images

One of the biggest problems with using AI in education is that the technology hallucinates. That’s the word the artificial intelligence community uses to describe how its newest large language models make up stuff that doesn’t exist or isn’t true. Math is a particular land of make-believe for AI chatbots. Several months ago, I tested Khan Academy’s chatbot, which is powered by ChatGPT. The bot, called Khanmigo, told me I had answered a basic high school Algebra 2 problem involving negative exponents wrong. I knew my answer was right. After typing in the same correct answer three times, Khanmigo finally agreed with me. It was frustrating.

Errors matter. Kids could memorize incorrect solutions that are hard to unlearn, or become more confused about a topic. I also worry about teachers using ChatGPT and other generative AI models to write quizzes or lesson plans. At least a teacher has the opportunity to vet what AI spits out before giving or teaching it to students. It’s riskier when you’re asking students to learn directly from AI. 

Computer scientists are attempting to combat these errors in a process they call “mitigating AI hallucinations.” Two researchers from University of California, Berkeley, recently documented how they successfully reduced ChatGPT’s instructional errors to near zero in algebra. They were not as successful with statistics, where their techniques still left errors 13 percent of the time. Their paper was published in May 2024 in the peer-reviewed journal PLOS One.

In the experiment, Zachary Pardos, a computer scientist at the Berkeley School of Education, and one of his students, Shreya Bhandari, first asked ChatGPT to show how it would solve an algebra or statistics problem. They discovered that ChatGPT was “naturally verbose” and they did not have to prompt the large language model to explain its steps. But all those words didn’t help with accuracy. On average, ChatGPT’s methods and answers were wrong a third of the time. In other words, ChatGPT would earn a grade of a D if it were a student. 

Current AI models are bad at math because they’re programmed to figure out probabilities, not follow rules. Math calculations are all about rules. It’s ironic because earlier versions of AI were able to follow rules, but unable to write or summarize. Now we have the opposite.

The Berkeley researchers took advantage of the fact that ChatGPT, like humans, is erratic. They asked ChatGPT to answer the same math problem 10 times in a row. I was surprised that a machine might answer the same question differently, but that is what these large language models do.  Often the step-by-step process and the answer were the same, but the exact wording differed. Sometimes the methods were bizarre and the results were dead wrong. (See an example in the illustration below.)

Researchers grouped similar answers together. When they assessed the accuracy of the most common answer among the 10 solutions, ChatGPT was astonishingly good. For basic high-school algebra, AI’s error rate fell from 25 percent to zero. For intermediate algebra, the error rate fell from 47 percent to 2 percent. For college algebra, it fell from 27 percent to 2 percent. 

ChatGPT answered the same algebra question three different ways, but it landed on the right response seven out of 10 times in this example

Source: Pardos and Bhandari, “ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills,” PLOS ONE, May 2024

However, when the scientists applied this method, which they call “self-consistency,” to statistics, it did not work as well. ChatGPT’s error rate fell from 29 percent to 13 percent, but still more than one out of 10 answers was wrong. I think that’s too many errors for students who are learning math. 

The big question, of course, is whether these ChatGPT’s solutions help students learn math better than traditional teaching. In a second part of this study, researchers recruited 274 adults online to solve math problems and randomly assigned a third of them to see these ChatGPT’s solutions as a “hint” if they needed one. (ChatGPT’s wrong answers were removed first.) On a short test afterwards, these adults improved 17 percent, compared to less than 12 percent learning gains for the adults who could see a different group of hints written by undergraduate math tutors. Those who weren’t offered any hints scored about the same on a post-test as they did on a pre-test.

Those impressive learning results for ChatGPT prompted the study authors to boldly predict that “completely autonomous generation” of an effective computerized tutoring system is “around the corner.” In theory, ChatGPT could instantly digest a book chapter or a video lecture and then immediately turn around and tutor a student on it.

Before I embrace that optimism, I’d like to see how much real students – not just adults recruited online – use these automated tutoring systems. Even in this study, where adults were paid to do math problems, 120 of the roughly 400 participants didn’t complete the work and so their results had to be thrown out. For many kids, and especially students who are struggling in a subject, learning from a computer just isn’t engaging

This story about AI hallucinations was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Researchers combat AI hallucinations in math appeared first on The Hechinger Report.

]]>
103071
PROOF POINTS: New studies of online tutoring highlight troubles with attendance and larger tutoring groups https://hechingerreport.org/proof-points-studies-online-tutoring-troubles-attendance-larger-groups/ Mon, 15 Jul 2024 10:00:00 +0000 https://hechingerreport.org/?p=101900

Ever since the pandemic shut down schools in the spring of 2020, education researchers have pointed to tutoring as the most promising way to help kids catch up academically. Evidence from almost 100 studies was overwhelming for a particular kind of tutoring, called high-dosage tutoring, where students focus on either reading or math three to […]

The post PROOF POINTS: New studies of online tutoring highlight troubles with attendance and larger tutoring groups appeared first on The Hechinger Report.

]]>

Ever since the pandemic shut down schools in the spring of 2020, education researchers have pointed to tutoring as the most promising way to help kids catch up academically. Evidence from almost 100 studies was overwhelming for a particular kind of tutoring, called high-dosage tutoring, where students focus on either reading or math three to five times a week.  

But until recently, there has been little good evidence for the effectiveness of online tutoring, where students and tutors interact via video, text chat and whiteboards. The virtual version has boomed since the federal government handed schools nearly $190 billion of pandemic recovery aid and specifically encouraged them to spend it on tutoring. Now, some new U.S. studies could offer useful guidance to educators.

Online attendance is a struggle

In the spring of 2023, almost 1,000 Northern California elementary school children in grades 1 to 4 were randomly assigned to receive online reading tutoring during the school day. Students were supposed to get 20 to 30 sessions each, but only one of five students received that much. Eighty percent didn’t and they didn’t do much better than the 800 students in the comparison group who didn’t get tutoring, according to a draft paper by researchers from Teachers College, Columbia University, which was posted to the Annenberg Institute website at Brown University in April 2024. (The Hechinger Report is an independent news organization based at Teachers College, Columbia University.)

Researchers have previously found that it is important to schedule in-person tutoring sessions during the school day, when attendance is mandatory. The lesson here with online tutoring is that attendance can be rocky even during the school day. Often, students end up with a low dose of tutoring instead of the high dose that schools have paid for.

However, online tutoring can be effective when students participate regularly. In this Northern California study, reading achievement increased substantially, in line with in-person tutoring, for the roughly 200 students who got at least 20 sessions across 10 weeks.

The students who logged in regularly might have been more motivated students in the first place, the researchers warned, indicating that it could be hard to reproduce such large academic benefits for all. During the periods when children were supposed to receive tutoring, researchers observed that some children – often ones who were slightly higher achieving –  regularly logged on as scheduled while others didn’t. The difference in student behavior and what the students were doing instead wasn’t explained. Students also seemed to log in more frequently when certain staff members were overseeing the tutoring and less frequently with others. 

Small group tutoring doesn’t work as well online

The large math and reading gains that researchers documented in small groups of students with in-person tutors aren’t always translating to the virtual world. 

Another study of more than 2,000 elementary school children in Texas tested the difference between one-to-one and two-to-one online tutoring during the 2022-23 school year. These were young, low-income children, in kindergarten through 2nd grade, who were just learning to read. Children who were randomly assigned to get one-to-one tutoring four times a week posted small gains on one test, but not on another, compared to students in a comparison group who didn’t get tutoring. First graders assigned to one-to-one tutoring gained the equivalent of 30 additional days of school. By contrast, children who had been tutored in pairs were statistically no different in reading than the comparison group of untutored children. A draft paper about this study, led by researchers from Stanford University, was posted to the Annenberg website in May 2024. 

Another small study in Grand Forks, North Dakota confirmed the downside of larger groups with online tutoring. Researchers from Brown University directly compared the math progress of middle school students when they received one-to-one tutoring versus small groups of three students. The study was too small, only 180 students, to get statistically strong results, but the half that were randomly assigned to receive individual tutoring appeared to gain eight extra percentile points, compared to the students who were assigned to small group tutoring. It was possible that students in the small groups learned a third as much math, the researchers estimated, but these students might have learned much less. A draft of this paper was posted to the Annenberg website in June 2024. 

In surveys, tutors said it was hard to keep all three kids engaged online at once. Students were more frequently distracted and off-task, they said.  Shy students were less likely to speak up and participate.  With one student at a time, tutors said they could move at a faster pace and students “weren’t afraid to ask questions” or “afraid of being wrong.” (On the plus side, tutors said groups of three allowed them to organize group activities or encourage a student to help a peer.)

Behavior problems happen in person too. However, when I have observed in-person small group tutoring in schools, each student is often working independently with the tutor, almost like three simultaneous sessions of one-to-one help. In-person tutors can   encourage a student to keep practicing through a silent glance, a smile or hand signal even as they are explaining something to another student. Online, each child’s work and mistakes are publicly exposed on the screen to the whole group. Private asides aren’t as easy; some platforms allow the tutor to text a child privately in a chat window, but that takes time. Tutors have told me that many teens don’t like seeing their face on screen, but turning the camera off makes it harder for them to sense if a student is following along or confused.

Matt Kraft, one of the Brown researchers on the Grand Forks study, suggests that bigger changes need to be made to online tutoring lessons in order to expand from one-to-one to small group tutoring, and he notes that school staff are needed in the classroom to keep students on-task. 

School leaders have until March 2026 to spend the remainder of their $190 billion in pandemic recovery funds, but contracts with tutoring vendors must be signed by September 2024. Both options — in person and virtual — involve tradeoffs. New research evidence is showing that virtual tutoring can work well, especially when motivated students want the tutoring and log in regularly. But many of the students who are significantly behind grade level and in need of extra help may not be so motivated. Keeping the online tutoring small, ideally one-to-one, improves the chances that it will be effective. But that means serving many fewer students, leaving millions of children behind. It’s a tough choice. 

This story about online tutoring was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post PROOF POINTS: New studies of online tutoring highlight troubles with attendance and larger tutoring groups appeared first on The Hechinger Report.

]]>
101900
PROOF POINTS: This is your brain. This is your brain on screens https://hechingerreport.org/proof-points-neuroscience-paper-v-screens-reading/ https://hechingerreport.org/proof-points-neuroscience-paper-v-screens-reading/#comments Mon, 24 Jun 2024 10:00:00 +0000 https://hechingerreport.org/?p=101581

Studies show that students of all ages, from elementary school to college, tend to absorb more when they’re reading on paper rather than screens. The advantage for paper is a small one, but it’s been replicated in dozens of laboratory experiments, particularly when students are reading about science or other nonfiction texts. Experts debate why […]

The post PROOF POINTS: This is your brain. This is your brain on screens appeared first on The Hechinger Report.

]]>
One brain study, published in May 2024, detected different electrical activity in the brain after students had read a passage on paper, compared with screens. Credit: Getty Images

Studies show that students of all ages, from elementary school to college, tend to absorb more when they’re reading on paper rather than screens. The advantage for paper is a small one, but it’s been replicated in dozens of laboratory experiments, particularly when students are reading about science or other nonfiction texts.

Experts debate why comprehension is worse on screens. Some think the glare and flicker of screens tax the brain more than ink on paper. Others conjecture that students have a tendency to skim online but read with more attention and effort on paper. Digital distraction is an obvious downside to screens. But internet browsing, texting or TikTok breaks aren’t allowed in the controlled conditions of these laboratory studies.

Neuroscientists around the world are trying to peer inside the brain to solve the mystery. Recent studies have begun to document salient differences in brain activity when reading on paper versus screens. None of the studies I discuss below is definitive or perfect, but together they raise interesting questions for future researchers to explore. 

One Korean research team documented that young adults had lower concentrations of oxygenated hemoglobin in a section of the brain called the prefrontal cortex when reading on paper compared with screens. The prefrontal cortex is associated with working memory and that could mean the brain is more efficient in absorbing and memorizing new information on paper, according to a study published in January 2024 in the journal Brain Sciences. An experiment in Japan, published in 2020, also noticed less blood flow in the prefrontal cortex when readers were recalling words in a passage that they had read on paper, and more blood flow with screens.

But it’s not clear what that increased blood flow means. The brain needs to be activated in order to learn and one could also argue that the extra brain activation during screen reading could be good for learning. 

Instead of looking at blood flow, a team of Israeli scientists analyzed electrical activity in the brains of 6- to 8-year-olds. When the children read on paper, there was more power in high-frequency brainwaves. When the children read from screens, there was more energy in low-frequency bands. 

The Israeli scientists interpreted these frequency differences as a sign of better concentration and attention when reading on paper. In their 2023 paper, they noted that attention difficulties and mind wandering have been associated with lower frequency bands – exactly the bands that were elevated during screen reading. However, it was a tiny study of 15 children and the researchers could not confirm whether the children’s minds were actually wandering when they were reading on screens. 

Another group of neuroscientists in New York City has also been looking at electrical activity in the brain. But instead of documenting what happens inside the brain while reading, they looked at what happens in the brain just after reading, when students are responding to questions about a text. 

The study, published in the peer-reviewed journal PLOS ONE in May 2024, was conducted by neuroscientists at Teachers College, Columbia University, where The Hechinger Report is also based. My news organization is an independent unit of the college, but I am covering this study just like I cover other educational research. 

In the study, 59 children, aged 10 to 12, read short passages, half on screens and half on paper. After reading the passage, the children were shown new words, one at a time, and asked whether they were related to the passage they had just read. The children wore stretchy hair nets embedded with electrodes. More than a hundred sensors measured electrical currents inside their brains a split second after each new word was revealed.

For most words, there was no difference in brain activity between screens and paper. There was more positive voltage when the word was obviously related to the text, such as the word “flow” after reading a passage about volcanoes. There was more negative voltage with an unrelated word like “bucket,” which the researchers said was an indication of surprise and additional brain processing. These brainwaves were similar regardless of whether the child had read the passage on paper or on screens. 

However, there were stark differences between paper and screens when it came to ambiguous words, ones where you could make a creative argument that the word was tangentially related to the reading passage or just as easily explain why it was unrelated. Take for example, the word “roar” after reading about volcanoes. Children who had read the passage on paper showed more positive voltage, just as they had for clearly related words like “flow.” Yet, those who had read the passage on screens showed more negative activity, just as they had for unrelated words like “bucket.”

For the researchers, the brainwave difference for ambiguous words was a sign that students were engaging in “deeper” reading on paper. According to this theory, the more deeply information is processed, the more associations the brain makes. The electrical activity the neuroscientists detected reveals the traces of these associations and connections. 

Despite this indication of deeper reading, the researchers didn’t detect any differences in basic comprehension. The children in this experiment did just as well on a simple comprehension test after reading a passage on paper as they did on screens. The neuroscientists told me that the comprehension test they administered was only to verify that the children had actually read the passage and wasn’t designed to detect deeper reading. I wish, however, the children had been asked to do something involving more analysis to buttress their argument that students had engaged in deeper reading on paper.

Virginia Clinton-Lisell, a reading researcher at the University of North Dakota who was not involved in this study, said she was “skeptical” of its conclusions, in part because the word-association exercise the neuroscientists created hasn’t been validated by outside researchers. Brain activation during a word association exercise may not be proof that we process language more thoroughly or deeply on paper.

One noteworthy result from this experiment is speed. Many reading experts have believed that comprehension is often worse on screens because students are skimming rather than reading. But in the controlled conditions of this laboratory experiment, there were no differences in reading speed: 57 seconds on the laptop compared to 58 seconds on paper –  statistically equivalent in a small experiment like this. And so that raises more questions about why the brain is acting differently between the two media. 

“I’m not sure why one would process some visual images more deeply than others if the subjects spent similar amounts of time looking at them,” said Timothy Shanahan, a reading research expert and a professor emeritus at the University of Illinois at Chicago. 

None of this work settles the debate over reading on screens versus paper. All of them ignore the promise of interactive features, such as glossaries and games, which can swing the advantage to electronic texts. Early research can be messy, and that’s a normal part of the scientific process. But so far, the evidence seems to be corroborating conventional reading research that something different is going on when kids log in rather than turn a page.

This story about reading on screens vs. paper was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post PROOF POINTS: This is your brain. This is your brain on screens appeared first on The Hechinger Report.

]]>
https://hechingerreport.org/proof-points-neuroscience-paper-v-screens-reading/feed/ 4 101581
OPINION: There’s a promising path to get students back on track to graduation https://hechingerreport.org/opinion-theres-a-promising-path-to-get-students-back-on-track-to-graduation/ Tue, 18 Jun 2024 05:00:00 +0000 https://hechingerreport.org/?p=101558

Rates of chronic absenteeism are at record-high levels. More than 1 in 4 students missed 10 percent or more of the 2021-22 school year. That means millions of students missed out on regular instruction, not to mention the social and emotional benefits of interacting with peers and trusted adults. Moreover, two-thirds of the nation’s students […]

The post OPINION: There’s a promising path to get students back on track to graduation appeared first on The Hechinger Report.

]]>

Rates of chronic absenteeism are at record-high levels. More than 1 in 4 students missed 10 percent or more of the 2021-22 school year. That means millions of students missed out on regular instruction, not to mention the social and emotional benefits of interacting with peers and trusted adults.

Moreover, two-thirds of the nation’s students attended a school where chronic absence rates reached at least 20 percent. Such levels disrupt entire school communities, including the students who are regularly attending.

The scope and scale of this absenteeism crisis necessitate the implementation of the next generation of student support.

Fortunately, a recent study suggests a promising path for getting students back in school and back on track to graduation. A group of nearly 50 middle and high schools saw reductions in chronic absenteeism and course failure rates after one year of harnessing the twin powers of data and relationships.

From the 2021-22 to 2022-23 school years, the schools’ chronic absenteeism rates dropped by 5.4 percentage points, and the share of students failing one or more courses went from 25.5 percent to 20.5 percent. In the crucial ninth grade, course failure rates declined by 9.2 percentage points.

These encouraging results come from the first cohort of rural and urban schools and communities partnering with the GRAD Partnership, a collective of nine organizations, to grow  the use of “student success systems” into a common practice.

Student success systems take an evidence-based approach to organizing school communities to better support the academic progress and well-being of all students.

They were developed with input from hundreds of educators and build on the successes of earlier student support efforts — like early warning systems and on-track initiatives — to meet students’ post-pandemic needs.

Related: Widen your perspective. Our free biweekly newsletter consults critical voices on innovation in education.

Importantly, student success systems offer schools a way to identify school, grade-level and classroom factors that impact attendance; they then deliver timely supports to meet individual students’ needs. They do this, in part, by explicitly valuing supportive relationships and responding to the insights that students and the adults who know them bring to the table.

Valuable relationships include not only those between students and teachers, and schools and families, but also those among peer groups and within the entire school community. Schools cannot address the attendance crisis without rebuilding and fostering these relationships.

When students feel a sense of connection to school they are more likely to show up.

For some students, this connection comes through extracurricular activities like athletics, robotics or band. For others, it may be a different connection to school.

Schools haven’t always focused on connections in a concrete way, partly because relationships can feel fuzzy and hard to track. We’re much better at tracking things like grades and attendance.

Still, schools in the GRAD Partnership cohort show that it can be done.

These schools established “student success teams” of teachers, counselors and others. The teams meet regularly to look at up-to-date student data and identify and address the root causes of absenteeism with insight and input from families and communities, as well as the students themselves.

The teams often use low-tech relationship-mapping tools to help identify students who are disconnected from activities or mentors. One school’s student success team used these tools to ensure that all students were connected to at least one activity — and even created new clubs for students with unique interests. Their method was one that any school could replicate —collaborating on a Google spreadsheet.

Another school identified students who would benefit from a new student mentoring program focused on building trusting relationships.

Related: PROOF POINTS: The chronic absenteeism puzzle

Some schools have used surveys of student well-being to gain insight on how students feel about school, themselves and life in general — and have then used the information to develop supports.

And in an example of building supportive community relationships, one of the GRAD Partnership schools worked with local community organizations to host a resource night event at which families were connected on the spot to local providers who could help them overcome obstacles to regular attendance — such as medical and food needs, transportation and housing issues and unemployment.

Turning the tide against our current absenteeism crisis does not have a one-and-done solution — it will involve ongoing collaborative efforts guided by data and grounded in relationships that take time to build.

Without these efforts, the consequences will be severe both for individual students and our country as a whole.

Robert Balfanz is a research professor at the Center for Social Organization of Schools at Johns Hopkins University School of Education, where he is the director of the Everyone Graduates Center.

This story about post-pandemic education was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s newsletter.

The post OPINION: There’s a promising path to get students back on track to graduation appeared first on The Hechinger Report.

]]>
101558
OPINION: It’s not just about tech and anxiety. What are kids learning? https://hechingerreport.org/opinion-its-not-just-about-tech-and-anxiety-what-are-kids-learning/ Tue, 28 May 2024 05:00:00 +0000 https://hechingerreport.org/?p=101254

Clouds of doom continue to hover over the debate about teens’ mental health and the role of technology. This spring, the warnings come from the bestselling book “The Anxious Generation” by sociologist Jonathan Haidt. Some parents and educators are calling for a ban on smartphones and laptops in schools. Others are trying to press pause […]

The post OPINION: It’s not just about tech and anxiety. What are kids learning? appeared first on The Hechinger Report.

]]>

Clouds of doom continue to hover over the debate about teens’ mental health and the role of technology. This spring, the warnings come from the bestselling book “The Anxious Generation” by sociologist Jonathan Haidt. Some parents and educators are calling for a ban on smartphones and laptops in schools. Others are trying to press pause on the panic by pointing to research that needs a longer look.

People feel forced into binary camps of “ban tech” and “don’t ban tech.”

But there is a way to reset the conversation that could help parents, educators and kids themselves make better choices about technology. As writers and researchers who focus on the science of learning, we see a gaping hole in the debate thus far. The problem is that decision-makers keep relying on only two sets of questions and data: One set focuses on questions about how youth are feeling (not so great). The other focuses on how kids are using their time (spending hours on their phones).

A third set of questions is missing and needs to be asked: What and how are children and youth learning? Is technology aiding their learning or getting in the way? Think of data on tech and learning as the third leg of the stool in this debate. Without it, we can’t find our way toward balance.

Haidt’s book focuses primarily on well-being, and it’s great that he recognizes the research on the importance of play and exploration offline to helping children’s mental health. But play and exploration are also critical for learning, and parents and educators need more examples of the many different places where learning happens, whether on screen, off screen or some hybrid of the two. Parents are at risk of becoming either too protectionist or too permissive if they don’t stop to consider whether technology is affording today’s kids opportunities to explore and stretch their minds.

Related: Become a lifelong learner. Subscribe to our free weekly newsletter to receive our comprehensive reporting directly in your inbox.

Harvard professor Michael Rich, author of the recent book “The Mediatrician’s Guide,” argues that our children are growing up in a world in which they move seamlessly between physical and digital information, with mountains of experiences and learning opportunities at their fingertips. This is their reality. Today, even children from under-resourced environments can virtually visit places that in the past were well beyond their reach.

Many parents and teachers know their kids can gain valuable skills and knowledge from using different forms of tech and media. In fact, they are already factoring in the potential for learning when they make decisions about technology. They restrict phones and laptops in certain contexts and make them available in others, depending on what they believe will provide a good learning environment for their children at different ages and stages.

Sometimes the technology, and the way kids explore and build things with it, is integral to what kids need to learn. This year, for example, students have been working in Seattle public libraries with University of Washington professor Jason Yip to build tools and games intended to help other kids identify and avoid disinformation. One game is an online maze built within the world of “Minecraft” that shows what it feels like to fall down rabbit holes of extreme information. “Digital play can open up a number of potentials that allow children to experience unknown and difficult situations, such as misinformation, and experiment with decision-making,” Yip said.

Related: Horticulture, horses and ‘Chill Rooms’: One district goes all-in on mental health support

More focus on the effect of technology on learning — good and bad — is needed at all ages. Studies of young children show that when parents are distracted by their phones, they are less able to help their kids build the language skills that are key for learning how to read. Maybe parents should model different behaviors with their phone use. Also consider a study at the University of Delaware in which researchers read books to 4-year-olds live, via video chat or in a prerecorded video. No significant differences in learning were found between the children reading live or via video chat. This study and others provide clear evidence that children can learn when people read storybooks to them online.

Instead of fighting with children over smartphone use, we should be making sure that there are enough teachers and mentors to help all kids use those phones and laptops to support learning, whether they are collaborating on science fair projects or creating video book trailers for YouTube. Kids need teachers and parents who can give them opportunities to explore, play and grapple with hard things in both the digital world and the real world.

Our society is good at creating polarization. But we don’t have to devolve into extreme “ban” or “don’t ban” positions on smartphones, laptops or other technology today.

Parents and teachers should make decisions about technology after viewing the issue from three perspectives: how much the kids are using the devices, how the devices are affecting kids’ well-being and — the missing leg — how the devices are affecting their learning. Maybe adding this new piece could even help adults see more than just an “anxious generation” but also one hungry to learn.

Kathy Hirsh-Pasek, Roberta Golinkoff and Lisa Guernsey are authors of several books on children’s learning and founders of The Learning Sciences Exchange, a fellowship program and problem-solving platform at New America that brings together experts in child development research, media and journalism, entertainment, social entrepreneurship and education leadership.

This story about teens and technology was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s newsletter.

The post OPINION: It’s not just about tech and anxiety. What are kids learning? appeared first on The Hechinger Report.

]]>
101254
PROOF POINTS: We have tried paying teachers based on how much students learn. Now schools are expanding that idea to contractors and vendors https://hechingerreport.org/proof-points-outcomes-based-contracting/ Mon, 27 May 2024 10:00:00 +0000 https://hechingerreport.org/?p=101232

Schools spend billions of dollars a year on products and services, including everything from staplers and textbooks to teacher coaching and training. Does any of it help students learn more? Some educational materials end up mothballed in closets. Much software goes unused. Yet central-office bureaucrats frequently renew their contracts with outside vendors regardless of usage […]

The post PROOF POINTS: We have tried paying teachers based on how much students learn. Now schools are expanding that idea to contractors and vendors appeared first on The Hechinger Report.

]]>

Schools spend billions of dollars a year on products and services, including everything from staplers and textbooks to teacher coaching and training. Does any of it help students learn more? Some educational materials end up mothballed in closets. Much software goes unused. Yet central-office bureaucrats frequently renew their contracts with outside vendors regardless of usage or efficacy.

One idea for smarter education spending is for schools to sign smarter contracts, where part of the payment is contingent upon whether students use the services and learn more. It’s called outcomes-based contracting and is a way of sharing risk between buyer (the school) and seller (the vendor). Outcomes-based contracting is most common in healthcare. For example, a health insurer might pay a pharmaceutical company more for a drug if it actually improves people’s health, and less if it doesn’t. 

Although the idea is relatively new in education, many schools tried a different version of it – evaluating and paying teachers based on how much their students’ test scores improved – in the 2010s. Teachers didn’t like it, and enthusiasm for these teacher accountability schemes waned. Then, in 2020, Harvard University’s Center for Education Policy Research announced that it was going to test the feasibility of paying tutoring companies by how much students’ test scores improved. 

The initiative was particularly timely in the wake of the pandemic.  The federal government would eventually give schools almost $190 billion to reopen and to help students who fell behind when schools were closed. Tutoring became a leading solution for academic recovery and schools contracted with outside companies to provide tutors. Many educators worried that billions could be wasted on low-quality tutors who didn’t help anyone. Could schools insist that tutoring companies make part of their payment contingent upon whether student achievement increased? 

The Harvard center recruited a handful of school districts who wanted to try an outcomes-based contract. The researchers and districts shared ideas on how to set performance targets. How much should they expect student achievement to grow from a few months of tutoring? How much of the contract should be guaranteed to the vendor for delivering tutors, and how much should be contingent on student performance? 

The first hurdle was whether tutoring companies would be willing to offer services without knowing exactly how much they would be paid. School districts sent out requests for proposals from online tutoring companies. Tutoring companies bid and the terms varied. One online tutoring company agreed that 40 percent of a $1.2 million contract with the Duval County Public Schools in Jacksonville, Florida, would be contingent upon student performance. Another online tutoring company signed a contract with Ector County schools in the Odessa, Texas, region that specified that the company had to accept a penalty if kids’ scores declined.

In the middle of the pilot, the outcomes-based contracting initiative moved from the Harvard center to the Southern Education Foundation, another nonprofit, and I recently learned how the first group of contracts panned out from Jasmine Walker, a senior manager there. Walker had a first-hand view because until the fall of 2023, she was the director of mathematics in Florida’s Duval County schools, where she oversaw the outcomes-based contract on tutoring. 

Here are some lessons she learned: 

Planning is time-consuming

Drawing up an outcomes-based contract requires analyzing years of historical testing data, and documenting how much achievement has typically grown for the students who need tutoring. Then, educators have to decide – based on the research evidence for tutoring –  how much they could reasonably hope student achievement to grow after 12 weeks or more. 

Incomplete data was a common problem

The first school district in the pilot group launched its outcome-based contract in the fall of 2021. In the middle of the pilot, school leadership changed, layoffs hit, and the leaders of the tutoring initiative left the district.  With no one in the district’s central office left to track it, there was no data on whether tutoring helped the 1,000 students who received it. Half the students attended 70 percent of the tutoring sessions. Half didn’t. Test scores for almost two-thirds of the tutored students increased between the start and the end of the tutoring program. But these students also had regular math classes each day and they likely would have posted some achievement gains anyway. 

Delays in settling contracts led to fewer tutored students

Walker said two school districts weren’t able to start tutoring children until January 2023, instead of the fall of 2022 as originally planned, because it took so long to iron out contract details and obtain approvals inside the districts. Many schools didn’t want to wait and launched other interventions to help needy students sooner. Understandably, schools didn’t want to yank these students away from those other interventions midyear. 

That delay had big consequences in Duval County. Only 451 students received tutoring instead of a projected 1,200.  Fewer students forced Walker to recalculate Duval’s outcomes-based contract. Instead of a $1.2 million contract with $480,000 of it contingent on student outcomes, she downsized it to $464,533 with $162,363 contingent. The tutored students hit 53 percent of the district’s growth and proficiency goals, leading to a total payout of $393,220 to the tutoring company – far less than the company had originally anticipated. But the average per-student payout of $872 was in line with the original terms of between $600 and $1,000 per student. 

The bottom line is still uncertain

What we don’t know from any of these case studies is whether similar students who didn’t receive tutoring also made similar growth and proficiency gains. Maybe it’s all the other things that teachers were doing that made the difference. In Duval County, for example, proficiency rates in math rose from 28 percent of students to 46 percent of students. Walker believes that outcomes-based contracting for tutoring was “one lever” of many. 

It’s unclear if outcomes-based contracting is a way for schools to save money. This kind of intensive tutoring – three times a week or more during the school day – is new and the school districts didn’t have previous pre-pandemic tutoring contracts for comparison. But generally, if all the student goals are met, companies stand to earn more in an outcomes-based contract than they would have otherwise, Walker said.

“It’s not really about saving money,” said Walker.  “What we want is for students to achieve. I don’t care if I spent the whole contract amount if the students actually met the outcomes, because in the past, let’s face it, I was still paying and they were not achieving outcomes.”

The biggest change with outcomes-based contracting, Walker said, was the partnership with the provider. One contractor monitored student attendance during tutoring sessions, called her when attendance slipped and asked her to investigate. Students were given rewards for attending their tutoring sessions and the tutoring company even chipped in to pay for them. “Kids love Takis,” said Walker. 

Advice for schools

Walker has two pieces of advice for schools considering outcomes-based contracts. One, she says, is to make the contingency amount at least 40 percent of the contract. Smaller incentives may not motivate the vendor. For her second outcomes-based contract in Duval County, Walker boosted the contingency amount to half the contract. To earn it, the tutoring company needs the students it is tutoring to hit growth and proficiency goals. That tutoring took place during the current 2023-24 school year. Based on mid-year results, students exceeded expectations, but full-year results are not yet in. 

More importantly, Walker says the biggest lesson she learned was to include teachers, parents and students earlier in the contract negotiation process.  She says “buy in” from teachers is critical because classroom teachers are actually making sure the tutoring happens. Otherwise, an outcomes-based contract can feel like yet “another thing” that the central office is adding to a teacher’s workload. 

Walker also said she wished she had spent more time educating parents and students on the importance of attending school and their tutoring sessions. ”It’s important that everyone understands the mission,” said Walker. 

Innovation can be rocky, especially at the beginning. Now the Southern Education Foundation is working to expand its outcomes-based contracting initiative nationwide. A second group of four school districts launched outcomes-based contracts for tutoring this 2023-24 school year. Walker says that the rate cards and recordkeeping are improving from the first pilot round, which took place during the stress and chaos of the pandemic. 

The foundation is also seeking to expand the use of outcomes-based contracts beyond tutoring to education technology and software. Nine districts are slated to launch outcomes-based contracts for ed tech this fall.  Her next dream is to design outcomes-based contracts around curriculum and teacher training. I’ll be watching. 

This story about outcomes-based contracting was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post PROOF POINTS: We have tried paying teachers based on how much students learn. Now schools are expanding that idea to contractors and vendors appeared first on The Hechinger Report.

]]>
101232