We are still in the early days of understanding the promise and peril of using generative AI in education. Very few researchers have evaluated whether students are benefiting, and one well-designed study showed that using ChatGPT for math actually harmed student achievement. 

The first scientific proof I’ve seen that ChatGPT can actually help students learn more was posted online earlier this year. It’s a small experiment, involving fewer than 200 undergraduates.  All were Harvard students taking an introductory physics class in the fall of 2023, so the findings may not be widely applicable. But students learned more than twice as much in less time when they used an AI tutor in their dorm compared with attending their usual physics class in person. Students also reported that they felt more engaged and motivated. They learned more and they liked it. 

A paper about the experiment has not yet been published in a peer-reviewed journal, but other physicists at Harvard University praised it as a well-designed experiment. Students were randomly assigned to learn a topic as usual in class, or stay “home” in their dorm and learn it through an AI tutor powered by ChatGPT. Students took brief tests at the beginning and the end of class, or their AI sessions, to measure how much they learned. The following week, the in-class students learned the next topic through the AI tutor in their dorms, and the AI-tutored students went back to class. Each student learned both ways, and for both lessons – one on surface tension and one on fluid flow –  the AI-tutored students learned a lot more. 

To avoid AI “hallucinations,” the tendency of chatbots to make up stuff that isn’t true, the AI tutor was given all the correct solutions. But other developers of AI tutors have also supplied their bots with answer keys. Gregory Kestin, a physics lecturer at Harvard and developer of the AI tutor used in this study, argues that his effort succeeded while others have failed because he and his colleagues fine-tuned it with pedagogical best practices. For example, the Harvard scientists instructed this AI tutor to be brief, using no more than a few sentences, to avoid cognitive overload. Otherwise, he explained, ChatGPT has a tendency to be “long-winded.”

The tutor, which Kestin calls “PS2 Pal,” after the Physical Sciences 2 class he teaches, was told to only give away one step at a time and not to divulge the full solution in a single message. PS2 Pal was also instructed to encourage students to think and give it a try themselves before revealing the answer. 

Unguided use of ChatGPT, the Harvard scientists argue, lets students complete assignments without engaging in critical thinking. 

Kestin doesn’t deliver traditional lectures. Like many physicists at Harvard, he teaches through a method called “active learning,” where students first work with peers on in-class problem sets as the lecturer gives feedback. Direct explanations or mini-lectures come after a bit of trial, error and struggle. Kestin sought to reproduce aspects of this teaching style with the AI tutor. Students toiled on the same set of activities and Kestin fed the AI tutor the same feedback notes that he planned to deliver in class.

Kestin provocatively titled his paper about the experiment, “AI Tutoring Outperforms Active Learning,” but in an interview he told me that he doesn’t mean to suggest that AI should replace professors or traditional in-person classes. 

“I don’t think that this is an argument for replacing any human interaction,” said Kestin. “This allows for the human interaction to be much richer.”

Kestin says he intends to continue teaching through in-person classes, and he remains convinced that students learn a lot from each other by discussing how to solve problems in groups. He believes the best use of this AI tutor would be to introduce a new topic ahead of class – much like professors assign reading in advance. That way students with less background knowledge won’t be as behind and can participate more fully in class activities. Kestin hopes his AI tutor will allow him to spend less time on vocabulary and basics and devote more time to creative activities and advanced problems during class.

Of course, the benefits of an AI tutor depend on students actually using it. In other efforts, students often didn’t want to use earlier versions of education technology and computerized tutors. In this experiment, the “at-home” sessions with PS2 Pal were scheduled and proctored over Zoom. It’s not clear that even highly motivated Harvard students will find it engaging enough to use regularly on their own initiative. Cute emojis – another element that the Harvard scientists prompted their AI tutor to use – may not be enough to sustain long-term interest. 

Kestin’s next step is to test the tutor bot for an entire semester. He’s also been testing PS2 Pal as a study assistant with homework. Kestin said he’s seeing promising signs that it’s helpful for basic but not advanced problems. 

The irony is that AI tutors may not be that effective at what we generally think of as tutoring. Kestin doesn’t think that current AI technology is good at anything that requires knowing a lot about a person, such as what the student already learned in class or what kind of explanatory metaphor might work.

“Humans have a lot of context that you can use along with your judgment in order to guide a student better than an AI can,” he said. In contrast, AI is good at introducing students to new material because you only need “limited context” about someone and “minimal judgment” for how best to teach it. 

Contact staff writer Jill Barshay at (212) 678-3595 or barshay@hechingerreport.org.

This story about an AI tutor was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post An AI tutor helped Harvard students learn more physics in less time appeared first on The Hechinger Report.

In the United States today, 9 out of 10 school superintendents are white and two-thirds are white men. When you think of a typical superintendent, the person you imagine probably doesn’t look like me. 

As a Latina, my leadership isn’t often expected, nor is it always welcome. 

Institutional biases block career advancement for educators of color, who constitute only 1 in 5 U.S. teachers and principals. We are promoted less often and experience higher turnover than our white colleagues. 

This is a serious problem: The caliber and stability of our educator workforce affects our education system’s quality and capacity for improvement. We must address these barriers: Educators of color enhance student learning and are key to closing educational gaps. 

Much has been written about why we need to break down barriers in order to diversify the educator workforce. Much less covered has been the formidable task of how to launch and sustain transformative solutions. I urge fellow superintendents from all racial, ethnic and cultural backgrounds to act now.

That’s what we are doing inWaukegan public schools in Illinois, which serve a diverse population of about 14,000 students from preschool through high school, near Lake Michigan, about 10 miles south of the Wisconsin border. I am using my leadership position to take strong, unapologetic action so that every student can graduate from high school prepared and supported to pursue their dreams. 

Related: Become a lifelong learner. Subscribe to our free weekly newsletter to receive our comprehensive reporting directly in your inbox.

Since taking on the superintendent role, I’ve found that when it comes to the young men in our district, we’ve got serious work to do. 

After analyzing a wide range of data and engaging in deep reflection last year, we realized that our Black male students often lack the necessary resources and support to reach their full potential. This aligns with national trends through which these students typically face low expectations, inequitable discipline that fuels the school-to-prison pipeline and a shortage of effective, culturally responsive teaching.

We launched an ambitious, systemwide, data-driven initiative aimed at creating equitable opportunities to help our Black male students and educators. I believe our efforts can provide an example for any school system dedicated to closing opportunity and achievement gaps for all students. 

Research confirms the intertwined success of Black students and educators. Studies show that low-income Black male students are 39 percent less likely to drop out of high school if they had at least one Black male teacher in elementary school. Our goal is to convince more Black male educators to build a career in our district because we know that hiring and retaining Black teachers and leaders can measurably improve math scores for Black students.

Related: White men have the edge in the school principal pipeline, researchers say

Some key insights from our work stand out as essential tools for continued success. First is the indispensable role of broad support from executive leadership. My commitment to addressing education inequities is deeply personal. I relate to many of the challenges our Black male educators face and, as a mother to a Black teenage boy, the urgency of this effort pulses through my veins.

Our board of education’s steadfast support has been equally key to launching our initiative, with board members helping drive us toward significant, measurable achievements.

Community engagement and leadership are our foundational principles. I know that the solutions we need won’t come from me alone. This acknowledgment led us to launch a task force that includes Black male students, teachers, principals, students’ fathers and other family members and community partners. 

We’ve also hosted planning sessions involving diverse stakeholders to try to foster buy-in and accountability as we move forward. And we’ve engaged national partners with unparalleled expertise to help us guide professional learning for district officials using an inclusive, equity-focused lens. 

We are also dedicating staff to oversee the work. We created a new position to catalyze our multiyear initiative and are investing in our teachers and leaders while we pursue systemic transformation. In particular, we launched a local leadership chapter for Men of Color in Educational Leadership, where our educators can share experiences, seek guidance and grow professionally within a community of practice.

We rely on a framework that highlights skills vital for the success of education leaders of color and contributes to the broader goal of systemic change in education. I often turn to these resources myself when reflecting on my own leadership as a woman of color. 

Acknowledging the extent of the challenge is just the start to fostering inclusive, equitable education. We have begun the critical process of setting goals so we can transparently track and communicate our progress. We are also trying to see how this focused initiative advances broader efforts to strengthen and diversify our entire educator workforce, including paraprofessionals, teachers and school leaders. 

Other superintendents can do this too. Find your champions, allies, community leaders and partners. The time for brave, visionary leadership is now.

Theresa Plascencia is superintendent of Waukegan Public Schools in Waukegan, Illinois. She sits on the Association of Latino Administrators and Superintendents Advisory Policy Committee and on the Men of Color in Educational Leadership National Advisory Council. 

This story about diversifying the educator workforce was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s weekly newsletter.

The post SUPERINTENDENT VOICE: As a Latina, my leadership sets me apart and gives me a chance to set an example appeared first on The Hechinger Report.

Matthew Kraft, an associate professor of education and economics at Brown University, was an early proponent of giving tutors — ordinarily a luxury for the rich — to the masses after the pandemic. The research evidence was strong; more than a hundred studies had shown remarkable academic gains for students who were frequently tutored every week at school. Sometimes, they caught up two grade levels in a single year. 

After Covid shuttered schools in the spring of 2020, Kraft along with a small group of academics lobbied the Biden administration to urge schools to invest in this kind of intensive tutoring across the nation to help students catch up from pandemic learning losses. Many schools did — or tried to do so. Now, in a moment of scholarly honesty and reflection, Kraft has produced a study showing that tutoring the masses isn’t so easy — even with billions of dollars from Uncle Sam. 

The study, which was posted online in late August 2024, tracked almost 7,000 students who were tutored in Nashville, Tennessee, and calculated how much of their academic progress could be attributed to the sessions of tutoring they received at school between 2021 and 2023. Kraft and his research team found that tutoring produced only a small boost to reading test scores, on average, and no improvement in math. Tutoring failed to lift course grades in either subject.

“These results are not as large as many in the education sector had hoped,” said Kraft in an interview. That’s something of an academic understatement. The one and only positive result for students was a tiny fraction of what earlier tutoring studies had found.

“I was and continue to be incredibly impressed with the rigorous and wide body of evidence that exists for tutoring and the large average effects that those studies produced,” said Kraft. “I don’t think I paid as much attention to whether those tutoring programs were as applicable to post-Covid era tutoring at scale.”

Going forward, Kraft said he and other researchers need to “recalibrate” or adjust expectations around the “eye-popping” or very large impacts that previous small-scale tutoring programs have achieved.

Kraft described the Nashville program as “multiple orders of magnitude” larger than the pre-Covid tutoring studies. Those were often less than 50 students, while some involved a few hundred. Only a handful included over 1,000 students. Nashville’s tutoring program reached almost 7,000 students, roughly 10 percent of the district’s student population. 

Tennessee was a trailblazer in tutoring after the pandemic. State lawmakers appropriated extra funding to schools to launch large tutoring programs, even before the Biden administration urged schools around the nation to do the same with their federal Covid recovery funds. Nashville partnered with researchers, including Kraft, to study its ramp up and outcomes for students to help advise on improvements along the way. 

As with the launching of any big new program, Nashville hit a series of snags. Early administrators were overwhelmed with “14 bazillion emails,” as educators described them to researchers in the study, before they hired enough staff to coordinate the tutoring program. They first tried online tutoring. But too much time and effort was wasted setting kids up on computers, coping with software problems, and searching for missing headphones. Some children had to sit in the hallway with their tablets and headphones; it was hard to concentrate. 

Meanwhile, remote tutors were frustrated by not being able to talk with teachers regularly. Often there was redundancy with tutors being told to teach topics identical to what the students were learning in class. 

The content of the tutoring lessons was in turmoil, too. The city scrapped its math curriculum midway. Different grades required different reading curricula. For each of them, Nashville educators needed to create tutor guides and student workbooks from scratch.

Eventually the city switched course and replaced its remote tutors, who were college student volunteers, with teachers at the school who could tutor in-person. That eliminated the headaches of troublesome technology. Also, teachers could adjust the tutoring lessons to avoid repeating exactly what they had taught in class. 

But school teachers were fewer in number and couldn’t serve as many students as an army of remote volunteers. Instead of one tutor for each student, teachers worked with three or four students at a time. Even after tripling and quadrupling up, there weren’t enough teachers to tutor everyone during school hours. Half the students had their tutoring sessions scheduled immediately before or right after school.

In interviews, teachers said they enjoyed the stronger relationships they were building with their students. But there were tradeoffs. The extra tutoring work raised concerns about teacher burnout.

Despite the flux, some things improved as the tutoring program evolved. The average number of tutoring sessions that students attended increased from 16 sessions in the earlier semesters to 24 sessions per semester by spring of 2023. 

Why the academic gains for students weren’t stronger is unclear. One of Kraft’s theories is that Nashville asked tutors to teach grade-level skills and topics, similar to what the children were also learning in their classrooms and what the state tests would assess. But many students were months, even years behind grade level, and may have needed to learn rudimentary skills before being able to grasp more advanced topics. (This problem surprised me because I thought the whole purpose of tutoring was to fill in missing skills and knowledge!) In the data, average students in the middle of the achievement distribution showed the greatest gains from Nashville’s tutoring program. Students at the bottom and top didn’t progress much, or at all. (See the graph below.)

“What’s most important is that we figure out what tutoring programs and design features work best for which students,” Kraft said. 

Average students in the middle of the achievement distribution gained the most from Nashville’s tutoring program, while students who were the most behind did not catch up much

Another reason for the disappointing academic gains from tutoring may be related to the individualized attention that many students were also receiving at Nashville’s schools. Tutoring often took place during frequently scheduled periods of “Personalized Learning Time” for students, and even students not selected for tutoring received other instruction during this period, such as small-group work with a teacher or individual services for children with special needs. Another set of students was assigned independent practice work using advanced educational software that adapts to a student’s level. To demonstrate positive results in this study, tutoring would have had to outperform all these other interventions. It’s possible that these other interventions are as powerful as tutoring. Earlier pre-Covid studies of tutoring generally compared the gains against those of students who had nothing more than traditional whole class instruction. That’s a starker comparison. (To be sure, one would still have hoped to see stronger results for tutoring as the Nashville program migrated outside of school hours; students who received both tutoring and personalized learning time should have meaningfully outperformed students who had only the personalized learning time.)

Other post-pandemic tutoring research has been rosier. A smaller study of frequent in-school tutoring in Chicago and Atlanta, released in March 2024, found giant gains for students in math, enough to totally undo learning losses for the average student. However, those results were achieved by only three-quarters of the roughly 800 students who had been assigned to receive tutoring and actually attended sessions.*

Kraft argued that schools should not abandon tutoring just because it’s not a silver bullet for academic recovery after Covid. “I worry,” he said, “that we may excuse ourselves from the hard work of iterative experimentation and continuous improvement by saying that we didn’t get the eye-popping results that we had hoped for right out of the gate, and therefore it’s not the solution that we should continue to invest in.”

Iteratively is how the business world innovates too. I’m a former business reporter, and this rocky effort to bring tutoring to schools reminds me of how Levi’s introduced custom-made jeans for the masses in the 1990s. These “personal pairs” didn’t cost much more than traditional mass-produced jeans, but it was time consuming for clerks to take measurements, often the jeans didn’t fit and reorders were a hassle. Levi’s pulled the plug in 2003. Eventually it brought back custom jeans — truly bespoke ones made by a master tailor at $750 or more a pop. For the masses? Maybe not. 

I wonder if customized instruction can be accomplished at scale at an affordable price. To really help students who are behind, tutors will need to diagnose each student’s learning gaps, and then develop a customized learning plan for each student. That’s pricey, and maybe impossible to do for millions of students all over the country. 

*Correction: An earlier version of this story incorrectly stated how many students were assigned to receive tutoring in the Chicago and Atlanta experiment. Only 784 students were to be tutored out of 1,540 students in the study. About three-quarters of those 784 students received tutoring. The sentence was also revised to clarify which students’ math outcomes drove the results.

This story about tutoring research was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Students aren’t benefiting much from tutoring, one new study shows appeared first on The Hechinger Report.

Does AI actually help students learn? A recent experiment in a high school provides a cautionary tale. 

Researchers at the University of Pennsylvania found that Turkish high school students who had access to ChatGPT while doing practice math problems did worse on a math test compared with students who didn’t have access to ChatGPT. Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning. 

A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better. Students who just did their practice problems the old fashioned way — on their own — matched their test scores.

The researchers titled their paper, “Generative AI Can Harm Learning,” to make clear to parents and educators that the current crop of freely available AI chatbots can “substantially inhibit learning.” Even a fine-tuned version of ChatGPT designed to mimic a tutor doesn’t necessarily help.

The researchers believe the problem is that students are using the chatbot as a “crutch.” When they analyzed the questions that students typed into ChatGPT, students often simply asked for the answer. Students were not building the skills that come from solving the problems themselves. 

ChatGPT’s errors also may have been a contributing factor. The chatbot only answered the math problems correctly half of the time. Its arithmetic computations were wrong 8 percent of the time, but the bigger problem was that its step-by-step approach for how to solve a problem was wrong 42 percent of the time. The tutoring version of ChatGPT was directly fed the correct solutions and these errors were minimized.

A draft paper about the experiment was posted on the website of SSRN, formerly known as the Social Science Research Network, in July 2024. The paper has not yet been published in a peer-reviewed journal and could still be revised. 

This is just one experiment in another country, and more studies will be needed to confirm its findings. But this experiment was a large one, involving nearly a thousand students in grades nine through 11 during the fall of 2023. Teachers first reviewed a previously taught lesson with the whole classroom, and then their classrooms were randomly assigned to practice the math in one of three ways: with access to ChatGPT, with access to an AI tutor powered by ChatGPT or with no high-tech aids at all. Students in each grade were assigned the same practice problems with or without AI. Afterwards, they took a test to see how well they learned the concept. Researchers conducted four cycles of this, giving students four 90-minute sessions of practice time in four different math topics to understand whether AI tends to help, harm or do nothing.

ChatGPT also seems to produce overconfidence. In surveys that accompanied the experiment, students said they did not think that ChatGPT caused them to learn less even though they had. Students with the AI tutor thought they had done significantly better on the test even though they did not. (It’s also another good reminder to all of us that our perceptions of how much we’ve learned are often wrong.)

The authors likened the problem of learning with ChatGPT to autopilot. They recounted how an overreliance on autopilot led the Federal Aviation Administration to recommend that pilots minimize their use of this technology. Regulators wanted to make sure that pilots still know how to fly when autopilot fails to function correctly. 

ChatGPT is not the first technology to present a tradeoff in education. Typewriters and computers reduce the need for handwriting. Calculators reduce the need for arithmetic. When students have access to ChatGPT, they might answer more problems correctly, but learn less. Getting the right result to one problem won’t help them with the next one.

This story about using ChatGPT to practice math was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Kids who use ChatGPT as a study assistant do worse on tests appeared first on The Hechinger Report.

It’s guys night out, and dinner is coming to an end. We’ve given our credit cards to the server, who returns and drops the stack of bills in the center of the table. As per usual, one of my oldest friends scoops them up and thrusts them to my side of the table.

“Lance can calculate the tips, he’s a math guy.”

The line elicits a huge laugh, as if my title as the group’s “math guy” is a comical one. We have our “movie guy,” our “soccer guy” and of course me, the one who can quickly compute reasonable tips because I’m “good at math.”

I’ve taught high school math in Texas for the last 16 years, and I wear the epithet of “math guy” proudly. What troubles me is the notion that being “good at math” is unique to those in my vocation. Often, when people find out that I teach Algebra, they quickly recite a plethora of bad experiences with math in all forms, without indicating that they could (or should) have gone down differently.

Equally troubling is the fixed idea among many of my friends that they are the opposite of me: inherently or irrevocably “bad at math,” and they believe that is totally OK.

In fact, my friends are more loath to admit that they’re bad at cooking than math. By contrast, when was the last time you heard an adult declare at a dinner table that they are bad at reading?

Illiteracy carries a large amount of stigma for many adults, whereas most seem to be far less self-conscious about having deficiencies in math. And that’s a problem we must solve as a society.

The solution to creating the next generation of strong “math people” starts early, and it involves families and schools working together to change mindsets.

Related: Become a lifelong learner. Subscribe to our free weekly newsletter to receive our comprehensive reporting directly in your inbox. 

Math, along with reading, is one of the fundamental skills we work to teach kids from a very young age. As educators, we focus on math and reading as the twin flames of childhood development, as the two progress in tandem and pave the path to academic success.

Math and reading are frequently linked when we devise legislation, whether it comes from the right or the left. Last year the Texas Legislature passed a popular bill, which put in place a plan to help more students attain access to advanced math instruction. Research shows a clear link between students’ postsecondary success and the highest level of math they took in high school. The bill gets the ball rolling by requiring school districts to automatically enroll sixth graders in an advanced math course if they performed in the 60th percentile or better on their fifth-grade state assessment or a similar local measure.

This is because students tend to find success via a cascading effect. To achieve at the postsecondary level, they need to take advanced high school math courses. To be prepared for those courses, they need access to Algebra I in eighth grade, and to be prepared for that, they need to be enrolled in advanced math in sixth grade.

Putting this bill to work will expose many students to the satisfaction of higher math just before middle school.

I was lucky to have had tangible experiences with problem-solving even younger. My mother taught me how to calculate a 20 percent tip as a child, by showing me how to move the decimal over one place to the left and double the result. I delighted in this shared family activity any time we went out to eat. My joy was only further enhanced in the elementary school classroom, where I learned the formal arithmetic that guides this technique of calculating percentages.

Sadly, few students cite a positive formative memory like mine when it comes to their own history with math, especially outside the classroom. In fact, many experience the opposite.

Related: PROOF POINTS: How a debate over the science of math could reignite the math wars

We must focus on developing an excited “growth mindset” for the learning of math, no matter how old we are, so that our children will view the subject with pleasure rather than fear.

And we need to start now. By 2030, it is estimated that over 60 percent of jobs in Texas will require at least some level of post-high school education. To get students there, we will need even more than innovative legislation.

We can lay a foundation for success in higher levels of math by showing kids the fun of math problem-solving early and often. Some examples: Have your kids sort their Halloween candy into different shapes, count the beats in their favorite songs or guess how many total Legos are in the closet.

Let’s emphasize the ability to develop numeracy with the same fervor and positivity that we emphasize literacy. Let’s stop instilling in our kids that they are either “good” or “bad” at math when they are still in single digits.

Let’s continually promote the narrative that we can all be “math people,” from grade school all the way to adulthood. That way, the next time you need to calculate a tip, or double a recipe or take measurements for a house project, you won’t have to call on the local “math guy.”

You might even take delight in solving the problem yourself.

Lance Barasch is a high school math teacher at the School of Science and Engineering in Dallas ISD. He is a Teach Plus senior writing fellow.

This story about teaching math was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s weekly newsletter.

The post TEACHER VOICE: I don’t mind being known as ‘the math guy,’ but everyone can and should be able to do the math appeared first on The Hechinger Report.

One of the biggest problems with using AI in education is that the technology hallucinates. That’s the word the artificial intelligence community uses to describe how its newest large language models make up stuff that doesn’t exist or isn’t true. Math is a particular land of make-believe for AI chatbots. Several months ago, I tested Khan Academy’s chatbot, which is powered by ChatGPT. The bot, called Khanmigo, told me I had answered a basic high school Algebra 2 problem involving negative exponents wrong. I knew my answer was right. After typing in the same correct answer three times, Khanmigo finally agreed with me. It was frustrating.

Errors matter. Kids could memorize incorrect solutions that are hard to unlearn, or become more confused about a topic. I also worry about teachers using ChatGPT and other generative AI models to write quizzes or lesson plans. At least a teacher has the opportunity to vet what AI spits out before giving or teaching it to students. It’s riskier when you’re asking students to learn directly from AI. 

Computer scientists are attempting to combat these errors in a process they call “mitigating AI hallucinations.” Two researchers from University of California, Berkeley, recently documented how they successfully reduced ChatGPT’s instructional errors to near zero in algebra. They were not as successful with statistics, where their techniques still left errors 13 percent of the time. Their paper was published in May 2024 in the peer-reviewed journal PLOS One.

In the experiment, Zachary Pardos, a computer scientist at the Berkeley School of Education, and one of his students, Shreya Bhandari, first asked ChatGPT to show how it would solve an algebra or statistics problem. They discovered that ChatGPT was “naturally verbose” and they did not have to prompt the large language model to explain its steps. But all those words didn’t help with accuracy. On average, ChatGPT’s methods and answers were wrong a third of the time. In other words, ChatGPT would earn a grade of a D if it were a student. 

Current AI models are bad at math because they’re programmed to figure out probabilities, not follow rules. Math calculations are all about rules. It’s ironic because earlier versions of AI were able to follow rules, but unable to write or summarize. Now we have the opposite.

The Berkeley researchers took advantage of the fact that ChatGPT, like humans, is erratic. They asked ChatGPT to answer the same math problem 10 times in a row. I was surprised that a machine might answer the same question differently, but that is what these large language models do.  Often the step-by-step process and the answer were the same, but the exact wording differed. Sometimes the methods were bizarre and the results were dead wrong. (See an example in the illustration below.)

Researchers grouped similar answers together. When they assessed the accuracy of the most common answer among the 10 solutions, ChatGPT was astonishingly good. For basic high-school algebra, AI’s error rate fell from 25 percent to zero. For intermediate algebra, the error rate fell from 47 percent to 2 percent. For college algebra, it fell from 27 percent to 2 percent. 

ChatGPT answered the same algebra question three different ways, but it landed on the right response seven out of 10 times in this example

However, when the scientists applied this method, which they call “self-consistency,” to statistics, it did not work as well. ChatGPT’s error rate fell from 29 percent to 13 percent, but still more than one out of 10 answers was wrong. I think that’s too many errors for students who are learning math. 

The big question, of course, is whether these ChatGPT’s solutions help students learn math better than traditional teaching. In a second part of this study, researchers recruited 274 adults online to solve math problems and randomly assigned a third of them to see these ChatGPT’s solutions as a “hint” if they needed one. (ChatGPT’s wrong answers were removed first.) On a short test afterwards, these adults improved 17 percent, compared to less than 12 percent learning gains for the adults who could see a different group of hints written by undergraduate math tutors. Those who weren’t offered any hints scored about the same on a post-test as they did on a pre-test.

Those impressive learning results for ChatGPT prompted the study authors to boldly predict that “completely autonomous generation” of an effective computerized tutoring system is “around the corner.” In theory, ChatGPT could instantly digest a book chapter or a video lecture and then immediately turn around and tutor a student on it.

Before I embrace that optimism, I’d like to see how much real students – not just adults recruited online – use these automated tutoring systems. Even in this study, where adults were paid to do math problems, 120 of the roughly 400 participants didn’t complete the work and so their results had to be thrown out. For many kids, and especially students who are struggling in a subject, learning from a computer just isn’t engaging. 

This story about AI hallucinations was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Researchers combat AI hallucinations in math appeared first on The Hechinger Report.

Four years after the pandemic shuttered schools, we all want to be done with COVID. But the latest analyses from three assessment companies paint a grim picture of where U.S. children are academically and that merits coverage. While there are isolated bright spots, the general trend is stagnation. 

One report documented that U.S. students did not make progress in catching up in the most recent 2023-24 school year and slid even further behind in math and reading, exacerbating pandemic learning losses.

“At the end of 2021-22, we optimistically concluded that the worst was behind us and that recovery had begun,” wrote Karyn Lewis, a researcher at NWEA, one of the assessment companies.  “Unfortunately, data from the past two school years no longer support this conclusion. Growth has slowed to lag pre-pandemic rates, resulting in achievement gaps that continue to widen, and in some cases, now surpass what we had previously deemed as the low point.” 

The starkest example is eighth grade students, who were in fourth grade when the pandemic first erupted in March of 2020. They now need nine months of additional school to catch up, according to NWEA’s analysis, released in July 2024.  “This is a crisis moment with middle schoolers,” said Lewis. “Where are we going to find an additional year to make up for these kiddos before they leave the education system?” 

All three analyses were produced by for-profit companies that sell assessments to schools. Teachers or parents may be familiar with them by the names of their tests:  MAP, i-Ready and Star. Unlike annual state tests, these interim assessments are administered at least twice a year to millions of students around the nation to help track progress, or learning, during the year. These companies may have a business motive in sounding an alarm to sell more of their products, but the reports are produced by well-regarded education statisticians.

Curriculum Associates did not detect as much deterioration as NWEA, but did find widespread stagnation in 2023-24, according to a report released on August 19, 2024. Their researcher Kristen Huff described the numerical differences as tiny ones that have to do with the fact that these are different tests, taken by different students and use different methods for crunching the numbers. The main takeaway from all the reports, she said, is the same. “As a nation, we are still seeing the lasting impact of the disruption to schooling and learning,” said Huff, vice president of assessment and research at Curriculum Associates. 

In short, children remain behind and haven’t recovered. That matters for these children’s future employment prospects and standard of living. Ultimately, a less productive labor force could hamper the U.S. economy, according to projections from economists and consulting firms. 

It’s important to emphasize that individual students haven’t regressed or don’t know less now than they used to. The average sixth grader knows more today in 2024 than he or she did in first grade in 2019. But the pace of learning, or rate of academic growth, has been rocky since 2020, with some students missing many months of instruction. Sixth graders in 2024, on average, know far less than sixth graders did back in 2019. 

Renaissance, a third company, found a mottled pattern of recovery, stagnation and deterioration depending upon the grade and the subject. (The company shared its preliminary mid-year results with me via email on Aug. 14, 2024.) Most concerning, it found that the math skills of older students in grades eight to 12 are progressing so slowly that they are even further behind than they were after the initial pandemic losses. These students were in grades four through eight when the pandemic first hit in March 2020.

On the bright side, the Renaissance analysis found that first grade students in 2023-24 had completely recovered and their performance matched what first graders used to be able to do before the pandemic. Elementary school students in grades two to six were making slow progress, and remained behind.

Curriculum Associates pointed to two unexpected bright spots in its assessment results. One is phonics. At the end of the 2023-24 school year, nearly as many kindergarteners were on grade level for phonics skills as kindergarteners in 2019. That’s four out of five kindergarteners. The company also found that schools where the majority of students are Black were showing relatively better catch-up progress. “It’s small, and disparities still exist, but it’s a sign of hope,” said Curriculum Associates’s Huff. 

Here are three charts and tables from the three different testing companies that provide different snapshots of where we are.

Months of additional school required to catch up to pre-pandemic achievement levels on NWEA’s MAP tests

Percentage of students below grade level by grade and year according to Curriculum Associates’s i-Ready tests

Catch-up progress as of the winter of 2023-24, according to Renaissance, maker of the Star assessments

Understanding why recovery is stagnating and sometimes worsening over the past year is difficult. These test score analyses don’t offer explanations, but researchers shared a range of theories. 

One is that once students have a lot of holes in their foundational skills, it’s really hard for them to learn new grade-level topics each year. 

 “I think this is a problem that’s growing and building on itself,” said NWEA’s Lewis. She cited the example of a sixth grader who is still struggling to read. “Does a sixth-grade teacher have the same skills and tools to teach reading that a second or third grade teacher does? I doubt that’s the case.” 

Curriculum Associates’s Huff speculated that the whole classroom changes when a high percentage of students are behind. A teacher may have been able to give more individual attention to a small group of students who are struggling, but it’s harder to attend to individual gaps when so many students have them. It’s also harder to keep up with the traditional pace of instruction when so many students are behind. 

One high school math teacher told me that she thinks learning failed to recover and continued to deteriorate because schools didn’t rush to fill the gaps right away. This teacher said that when in-person school resumed in her city in 2021, administrators discouraged her from reviewing old topics that students had missed and told her to move forward with grade-level material. 

“The word that was going around was ‘acceleration not remediation’,” the teacher said. “These kids just missed 18 months of school. Maybe you can do that in social studies. But math builds upon itself. If I miss sixth, seventh and eighth grade, how am I going to do quadratic equations? How am I going to factor? The worst thing they ever did was not provide that remediation as soon as they walked back in the door.” This educator quit her public school teaching job in 2022 and has since been tutoring students to help them catch up from pandemic learning losses.

Chronic absenteeism is another big factor. If you don’t show up to school, you’re not likely to catch up. More than one in four students in the 2022-23 school year were chronically absent, missing at least 10 percent of the school year. 

Deteriorating mental health is also a leading theory for school struggles. A study by researchers at the University of Southern California, released Aug. 15, 2024, documented widespread psychological distress among teenage girls and preteen boys since the pandemic. Preteen boys were likely to struggle with hyperactivity, inattentiveness and conduct, such as losing their temper and fighting. These mental health struggles correlated with absenteeism and low grades. 

It’s easy to jump to the conclusion that the $190 billion that the federal government gave to schools for pandemic recovery didn’t work. (The deadline for signing contracts to spend whatever is left of that money is September 2024.) But that doesn’t tell the whole story. Most of the spending was targeted at reopening schools and upgrading heating, cooling and air ventilation systems. A much smaller amount went to academic recovery, such as tutoring or summer school. Earlier this summer two separate groups of academic researchers concluded that this money led to modest academic gains for students. The problem is that so much more is still needed.

This story about academic recovery was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post PROOF POINTS: Why are kids still struggling in school four years after the pandemic? appeared first on The Hechinger Report.

If we graded schools on how accurately they grade students, they’d fail. Nearly six out of 10 course grades are inaccurate, according to a new study of grades that teachers gave to 22,000 middle and high school students in 2022 and 2023.

The Equitable Grading Project, a nonprofit organization that seeks to change grading practices, compared 33,000 course grades with students’ scores on standardized exams, including Advanced Placement tests and annual state assessments. The organization considered a course grade to be inaccurate if a student’s test score indicated a level of knowledge that was at least a letter grade off from what the teacher had issued. For example, a grade was classified as inaccurate if a student’s test score indicated a C-level of skills and knowledge, but the student received an A or a B in the course. In this example, a D or an F grade would also be inaccurate.

Inflated grades were more common than depressed grades. In this analysis, over 40 percent of the 33,000 grades analyzed – more than 13,000 transcript grades – were higher than they should have been, while only 16 percent or 5,300 grades were lower than they should have been.  In other words, two out of five transcript grades indicated that students were more competent in the course than they actually were, while nearly one out of six grades was lower than the student’s true understanding of the course content.

The discrepancy matters, the white paper says, because inaccurate grades make it harder to figure out which students are prepared for advanced coursework or ready for college. With inflated grades, students can be promoted to difficult courses without the foundation or extra help they need to succeed. Depressed grades can discourage a student from pursuing a subject or prompt them to drop out of school altogether. 

“This data suggests that hundreds, perhaps thousands, of students in this study may have been denied, or not even offered, opportunities that they were prepared and eligible for,” the white paper said.

This analysis is evidence that widespread grade inflation, which has also been documented by the ACT, the National Center for Education Statistics and independent scholars, has persisted through 2023. In this transcript analysis, grade inflation occurred more frequently for Black and Hispanic students than Asian and white students. It was also more common for low-income students. 

Large discrepancies were documented. Almost 4,800 of the inflated grades were two letters higher than the student’s test score would indicate. An AP exam might have indicated a D-level of mastery, but the student earned a B in the class. On the flip side, more than 1,000 students received grades that were two letter grades lower than their assessment score. 

The report rejected the possibility that test anxiety is the main culprit for such widespread and large discrepancies, and laid out a list of other reasons for why grades don’t reflect a student’s skills and content mastery. Some teachers feel pressure from parents and school administrators to give high grades. Many teachers factor in participation, behavior and handing in homework assignments – things that have little to do with what a student has learned or knows. Meanwhile, grades can be depressed when teachers make deductions for late work or when students fail to turn in assignments. Group projects that are weighed heavily in the final grade can swing a student’s final transcript grade up or down. In the report, one superintendent described how teachers in his district awarded students points toward their grade based on whether their parents attended Back to School Night. 

Reasonable people can debate how much grades should be used to promote good behavior. The Equitable Grading Project argues that schools should use other rewards and consequences, and keep grades tied to academic achievement. 

However, solutions aren’t quick or easy. The organization worked with over 260 teachers during the 2022-23 school year to implement a version of “mastery-based grading,” which excludes homework, class assignments and student behavior from the final grade, but uses a range of assessments – not only tests and papers – to ascertain a student’s proficiency. Teachers were encouraged to allow students multiple retakes.  After five workshops and four coaching sessions, teachers’ grading accuracy improved by only 3 percentage points, from 37.6 percent of their grades accurately reflecting student proficiency to 40.6 percent. 

Part of the challenge may be changing the minds of teachers, who tend to think that their own grades are fine but the problem lies with their colleagues. In a survey of almost 1,200 teachers that accompanied this quantitative study, more than 4 out of 5 teachers agreed or somewhat agreed that their grades accurately reflect student learning and academic readiness. But nearly half of those same teachers doubted the accuracy of grades assigned by other teachers in their own school and department.

Grading practices are an area where schools and teachers could really use some research on what works. I’ll be keeping my eye out for solutions with evidence behind them. 

This story about the Equitable Grading Project was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post PROOF POINTS: Nearly 6 out of 10 middle and high school grades are wrong, study finds appeared first on The Hechinger Report.

It was the start of my sophomore year at a new public high school in New York City. With its dark brick exterior and barbed wire on the roof, my school already resembled a small prison — and staff had just installed several metal detectors at the front entrance.

As my classmates begrudgingly walked through security in packed lines stretching out to the street, I asked why. One of the administrators said, “Because it will keep everyone safe.”

This was a majority-Black high school, and I knew what that meant: We, the students, were perceived to be a threat — and we were being punished for something we didn’t do.

Situations like this are the reality for too many students across the United States. Black middle and high school students are over three times more likely than white students to attend a school with more security staff than mental health personnel. And data has consistently shown disparities in school discipline practices. Black students, for example, are 2.2 times more likely to be referred for disciplinary action than white students for school-related incidents.

Related: Become a lifelong learner. Subscribe to our free weekly newsletter to receive our comprehensive reporting directly in your inbox.

Meanwhile, the rising number of school shootings in our country has sparked an intense debate on how to best keep students safe. There’s a big push for more police in schools right now to provide an illusion of safety.

In my view, more law enforcement is not the answer. School safety doesn’t require more policing. Instead, schools need more structured support, such as access to mental health and counseling resources.

Increased police presence in schools is intended both to prevent and to disrupt active violence. But it can be woefully ineffective, as was the case during the Uvalde school shootings, when police not only delayed their response but also failed to adhere to safety protocols. The Uvalde disaster displayed the systemic challenges of using police in schools to create safety, including communication issues between a school district and law enforcement.

Yet despite research showing that increased physical security measures do not actually foster safe and inclusive learning environments, U.S. schools spend over $3 billion each year on security services and products, including surveillance cameras, metal detectors and armed guards or police, also known as school resource officers (SROs).

Disturbingly, SROs are more likely to be placed in schools with a high percentage of Black and Latino students, and the SROs who work in such schools are more likely to believe that the students themselves are the biggest threat, while those in white-majority schools are more likely to cite external threats. One study found that increased exposure to police in schools significantly reduced the educational performance of Black boys and lowered their graduation and college attendance rates.

Related: PROOF POINTS: Four things a mountain of school discipline records taught us

This extra policing of schools comes at a time when legislators are changing laws to subject young people, particularly Black and Latino students and students from low-income backgrounds who are already overpoliced, to increasingly harsher criminal penalties. This trend includes Washington, D.C.’s anticrime bill and Louisiana’s slew of tough-on-crime bills.

The new measures reverse some recent progress: After George Floyd’s murder in 2020, many school districts listened to families and students and removed police from schools amid national protests about law enforcement.

But removing SROs wasn’t enough. Some students returning to school after the pandemic exhibited difficulty readjusting — a manifestation of pandemic loss, racial inequality, discrimination, mental health issues, loss or sickness of family members or caregivers and more. School districts should also have added the kinds of practices that are proven to create safer schools, such as including the voices and needs of students and families in the crafting of inclusive school policies, investing in restorative practices and social and emotional learning efforts, hiring and training culturally responsive school counselors or educators and creating multitiered systems of support.

After the pandemic, teachers didn’t have the resources essential to providing the care that students needed, most notably mental health support. As a result, school districts are now bringing school resource officers back, and it’s a mistake.

Effective approaches to school safety can give students a strong sense of belonging and support in handling conflicts appropriately — before they escalate to violence. To truly keep students safe, federal and state policymakers and school principals should champion policies that support students’ physical and mental well-being and consider proposals that provide federal funds to states and schools committed to reducing harmful disciplinary practices.

As part of this effort, they should support the Counseling Not Criminalization in Schools Act and the Ending PUSHOUT Act, which would divert federal funding away from placing police in schools.

Now is an ideal time for school leaders to rethink their discipline policies and create a safe and welcoming school climate. School shootings are terrifying, but the correct response isn’t more police and metal detectors, especially in majority-Black schools that are already hyper-criminalized.

Students should not have to look back on their middle and high school years, as I do, and associate images of prison with their educational development. All students deserve an education in an inclusive, nurturing environment where they are not only safe, but can also learn and thrive.

Manny Zapata is a former teacher and is now a Ph.D. student and a policy and research intern at EdTrust, working on social, emotional and academic development.

This story about safer schools was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s weekly newsletter.

The post OPINION: Instead of hiring security staff, let’s find other ways to create safer schools appeared first on The Hechinger Report.

Too many of our public schools are missing teachers, especially in hard-to-staff subject areas. We have a teacher shortage, and it could get worse.

According to a recent report, last school year, 45 percent of public schools said they were understaffed, and nearly 9 in 10 school districts reported struggling to hire teachers heading into the 2023-24 school year. They are still struggling. In addition to the current shortage, our teacher-preparation programs report decreased enrollment.

Shortages are most common in the traditionally tricky subject areas of special education, math, science and foreign languages. School districts in high-poverty areas or those that predominantly serve students of color have been especially hard hit.

There is an emerging solution: Partnerships at the local level between minority-serving institutions and local districts can help more children learn from highly effective, well-qualified teachers.

These partnerships can be far more effective than many current efforts to fill vacancies that resort to rushing adults who are not fully certified into the classrooms.

As you’d expect, those newly minted teachers need additional training in order to provide their students with the high-quality education they deserve. As a former college of education dean, I’m confident we can increase the number of high-quality teachers in the classrooms while closing the teacher diversity gap.

We can do this by investing in, supporting and leveraging the unique role of educator preparation programs at minority-serving institutions, which have a long and successful track record of preparing diverse educators to persist and thrive in teaching.

Related: Widen your perspective. Our free biweekly newsletter consults critical voices on innovation in education.

Minority-serving institutions include our nation’s historically Black colleges and universities, Hispanic-serving institutions, tribal colleges and universities and institutions serving 25 percent or more of Asian Americans, Native Americans and Pacific Islanders.

Many of these schools prepare their graduates to be effective teachers from day one, emphasizing subject and content knowledge along with teaching techniques. They train new teachers to recognize and value diversity as an asset and encourage engagement with families and local communities. They create lesson plans and curriculums that are both rigorous and accessible and foster environments in which all students can thrive.

In the 2020-21 academic year, minority-serving institutions enrolled 28 percent of all education prep candidates nationwide and 51 percent of all candidates of color, according to federal Title II data. Many of these schools actively engage in robust partnerships with their local education systems to strengthen teaching and learning.

One example: A partnership between Laredo Independent School District and Texas A&M International University’s Educator Preparation, which relies on high expectations and data-informed recruitment to develop terrific local teachers.

The partnership features initiatives like expanded dual credit programs to improve access to higher education and an effort to promote career mobility for students in Laredo ISD. It also fosters trust-based relationships and a collective vision, so that the entire local education ecosystem is deeply committed to the preparation and success of teacher candidates.

Another example comes from the organization I lead, the Branch Alliance for Educator Diversity, or BranchED, and our National Teacher Preparation Transformation Center. In North Carolina, we helped get faculty and teaching candidates from North Carolina A&T working alongside teachers in Guilford County Schools.

These collaborative efforts more effectively prepare novice teachers for the realities of teaching while enhancing their skills and knowledge for a smoother transition into the education system.

Also, more than a dozen teams from teacher preparation programs at minority-serving institutions are working with local district partners to support school hiring needs using BranchED’s Vacancy Data Tool. The tool helps align district teacher staffing needs and teacher preparation program certification areas, and it supports the development of targeted recruitment efforts to meet those needs.

Early results for all these partnership efforts are promising, prompting discussions on candidate placement strategies, district recruitment processes and marketing opportunities for upcoming vacancies. And we can further develop these partnerships by building additional tools that leverage the unique strengths and assets of districts and educator preparation programs to meet the needs of the communities they serve.

Related: OPINION: Arkansas is having success solving teacher shortages, and other states should take notice 

Nationally, over 300,000 teaching positions were vacant or filled by teachers who were not fully certified in recent years. That’s why better alignment between teacher preparation programs’ production of teachers and school district staffing needs is warranted.

Through community partnerships, we can build a pipeline of educators equipped to thrive in today’s classrooms on day one, delivering a more inclusive and equitable education system for all.

Minority-serving institutions are doing their share by partnering with their communities to solve persistent problems at the local and district levels. They are also becoming more prominent players in our teacher preparation environment, leading the way in addressing local teacher shortages and driving innovative pathways into the profession.

By 2025, students of color will comprise nearly half of all college students. As minority-serving institutions continue to be major accelerants of this progress, these historic institutions will play a pivotal role in building a high-quality teacher workforce that looks more like America’s students.

Cassandra Herring is founder, president and CEO of Branch Alliance for Educator Diversity.

This story about the teacher shortage and minority-serving institutions was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s weekly newsletter.

The post OPINION: Powerful partnerships can help solve the national teaching shortage appeared first on The Hechinger Report.