Does AI actually help students learn? A recent experiment in a high school provides a cautionary tale. 

Researchers at the University of Pennsylvania found that Turkish high school students who had access to ChatGPT while doing practice math problems did worse on a math test compared with students who didn’t have access to ChatGPT. Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning. 

A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better. Students who just did their practice problems the old fashioned way — on their own — matched their test scores.

The researchers titled their paper, “Generative AI Can Harm Learning,” to make clear to parents and educators that the current crop of freely available AI chatbots can “substantially inhibit learning.” Even a fine-tuned version of ChatGPT designed to mimic a tutor doesn’t necessarily help.

The researchers believe the problem is that students are using the chatbot as a “crutch.” When they analyzed the questions that students typed into ChatGPT, students often simply asked for the answer. Students were not building the skills that come from solving the problems themselves. 

ChatGPT’s errors also may have been a contributing factor. The chatbot only answered the math problems correctly half of the time. Its arithmetic computations were wrong 8 percent of the time, but the bigger problem was that its step-by-step approach for how to solve a problem was wrong 42 percent of the time. The tutoring version of ChatGPT was directly fed the correct solutions and these errors were minimized.

A draft paper about the experiment was posted on the website of SSRN, formerly known as the Social Science Research Network, in July 2024. The paper has not yet been published in a peer-reviewed journal and could still be revised. 

This is just one experiment in another country, and more studies will be needed to confirm its findings. But this experiment was a large one, involving nearly a thousand students in grades nine through 11 during the fall of 2023. Teachers first reviewed a previously taught lesson with the whole classroom, and then their classrooms were randomly assigned to practice the math in one of three ways: with access to ChatGPT, with access to an AI tutor powered by ChatGPT or with no high-tech aids at all. Students in each grade were assigned the same practice problems with or without AI. Afterwards, they took a test to see how well they learned the concept. Researchers conducted four cycles of this, giving students four 90-minute sessions of practice time in four different math topics to understand whether AI tends to help, harm or do nothing.

ChatGPT also seems to produce overconfidence. In surveys that accompanied the experiment, students said they did not think that ChatGPT caused them to learn less even though they had. Students with the AI tutor thought they had done significantly better on the test even though they did not. (It’s also another good reminder to all of us that our perceptions of how much we’ve learned are often wrong.)

The authors likened the problem of learning with ChatGPT to autopilot. They recounted how an overreliance on autopilot led the Federal Aviation Administration to recommend that pilots minimize their use of this technology. Regulators wanted to make sure that pilots still know how to fly when autopilot fails to function correctly. 

ChatGPT is not the first technology to present a tradeoff in education. Typewriters and computers reduce the need for handwriting. Calculators reduce the need for arithmetic. When students have access to ChatGPT, they might answer more problems correctly, but learn less. Getting the right result to one problem won’t help them with the next one.

This story about using ChatGPT to practice math was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Kids who use ChatGPT as a study assistant do worse on tests appeared first on The Hechinger Report.

One of the biggest problems with using AI in education is that the technology hallucinates. That’s the word the artificial intelligence community uses to describe how its newest large language models make up stuff that doesn’t exist or isn’t true. Math is a particular land of make-believe for AI chatbots. Several months ago, I tested Khan Academy’s chatbot, which is powered by ChatGPT. The bot, called Khanmigo, told me I had answered a basic high school Algebra 2 problem involving negative exponents wrong. I knew my answer was right. After typing in the same correct answer three times, Khanmigo finally agreed with me. It was frustrating.

Errors matter. Kids could memorize incorrect solutions that are hard to unlearn, or become more confused about a topic. I also worry about teachers using ChatGPT and other generative AI models to write quizzes or lesson plans. At least a teacher has the opportunity to vet what AI spits out before giving or teaching it to students. It’s riskier when you’re asking students to learn directly from AI. 

Computer scientists are attempting to combat these errors in a process they call “mitigating AI hallucinations.” Two researchers from University of California, Berkeley, recently documented how they successfully reduced ChatGPT’s instructional errors to near zero in algebra. They were not as successful with statistics, where their techniques still left errors 13 percent of the time. Their paper was published in May 2024 in the peer-reviewed journal PLOS One.

In the experiment, Zachary Pardos, a computer scientist at the Berkeley School of Education, and one of his students, Shreya Bhandari, first asked ChatGPT to show how it would solve an algebra or statistics problem. They discovered that ChatGPT was “naturally verbose” and they did not have to prompt the large language model to explain its steps. But all those words didn’t help with accuracy. On average, ChatGPT’s methods and answers were wrong a third of the time. In other words, ChatGPT would earn a grade of a D if it were a student. 

Current AI models are bad at math because they’re programmed to figure out probabilities, not follow rules. Math calculations are all about rules. It’s ironic because earlier versions of AI were able to follow rules, but unable to write or summarize. Now we have the opposite.

The Berkeley researchers took advantage of the fact that ChatGPT, like humans, is erratic. They asked ChatGPT to answer the same math problem 10 times in a row. I was surprised that a machine might answer the same question differently, but that is what these large language models do.  Often the step-by-step process and the answer were the same, but the exact wording differed. Sometimes the methods were bizarre and the results were dead wrong. (See an example in the illustration below.)

Researchers grouped similar answers together. When they assessed the accuracy of the most common answer among the 10 solutions, ChatGPT was astonishingly good. For basic high-school algebra, AI’s error rate fell from 25 percent to zero. For intermediate algebra, the error rate fell from 47 percent to 2 percent. For college algebra, it fell from 27 percent to 2 percent. 

ChatGPT answered the same algebra question three different ways, but it landed on the right response seven out of 10 times in this example

However, when the scientists applied this method, which they call “self-consistency,” to statistics, it did not work as well. ChatGPT’s error rate fell from 29 percent to 13 percent, but still more than one out of 10 answers was wrong. I think that’s too many errors for students who are learning math. 

The big question, of course, is whether these ChatGPT’s solutions help students learn math better than traditional teaching. In a second part of this study, researchers recruited 274 adults online to solve math problems and randomly assigned a third of them to see these ChatGPT’s solutions as a “hint” if they needed one. (ChatGPT’s wrong answers were removed first.) On a short test afterwards, these adults improved 17 percent, compared to less than 12 percent learning gains for the adults who could see a different group of hints written by undergraduate math tutors. Those who weren’t offered any hints scored about the same on a post-test as they did on a pre-test.

Those impressive learning results for ChatGPT prompted the study authors to boldly predict that “completely autonomous generation” of an effective computerized tutoring system is “around the corner.” In theory, ChatGPT could instantly digest a book chapter or a video lecture and then immediately turn around and tutor a student on it.

Before I embrace that optimism, I’d like to see how much real students – not just adults recruited online – use these automated tutoring systems. Even in this study, where adults were paid to do math problems, 120 of the roughly 400 participants didn’t complete the work and so their results had to be thrown out. For many kids, and especially students who are struggling in a subject, learning from a computer just isn’t engaging. 

This story about AI hallucinations was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post Researchers combat AI hallucinations in math appeared first on The Hechinger Report.

Parents know they should talk and read to their young children. Dozens of nonprofit organizations have promoted the research evidence that it will help their children do better in school.

But the focus has been on improving literacy. Are there similar things that parents can do with their children to lay the foundation for success in math? 

That’s important because Americans struggle with math, ranking toward the bottom on international assessments. Weak math skills impede a child’s progress later in life, preventing them from getting through college, a vocational program or even high school. Math skills, or the lack of them, can open or close the doors to lucrative science and technology fields.

A new wave of research over the past decade has looked at how much parents talk about numbers and shapes with their children, and whether these spontaneous and natural conversations help children learn the subject. Encouraging parents to talk about numbers could be a cheap and easy way to improve the nation’s dismal math performance. 

A team of researchers from the University of Pittsburgh and the University of California, Irvine, teamed up to summarize the evidence from 22 studies conducted between 2010 and 2022. Their meta-analysis was published in the July 2024 issue of the Journal of Experimental Child Psychology. 

Here are four takeaways:

There’s a link between parent math talk and higher math skills

After looking at 22 studies, researchers found that the more parents talked about math with their children, the stronger their children’s math skills. In these studies, researchers typically observed parents and children interacting in a university lab, a school, a museum or at home and kept track of how often parents mentioned numbers or shapes. Ordinary sentences that included numbers counted. An example could be: “Hand me three potato chips.” Researchers also gave children a math test and found that children who scored higher tended to have parents who talked about math more during the observation period. 

The link between parents’ math talk and a child’s math skills was strongest between ages three and five. During these preschool years, parents who talked more about numbers and shapes tended to have children with higher math achievement. Parents who didn’t talk as much about numbers and shapes tended to have children with lower math achievement. 

With older children, the amount of time that parents spent talking about math was not as closely related to their math achievement. Researchers speculated that this was because once children start school, their math abilities are influenced more by the instruction they receive from their teachers. 

None of these studies proves that talking to your preschooler about math causes their math skills to improve. Parents who talk more about math may also have higher incomes and more education. Stronger math skills could be the result of all the other things that wealthier and more educated parents are giving their kids –  nutritious meals, a good night’s sleep, visits to museums and vacations –  and not the math talk per se. So far, studies haven’t been able to disentangle math talk from everything else that parents do for their children.

“What the research is showing at this point is that talking more about math tends to be associated with better outcomes for children,” said Alex Silver, a psychologist at the University of Pittsburgh who led the meta-analysis. “It’s an easy way to bring math concepts into your day to day life that doesn’t require buying special equipment, or setting aside time to tutor your child and try to teach them arithmetic.” 

Keep it natural

The strongest link between parent talk about math and a child’s math performance was detected when researchers didn’t tell parents to do a math activity. Parents who naturally brought up numbers or shapes in a normal conversation had children who scored higher on math assessments. When researchers had parents do a math exercise with children, the amount of math-related words that a parent used wasn’t as strongly associated with better math performance for their children. 

Silver, a postdoctoral research associate at the University of Pittsburgh’s Learning Research & Development Center, recommends bringing math into something that the child is paying attention to, rather than doing flashcards or workbooks. It could be as simple as asking  “How many?” Here’s an example Silver gave me:  “Oh, look, you have a whole lot of cars. How many cars do you have? Let’s count them. You have one, two, three. There’s three cars there.”

When you’re doing a puzzle together, turn the shape in a different direction and talk about what it looks like. Setting the dinner table, grocery shopping and keeping track of money are opportunities to talk about numbers or shapes.

“The idea is to make it fun and playful,” said Silver. “As you’re cooking, say, ‘We need to add two eggs. Oh wait, we’re doubling the recipe, so we need two more eggs. How many is that all together?’ ”

I asked Silver about the many early childhood math apps and exercises on the market, and whether parents should be spending time doing them with their children. Silver said they can be helpful for parents who don’t know where to start, but she said parents shouldn’t feel guilty if they’re not doing math drills with their kids. “It’s enough to just talk about it naturally, to find ways to bring up numbers and shapes in the context of what you’re already doing.”

Quality may matter more than quantity

In the 22 studies, more math talk was associated with higher math achievement. But researchers are unable to advise parents on exactly how much or how often to talk about math during the day. Silver said 10 utterances a day about math is probably more beneficial than just one mention a day. “Right now the evidence is that more is better, but at some point it’s so much math, you need to talk about something else now,” she said. The point of diminishing returns is unknown.

Ultimately, the quantity of math talk may not be as important as how parents talk about math, Silver said. Reading a math textbook to your child probably wouldn’t be helpful, Silver said. It’s not just about saying a bunch of math words. Still, researchers don’t know if asking questions or just talking about numbers is what makes a difference. It’s also not clear how important it is to tailor the number talk to where a child is in his math development. These are important areas of future research.

Technology may help. The latest studies are using wearable audio recorders, enabling researchers to “listen” to hours of conversations inside homes, and analyzing these conversations with natural language processing algorithms to get a more accurate understanding of parents’ math talk. The 22 studies in this meta-analysis captured as little as three minutes and as much as almost 14 hours of parent-child interactions, and these snippets of life, often recorded in a lab setting, may not reflect how parents and children talk about math in a typical week.

Low-income kids appear to benefit as much from math talk as high-income kids

Perhaps the most inspiring conclusion from this meta-analysis is that the association between a parent’s math talk and a child’s math performance was as strong for a low-income child as it was for a high-income child. 

“That’s a happy thing to see that this transcends other circumstances,” said Silver. “Targeting the amount of math input that a child receives is hopefully going to be easier, and more malleable than changing broader, systemic challenges.”

While there are many questions left to answer, Silver is already putting her research into practice with her own three-year old son. She’s asked counting questions so many times that her little one has begun to tease her. Every time he sees a group of things, he pretends to be Mommy and asks, “How many? Let’s count them!” 

 “It’s very funny,” Silver said. “I’m like, ‘Wow, Mommy really drilled that one into you, huh?’ Buddy knows what you’re up to.”

This story about math with preschoolers was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters.

The post PROOF POINTS: A little parent math talk with kids might really add up, a new body of education research suggests appeared first on The Hechinger Report.

Writing lesson plans has traditionally been a big part of a teacher’s job.  But this doesn’t mean they should be starting from a blank slate. Ideally, teachers are supposed to base their lessons on the textbooks, worksheets and digital materials that school leaders have spent a lot of time reviewing and selecting. 

But a recent national survey of more than 1,000 math teachers reveals that many are rejecting the materials they should be using and cobbling together their own.

“A surprising number of math teachers, particularly at the high school level, simply said we don’t use the district or school-provided materials, or they claimed they didn’t have any,” said William Zahner, an associate professor of mathematics at San Diego State University, who presented the survey at the April 2024 annual meeting of the American Educational Research Association in Philadelphia. Students, he said, are often being taught through a “bricolage” of materials that teachers assemble themselves from colleagues and the internet. 

“What I see happening is a lot of math teachers are rewriting a curriculum that has already been written,” said Zahner. 

The survey results varied by grade level. More than 75 percent of elementary school math teachers said they used their school’s recommended materials, but fewer than 50 percent of high school math teachers said they did. 

Share of math teachers who use their school’s recommended materials

The do-it-yourself approach has two downsides, Zahner said, both of which affect students. One problem is that it’s time consuming. Time spent finding materials is time not spent giving students feedback, tailoring existing lessons for students or giving students one-to-one tutoring help. The hunt for materials is also exhausting and can lead to teacher burnout, Zahner said.

Related: Education research, condensed. The free Proof Points newsletter delivers one story every Monday.

The other problem is that teacher-made materials may sacrifice the thoughtful sequencing of topics planned by curriculum designers.  When teachers create or take materials from various sources, it is hard to maintain a “coherent development” of ideas, Zahner explained. Curriculum designers may weave a review of previous concepts to reinforce them even as new ideas are introduced. Teacher-curated materials may be disjointed. Separate research has found that some of the most popular materials that teachers grab from internet sites, such as Teachers Pay Teachers, are not high quality. 

The national survey was conducted in 2021 by researchers at San Diego State University, including Zahner, who also directs the university’s Center for Research in Mathematics and Science Education, and the English Learners Success Forum, a nonprofit that seeks to improve the quality of instructional materials for English learners. The researchers sought out the views of teachers who worked in school districts where more than 10 percent of the students were classified as English learners, which is the national average. More than 1,000 math teachers, from kindergarten through 12th grade, responded. On average, 30 percent of their students were English learners, but some teachers had zero English learners and others had all English learners in their classrooms.

Teachers were asked about the drawbacks of their assigned curriculum for English learners. Many said that their existing materials weren’t connected to their students’ languages and cultures. Others said that the explanations of how to tailor a lesson to an English learner were too general to be useful.  Zahner says that teachers have a point and that they need more support in how to help English learners develop the language of mathematical reasoning and argumentation.

It was not clear from this survey whether the desire to accommodate English learners was the primary reason that teachers were putting together their own materials or whether they would have done so anyway. 

Related: Most English lessons on Teachers Pay Teachers and other sites are ‘mediocre’ or ‘not worth using,’ study finds

“There are a thousand reasons why this is happening,” said Zahner. One high school teacher in Louisiana who participated in the survey said his students needed a more advanced curriculum. Supervisors inside a school may not like the materials that officials in a central office have chosen. “Sometimes schools have the materials but they’re all hidden in a closet,” Zahner said.

In the midst of a national debate on how best to teach math, this survey is an important reminder of yet another reason why many students aren’t getting the instruction that they need. 

This story about math lessons was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Proof Points and other Hechinger newsletters. 

The post PROOF POINTS: Many high school math teachers cobble together their own instructional materials from the internet and elsewhere, a survey finds appeared first on The Hechinger Report.

Jo Boaler is a professor at the Stanford Graduate School of Education with a devoted following of teachers who cheer her call to make math education more exciting. But despite all her fans, she has sparked controversy at nearly every stage of her career. Critics say she misrepresents research to make her case and her ideas actually impede students. Now, with a new book coming out in May, provocatively titled “MATH-ish,” Boaler is fighting back. 

“This is a whole effort to shut me down, my research and my writing,” said Boaler. “I see it as a form of knowledge suppression.”

Academic fights usually don’t make it beyond the ivory tower. But Boaler’s popularity and influence have made her a focal point in the current math wars, which also seem to reflect the broader culture wars.  In the last few months, tabloids and conservative publications have turned Boaler into something of an education villain who’s captured the attention of Elon Musk and Texas Sen. Ted Cruz on social media. Critics have even questioned Boaler’s association with a former reality TV star.

“I am the next target,” Boaler said, describing the death threats and abusive email she’s been receiving.

This controversy matters on a much larger level because there is a legitimate debate about how math should be taught in American schools. Cognitive science research suggests that students need a lot of practice and memorization to master math. And once students achieve success through practice, this success will motivate them to learn and enjoy math. In other words, success increases motivation at least as much as motivation produces success. 

Yet, from Boaler’s perspective, too many students feel like failures in math class and hate the subject. That leaves us with millions of Americans who are innumerate. Nearly 2 out of every 5 eighth graders don’t even have the most basic math skills, according to the 2022 National Assessment of Educational Progress (NAEP). On the international Program for International Student Assessment (PISA), American 15-year-olds rank toward the bottom of economically advanced nations in math achievement. 

Boaler draws upon a different body of research about student motivation that looks at the root causes of why students don’t like math based on surveys and interviews. Students who are tracked into low-level classes feel discouraged. Struggling math students often describe feelings of anxiety from timed tests. Many students express frustration that math is just a collection of meaningless procedures. 

Boaler seeks to fix these root causes. She advocates for ending tracking by ability in math classes, getting rid of timed tests and starting with conceptual understanding before introducing procedures. Most importantly, she wants to elevate the work that students tackle in math classes with more interesting questions that spark genuine curiosity and encourage students to think and wonder. Her goal is to expose students to the beauty of mathematical thinking as mathematicians enjoy the subject. Whether students actually learn more math the Boaler way is where this dispute centers. In other words, how strong is the evidence base?

The latest battle over Boaler’s work began with an anonymous complaint published in March by the Washington Free Beacon, the same conservative website that first surfaced plagiarism accusations against Claudine Gay, the former president of Harvard University. The complaint accuses Boaler of a “reckless disregard for accuracy” by misrepresenting research citations 52 times and asks Stanford to discipline Boaler, a full professor with an endowed chair. Stanford has said it’s reviewing the complaint and hasn’t decided whether to open an investigation, according to news reports. Boaler stands by her research (other than one citation that she says has been fixed) and calls the anonymous complaint “bogus.” (UPDATE: The Hechinger Report learned after this article was published that Stanford has decided not to open an investigation.)

“They haven’t even got the courage to put their name on accusations like this,” Boaler said. “That tells us something.”

Boaler first drew fire from critics in 2005, when she presented new research claiming that students at a low-income school who were behind grade level had outperformed students at higher-achieving schools when they were taught in classrooms that combined students of different math achievement levels. The supposed secret sauce was an unusual curriculum that emphasized group work and de-emphasized lectures. Critics disparaged the findings and hounded her to release her data. Math professors at Stanford and Cal State University re-crunched the numbers and declared they’d found the opposite result.

Boaler, who is originally from England, retreated to an academic post back in the U.K., but returned to Stanford in 2010 with a fighting spirit. She had written a book, “What’s Math Got to Do with It?: How Parents and Teachers Can Help Children Learn to Love Their Least Favorite Subject,” which explained to a general audience why challenging, open-ended problems would help more children to embrace math and how the current approach of boring drills and formulas was turning too many kids off. Teachers loved it.

Boaler accused her earlier critics of academic bullying and harassment. But she didn’t address their legitimate research questions. Instead, she focused on changing classrooms. Tens of thousands of teachers and parents flocked to her 2013 online course on how to teach math. Building on this new fan base, she founded a nonprofit organization at Stanford called youcubed to train teachers, conduct research and spread her gospel. Boaler says a half million teachers now visit youcubed’s website each month.

Boaler also saw math as a lever to promote social justice. She lamented that too many low-income Black and Hispanic children were stuck in discouraging, low-level math classes. She advocated for change. In 2014, San Francisco heeded that call, mixing different achievement levels in middle school classrooms and delaying algebra until ninth grade. Parents, especially in the city’s large Asian community, protested that delaying algebra was holding their children back. Without starting algebra in middle school, it was difficult to progress to high school calculus, an important course for college applications. Parents blamed Boaler, who applauded San Francisco for getting math right. Ten years later, the city is slated to reinstate algebra for eighth graders this fall. Boaler denies any involvement in the unpopular San Francisco reforms.

Before that math experiment unraveled in San Francisco, California education policymakers tapped Boaler to be one of the lead writers of a new math framework, which would guide math instruction throughout the state. The first draft discouraged tracking children into separate math classes by achievement levels, and proposed delaying algebra until high school. It emphasized “social justice” and suggested that students could take data science instead of advanced algebra in high school. Traditional math proponents worried that the document would water down math instruction in California, hinder advanced students and make it harder to pursue STEM careers. And they were concerned that California’s proposed reforms could spread across the nation. 

In the battle to quash the framework, critics attacked Boaler for trying to institute “woke” mathematics. The battle became personal, with some criticizing her for taking $5,000-an-hour consulting and speaking fees at public schools while sending her own children to private school. 

Critics also dug into the weeds of the framework document, which is how this also became a research story. A Stanford mathematics professor catalogued a list of what he saw as research misrepresentations. Those citations, together with additional characterizations of research findings throughout Boaler’s writings, eventually grew into the anonymous complaint that’s now at Stanford.

By the time the most recent complaint against Boaler was lodged, the framework had already been revised in substantial ways. Boaler’s critics had arguably won their main policy battles. College-bound students still need the traditional course sequence and cannot substitute data science for advanced algebra. California’s middle schools will continue to have the option to track children into separate classes and start algebra in eighth grade. 

But the attacks on Boaler continue. In addition to seeking sanctions from Stanford, her anonymous critics have asked academic journals to pull down her papers, according to Boaler. They’ve written to conference organizers to stop Boaler from speaking and, she says, they’ve told her funders to stop giving money to her. At least one, the Valhalla Foundation, the family foundation of billionaire Scott Cook (co-founder of the software giant Intuit), stopped funding youcubed in 2024. In 2022 and 2023, it gave Boaler’s organization more than $560,000. 

Boaler sees the continued salvos against her as part of the larger right-wing attack on diversity, equity and inclusion or DEI. She also sees a misogynistic pattern of taking down women who have power in education, such as Claudine Gay. “You’re basically hung, drawn and quartered by the court of Twitter,” she said.

From my perch as a journalist who covers education research, I see that Boaler has a tendency to overstate the implications of a narrow study. Sometimes she cites a theory that’s been written about in an academic journal but hasn’t been proven and labels it research. While technically true – most academic writing falls under the broad category of research –  that’s not the same as evidence from a well-designed classroom experiment.  And she tends not to factor in evidence that runs counter to her views or adjust her views as new studies arise. Some of her numerical claims seem grandiose. For example, she says one of her 18-lesson summer courses raised achievement by 2.8 years.

“People have raised questions for a long time about the rigor and the care in which Jo makes claims related to both her own research and others,” said Jon Star, a professor of math education at Harvard Graduate School of Education. 

But Star says many other education researchers have done exactly the same, and the “liberties” Boaler takes are common in the field. “That’s not to suggest that taking these liberties is okay,” Star said, “but she is being called out for it.”

Boaler is getting more scrutiny than her colleagues, he said, because she’s influential, has a large following of devoted teachers and has been involved in policy changes at schools. Many other scholars of math education share Boaler’s views. But Boaler has become the public face of nontraditional teaching ideas in math. And in today’s polarized political climate, that’s a dangerous public face to be.

The citation controversy reflects bigger issues with the state of education research. It’s often not as precise as the hard sciences or even social sciences like economics. Academic experts are prone to make wide, sweeping statements. And there are too few studies in real classrooms or randomized controlled trials that could settle some of the big debates. Star argues that more replication studies could improve the quality of evidence for math instruction. We can’t know which teaching methods are most effective unless the method can be reproduced in different settings with different students. 

It’s also possible that more research may never settle these big math debates and we may continue to generate conflicting evidence. There’s the real possibility that traditional methods could be more effective for short-term achievement gains, while nontraditional methods might attract more students to the subject, and potentially lead to more creative problem-solvers in the future. 

Even if Boaler is loose with the details of research studies, she could still be right about the big picture. Maybe advanced students would be better off slowing down on the current racetrack to calculus to learn math with more depth and breadth. Her fun hands-on approach to math might spark just enough motivation to inspire more kids to do their homework. Might we trade off a bit of short-term math achievement for a greater good of a numerate, civic society?

In her new book, “MATH-ish,” Boaler is doubling down on her approach to math with a title that seems to encourage inexactitude. She argues that approaching a problem in a “math-ish” way gives students the freedom to take a guess and make mistakes, to step back and think rather than jumping to numerical calculations. Boaler says she’s hearing from teachers that “ish” is far more fun than making estimates.

“I’m hoping this book is going to be my salvation,” she said, “that I have something exciting to do and focus on and not focus on the thousands of abusive messages I’m getting.”

This story about Jo Boaler was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Proof Points newsletter.

The post PROOF POINTS: Stanford’s Jo Boaler talks about her new book ‘MATH-ish’ and takes on her critics appeared first on The Hechinger Report.

Universities, philanthropies, and even the U.S. government are all trying to encourage more young Americans to pursue careers in STEM,  an acronym for science, technology, engineering and mathematics. Many business sectors, from high tech to manufacturing, are plagued with shortages of workers with technical skills. In New York City, where I live, the subway is frequently plastered with advertisements carrying the message that STEM fields pay well. But studying STEM requires more than an interest in science or a desire to make good money. Students also need adequate training, even in elementary and middle school.

That’s why it’s concerning that high-achieving students, who’ve received less public attention than lower achieving students, were also set back by remote learning and pandemic uncertainty.  Fewer students with math skills shrinks the pool of people who are likely to cultivate an expertise in science, engineering and technology a decade from now. In other words, the STEM pipeline –  a metaphor for the development of future scientists, engineers and other high tech workers –  likely starts with a narrower funnel in the post-pandemic era.

The stakes are high not only for Gen Z, as they age out of school and enter the workforce, but also for the future of the U.S. economy, which needs skilled scientists and engineers to grow.

The leading indicators of STEM troubles ahead are apparent within the 2022 scores from a national test called the National Assessment of Educational Progress (NAEP). The immediate headlines from that first post-pandemic test focused on the fact that two decades of academic progress had been suddenly erased. Low-achieving children, who tend to be poor, had lost the most ground. An alarming number of American children – as high as 38 percent of eighth graders  – were functioning below the “basic” level in math, meaning that they didn’t have even the most rudimentary math skills.

Statisticians at the National Center for Education Statistics (NCES) have continued to dig into the 2022 data, and they’ve been also turning their attention to students at the top. These children are on grade level, but the eighth grade NAEP assessment shows that far fewer of them are hitting an advanced performance level, or even a proficient one. Math scores among top performers dropped as steeply as scores did among low performers. Even the scores of students at Catholic schools, who otherwise weathered the pandemic well, plummeted in eighth grade math. 

We don’t have data for other private schools because they have refused to participate in NAEP testing, but the eighth grade math declines among both high-achieving public school and Catholic school students are not good signs. 

NAEP tests reading and math in both fourth and eighth grades every two years in order to track educational progress. It’s one of the only tests that can be used for comparisons across states and generations. More than 400,000 students are specially selected to represent the regions and demographic characteristics of the nation. 

Among the four NAEP tests, eighth grade math showed the sharpest pandemic drop.  Math took a bigger hit than reading because kids can still read at home, while math is something that students primarily learn at school. If you didn’t read “The Hobbit” in your seventh grade English class because you were out sick with Covid, you can still be a good lifelong reader  But not getting enough practice with rates, ratios and percentages in middle school can derail someone who might have otherwise excelled. 

Why eighth grade math was hit harder than fourth grade math is a bit less obvious. One explanation is that the concepts that students need to learn are more difficult. Square roots and exponents are possibly more challenging to master than multiplication and division. And fewer parents are able to assist with homework as the math increases in complexity.

Yet another explanation is a psychological one. These eighth graders were in sixth grade when the pandemic erupted in the spring of 2020. This is a critical time in adolescent development when children are figuring out who they are and where they belong. A lot of this development occurs through social interaction. The isolation may have stunted psychological development and that ultimately affected motivation, study skills and the ability to delay gratification – all necessary to excel in math.

Let’s walk through the numbers together.

Highest achieving students lost ground in eighth grade math

This chart shows that the highest performing students, those at the top 10 percent and the top 25 percent, lost as much as low-achieving students at the bottom in eighth grade math. These eighth graders were in the spring of sixth grade when the pandemic hit in 2020, and it’s possible that they didn’t master important prerequisite skills, such as rates and ratios. These kids at the top are performing at grade level, but not as high performing as past eighth graders.

Fewer eighth grade students hit advanced and proficient levels

This bar chart shows that before the pandemic 10 percent of the nation’s eighth graders were performing at an advanced level in math. That fell to 7 percent. And the number of students deemed proficient in eighth grade math fell even more, from 24 percent to 20 percent. Before the pandemic, arguably, 34 percent of the eighth grade population was on track to pursue advanced math in high school and a future STEM career if they wanted one. After the pandemic in 2022, only 27 percent were well prepared.

Students at Catholic schools are generally much higher performing than students at public schools. In large part, that’s because of family income; wealthier students tend to have higher test scores than poorer students. Catholic school students tend to be wealthier; their families can afford private school tuition. In recent years, the Catholic Church has closed hundreds of schools that catered to low-income families, leaving a higher income population in its remaining classrooms. 

Catholic schools outperformed public schools but also dropped 

This chart shows that Catholic school students, depicted by the diamonds, outperformed public school students, depicted by the circles, in eighth grade math. But it was still a sharp five-point decline in eighth grade math performance for Catholic school students, almost as large as the eight-point decline for public school students. Scores of white students at Catholic schools declined five points; scores of students at Catholic schools in the suburbs declined seven points. Almost a quarter of Catholic school students are now functioning below a basic level in math for their grade. 

Despite the good academic reputation of Catholic schools and the praise Catholic schools received for resuming in-person instruction sooner, math scores suggest a problem. And it’s a problem that potentially extends to the whole private school universe, where 9 percent of students are enrolled, according to the most recently available data from 2019. 

I talked with Ron Reynolds, the executive director of the California Association of Private School Organizations, who explained that not just Catholic schools, but also many other private schools suffered even if they hadn’t been closed for long. Reynolds said that private schools were still hit by illnesses, deaths and absences and that might have affected instruction.

“Private schools are tightly knit communities in which teachers tend to be more intertwined in the lives of the children and families they serve,” he said. “When you have a crisis, and so many people experiencing stress and loss, that can certainly impact the teacher in some significant ways.”

Unfortunately, we don’t know exactly how other private schools fared during the pandemic because they have refused to participate in the NAEP tests for the past decade. Reynolds, who serves on the governing board that oversees the NAEP exam, has been trying to lobby more private schools to participate, but so far, to no avail.

Together private schools, selective public schools and affluent suburban schools have been important training grounds for the nation’s future scientists and engineers. Of course, it is possible that these high achieving students, now 10th graders, will catch up. Many of them are from wealthier families who can afford tutors, or attend well-resourced schools. But I am not seeing much evidence that schools have had the ability to think about the pipeline of advanced students when many students are so needy. And with post-pandemic grade inflation, students and parents may not be getting the signals they need to seek extra help independently. 

The administration of the 2024 NAEP test wrapped up in March, but results won’t be known for many months. I’ll be keeping an eye on eighth grade math and on SAT, ACT and Advanced Placement scores in the years to come.

This story about math scores was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Proof Points newsletter.

The post PROOF POINTS: How Covid narrowed the STEM pipeline appeared first on The Hechinger Report.

Editor’s note: This story led off this week’s Future of Learning newsletter, which is delivered free to subscribers’ inboxes every other Wednesday with trends and top stories about education innovation. Subscribe today!

Last year, I began reporting on the growing interest in teaching young people about data science amid calls that Algebra II and other higher-level math classes are being taught in outdated ways and need to be modernized. Experts were already raising concerns about falling math scores before the pandemic, and those scores nationwide have only continued to worsen.

There’s no easy answer – math experts, STEM professors, high school educators, parents, advocates and even students have vastly different opinions on what math knowledge and courses should be required for students to succeed in college and careers.

Nowhere has this been clearer than in California. As I wrote in my latest story, co-published with The Washington Post, the state’s public higher education system has gone back and forth on whether data science (an interdisciplinary field that combines computer programming, math and statistics) and other statistics-based courses fit into existing math pathways and can serve as an alternative to Algebra II in admissions.

But missing from these debates was the voices of students and educators – those most affected by any decisions made by the state’s public university system. I wanted to see for myself what students were learning in high school data science classes, why they were signing up for the course and how decisions about which math classes to take were being determined.

In December, I visited Oxnard Union High School District, which launched a data science pathway in 2020. The class targeted students who didn’t plan to major in STEM fields in college, as well as those who planned to attend a community college or go straight into the workforce or military. A “math class for poets” was how the district’s superintendent, Tom McCoy, had jokingly described it.

From my visits to the district’s high school data science classes and my conversations with teachers and students, two things became clear: The course’s structure is very different than a traditional math class – it’s an applied, project-based learning course in which students collaborate closely as they learn the material. And the way different teachers and schools approach the class differs greatly, even within a single district. Some teachers emphasize data literacy (teaching students how to read and analyze data); others incorporate math concepts from algebra and statistics; and still others may inject more computer programming or coding.

That variation — both in how the classes are taught and their content – has added to concerns that data science courses are low quality and insufficiently rigorous. And it’s in part why there’s an emerging push to develop standards around the course, and tackle the question of what an effective data science course should look like.

Much of the concern around data science in California centers around three programs — Introduction to Data Science, Youcubed and CourseKata — that make up the majority of data science courses available there. According to a recent report from University of California committee that sets admissions standards, none of the courses “even come close to meeting the required standard to be a ‘more advanced’ course,” and are more similar to data literacy courses than advanced mathematics. (Oxnard Union uses a different curricula, one developed by ed tech vendor Bootstrap.)  

Mahmoud Harding is the instructional design director at Data Science 4 Everyone, a national initiative based at the University of Chicago. He co-developed a high school data science program at the North Carolina School of Science and Mathematics and teaches a course  at North Carolina State’s Data Science Academy. He said a high school data science course should help students find more real-world applications for concepts they learn in algebra.

In addition, the class should build conceptual knowledge of statistical topics through computation, visualizations and simulations, and help students understand bias within data and ethical concerns in using flawed data. Data science courses also need to be substantively different from statistics or computer programing courses, he said, noting that data science is “inherently interdisciplinary.”

“I don’t think a data science course is the same as an Algebra II course,” Harding said. “But it doesn’t mean that a data science course isn’t rigorous, or it doesn’t mean that you can’t matriculate into higher forms of algebra because you’ve taken data science.”

Harding’s group, Data Science 4 Everyone, is helping to lead the new effort to develop standards for data science. Zarek Drozda, the group’s executive director, said this year it will convene a working group of experts, K-12 educators, STEM professors, curriculum providers, state and district leaders, students and industry and workforce professionals including those with tech companies, to help create a list of recommendations of baseline data science standards.

As career opportunities involving AI, computing and data increase, Drozda said it is “critical” that we think about the foundational knowledge students need by the time they graduate from college. The group is engaging people from all sides of the data science debate to look critically at the courses currently offered and identify how to create classes that will better meet the needs of students.

Drozda said he also hopes the working group will consider how exposure to data science classes can help more students get excited about STEM fields that don’t necessarily require a four-year degree.

“I think there’s a false perception that we are trying to replace fundamental mathematics,” Drozda said. “In reality, we are trying to modernize, add options and enhance the relevance of mathematics and prove to students that math matters in the 21st century.”

This story about data science classes was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Hechinger newsletter. 

The post Calculating the value of data science classes appeared first on The Hechinger Report.

A growing chorus of education researchers, pundits and “science of reading” advocates are calling for young children to be taught more about the world around them. It’s an indirect way of teaching reading comprehension. The theory is that what we grasp from what we read depends on whether we can hook it to concepts and topics that we already know. Natalie Wexler’s 2019 best-selling book, The Knowledge Gap, championed knowledge-building curricula and more schools around the country, from Baltimore to Michigan to Colorado, are adopting these content-filled lesson plans to teach geography, astronomy and even art history. 

Makers of knowledge-building curricula say their lessons are based on research, but the truth is that there is scant classroom evidence that building knowledge first increases future reading comprehension. 

In 2023, University of Virginia researchers promoted a study of Colorado charter schools that had adopted E.D. Hirsch’s Core Knowledge curriculum. Children who had won lotteries to attend these charter schools had higher reading scores than students who lost the lotteries. But it was impossible to tell whether the Core Knowledge curriculum itself made the difference or if the boost to reading scores could be attributed to other things that these charter schools were doing, such as hiring great teachers and training them well. 

More importantly, the students at these charter schools were largely from middle and upper middle class families. And what we really want to know is whether knowledge building at school helps poorer children, who are less likely to be exposed to the world through travel, live performances, and other experiences that money can buy.

A new study, published online on Feb. 26, 2024, in the peer-reviewed journal Developmental Psychology, now provides stronger causal evidence that building background knowledge can translate into higher reading achievement for low-income children. The study took place in an unnamed, large urban school district in North Carolina where most of the students are Black and Hispanic and 40 percent are from low-income families.

In 2019, a group of researchers, led by James Kim, a professor at Harvard University’s Graduate School of Education, randomly selected 15 of the district’s 30 elementary schools to teach first graders special knowledge-building lessons for three years, through third grade. Kim, a reading specialist, and other researchers had developed two sets of multi-year lesson plans, one for science and one for social studies. Students were also given related books to read during the summer. (This research was funded by the Chan Zuckerberg Initiative, which is among the many funders of The Hechinger Report.)

The remaining 15 elementary schools in the district continued to teach their students as usual, still delivering some social studies and science instruction, but not these special lessons. Regular reading class was untouched in the experiment. All 30 schools were using the same reading curriculum, Expeditionary Learning, which follows science of reading principles and teaches phonics. 

Covid hit in the middle of the experiment. When schools shut down in the spring of 2020, the researchers scrapped the planned social studies units for second graders. In 2021, students were still not attending school in person. The researchers revised their science curriculum and decided to give an abridged online version to all 30 schools instead of just half. In the end, children in the original 15 schools received one year of social studies lessons and three years of science lessons compared to only one year of science in the comparison group. 

Still, approximately 1,000 students who had received the special science and social studies lessons in first and second grades outperformed the 1,000 students who got only the abbreviated online science in third grade. Their reading and math scores on the North Carolina state tests were higher not only in third grade, but also in fourth grade, more than a year after the knowledge-building experiment ended. 

It wasn’t a huge boost to reading achievement, but it was significant and long-lasting. It cost about $400 per student in instructional materials and teacher training.

Timothy Shanahan, a literacy expert and a professor emeritus at the University of Illinois at Chicago who was not involved in this research or the development of these science lessons, praised the study. “The study makes it very clear (as have a few others recently) that it is possible to combine reading with social studies and science curriculum in powerful ways that can improve both literacy and content knowledge,” he said by email. 

Connecting background knowledge to reading comprehension is not a new idea.  A famous 1987 experiment documented that children who were weaker readers but knowledgeable about baseball understood a reading passage about baseball better than children who were stronger readers but didn’t know much about the sport. 

Obviously, it’s not realistic for schools to attempt to familiarize students with every topic they might encounter in a book. And there is disagreement among researchers about how general knowledge of the world translates into higher reading performance.

Kim thinks that a knowledge-building curriculum doesn’t need to teach many topics. Random facts, he says, are not important. He argues for depth instead of breadth. He says it’s important to construct a thoughtful sequence of lessons over the years, allowing students to see how the same patterns crop up in different ways. He calls these patterns “schemas.” In this experiment, for example, students learned about animal survival in first grade and dinosaur extinction in second grade. In third grade, that evolved into a more general understanding of how living systems function. By the end of third grade, many students were able to see how the idea of functioning systems can apply to inanimate objects, such as skyscrapers. 

It’s the patterns that can be analogized to new circumstances, Kim explained. Once a student is familiar with the template, a new text on an unfamiliar topic can be easier to grasp.

Kim and his team also paired the science lessons with clusters of vocabulary words that were likely to come up again in the future – almost like wine pairings with a meal. 

The full benefits of this kind of knowledge building didn’t materialize until after several years of coordinated instruction. In the first years, students were only able to transfer their ability to comprehend text on one topic to another if the topics were very similar. This study indicates that as their content knowledge deepened, their ability to generalize increased as well.  

There’s a lot going on here: a spiraling curriculum that revisits and builds upon themes year after year; an explicit teaching of underlying patterns; new vocabulary words, and a progression from the simple to the complex. 

There are many versions of knowledge-rich curricula and this one isn’t about exposing students to a classical canon. It remains unclear if all knowledge-building curricula work as well. Other programs sometimes replace the main reading class with knowledge-building lessons. This one didn’t tinker with regular reading class. 

The biggest challenge with the approach used in the North Carolina experiment is that it requires schools to coordinate lessons across grades. That’s hard. Some teachers may want to keep their favorite units on, say, growing a bean plant, and may bristle at the idea of throwing away their old lesson plans.

It’s also worth noting that students’ math scores improved as much as their reading scores did in this North Carolina experiment. It might seem surprising that a literacy intervention would also boost math. But math also requires a lot of reading; the state’s math tests were full of word problems. Any successful effort to boost reading skills is also likely to have positive spillovers into math, researchers explained.

School leaders are under great pressure to boost test scores. To do that, they’ve often doubled time spent on reading and cut science and social studies classes. Studies like this one suggest that those cuts may have been costly, further undermining reading achievement instead of improving it. As researchers discover more about the science of reading, it may well turn out to be that more time on science itself is what kids need to become good readers.

This story about background knowledge was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Proof Points newsletter.

The post PROOF POINTS: Learning science might help kids read better appeared first on The Hechinger Report.

For a few weeks in the spring of 2016, nearly all the eighth graders at a small public school affiliated with Columbia University agreed to stay late after school to study math. They were preparing for a critical test, the New York State’s Regents examination in algebra. Half of the kids came from families that lived below the poverty line in Harlem and upper Manhattan. They attended a selective middle school, and were advanced enough to be taking algebra in eighth instead of ninth grade. Many others were the children of Columbia professors, and none of them – rich or poor – really needed help passing the test.

But researchers set up a review class to test a theory about the best way to study for a test. For the first eight sessions, half the students had a traditional review class. They were given a Barron’s Regents review study guide with lots of practice problems. Their teachers worked through the first half of the problems, explaining how to solve them step by step. 

The other half of their classmates studied the same algebra topics in a different way. They spent the first 45-minute session taking a mini practice test. They received no instruction and worked independently. The following day, their teachers went over the students’ errors. The students had four test-and-review cycles like this, for a total of four mini tests and four sessions of error review. 

Then, the two groups swapped. The kids who had been taught via traditional, explicit instruction switched to reviewing the remaining algebra topics through their errors. And the kids who had been correcting their errors received eight sessions of traditional test prep. Their teachers taught both ways too, so that differences between the two modes couldn’t be attributed to a particular teacher. The following year, the same four teachers repeated the entire experiment with a fresh group of eighth graders. All told, 175 kids participated in the experiment. 

Which method worked best? 

On the surface, it was a tie. Students improved by about the same amount – 12 percent – whether they learned through explicit instruction or error review. Students had taken tests before and after the test prep course. Noting how much they improved on various algebra topics, researchers were able to trace those gains back to whether students learned that topic through explicit instruction or through their errors.

There was one big difference, however. Learning through errors was twice as powerful based on instructional time. Teachers had to teach all eight sessions in the traditional instruction condition, totaling 360 minutes of instructional time. But teachers only had to teach every other session when students learned through errors, adding up to only 180 minutes.

“You get more bang from your teacher buck, if you will, from the learning from errors condition,” said Janet Metcalfe, a psychologist at Columbia University who led the study, which was published online in the British Journal of Educational Psychology in January 2024. (The Hechinger Report is an independent news organization based at Teachers College, Columbia University, but has no relationship with the middle school or the researchers involved in the study.)

Of course, students might not see it that way. They still had to be in class for the full 360 minutes after school, with half the time spent taking practice tests in order to generate the errors.

The study is not a repudiation of explicit instruction. The students were also taking an algebra class during the school day where they had likely had a lot of explicit instruction and were already familiar with the concepts. Metcalfe said that having this background knowledge is critical for learning by errors to work. The students aren’t just guessing, but they’re making common mistakes. 

“They’re just doing one little thing wrong,” said Metcalfe. “And once they understand what that one little thing is, and remember not to fall into a habit where they’ll make the same mistake, they can overcome it.”

Metcalfe offered the example of fractions. A student might mistakenly think that a large denominator means it’s a large number, but then remembers discussing the error in a review session and knows that a fraction with a large denominator might actually be a tiny number. The memory of discussing the error stops the student from making it again, she explained.

Learning from errors, however, was inconsistent. One of the four teachers produced more than twice the test score gains for students than a colleague. It’s not that this teacher was much better than the others. All four teachers produced almost identical test score gains when they taught explicitly how to solve problems. They were all good explainers. 

But being a good explainer is not always the same thing as being a good teacher. Metcalfe and her team analyzed videos and transcripts of the review sessions to understand what the teachers were doing differently. And it turns out there are multiple ways to teach through errors. 

The teacher who got the best results employed a sort of Socratic method. “Okay, you guys got this wrong? Why would somebody get this wrong?” recalled Metcalfe. “And he did very little lecturing, almost none.” 

This teacher asked his students to talk about how they had solved the problem and why they did it that way.  He asked them to talk about what they found difficult. Students would often explain their thinking to each other. Finally, the teacher would ask his students to come up with ideas on how to recognize and avoid such mistakes in the future. This teacher had a knack for maintaining a fast pace and getting through a lot of problems from the previous day’s mini test. His students’ test scores jumped by far more than 12 percent when he taught this way.

By contrast, the teacher who produced the lowest test score gains tended to lecture students on the correct way to solve the problems that they had gotten wrong. The focus was on the corrections, not the errors. His classes weren’t very interactive.  His students’ test scores improved by only 6 percent instead of 12 percent. Still, on a per minute basis, he was as effective teaching through errors as he had been teaching traditionally. 

Another teacher was extremely slow paced. “I was convinced when I was watching the teachers that the second teacher would have no success at all,” recalled Metcalfe. “He would take five minutes on one problem, and just let them mull over it.”

Her prediction was wrong. “His students did really well,”  Metcalfe said, laughing. Perhaps this is an error that Metcalfe won’t make again. Like the star teacher, he didn’t lecture.

It’s worth emphasizing again that these were highly motivated, high-achieving students who cared about their Regents exam scores. This method might not work with less motivated students who are struggling in school.  

Even with ideal students, it also seems like it takes a special teacher to pull off this kind of teaching. It reminds me of other progressive teaching approaches, from inquiry learning to project-based learning, for which researchers have documented remarkable results with masterful teachers. But maybe it’s asking too much of the average teacher to teach this way, thinking of questions on the fly that will magically steer students to the right answers. Should we be promoting ways of teaching that only a small minority of teachers can realistically do well? 

My big takeaway from this study is for students. When preparing for a math exam, they should take a practice test, go over mistakes and make sure they understand why they made them. 

This story about test prep was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Proof Points newsletter.

The post PROOF POINTS: The surprising effectiveness of having kids study why they failed appeared first on The Hechinger Report.

Silence may be golden, but when it comes to learning with a tutor, talking is pure gold. It’s audible proof that a student is paying attention and not drifting off, research suggests. More importantly, the more a student articulates his or her reasoning, the easier it is for a tutor to correct misunderstandings or praise a breakthrough. Those are the moments when learning happens.

One India-based tutoring company, Cuemath, trains its tutors to encourage students to talk more. Its tutors are in India, but many of its clients are American families with elementary school children. The tutoring takes place at home via online video, like a Zoom meeting with a whiteboard, where both tutor and student can work on math problems together. 

The company wanted to see if it could boost student participation so it collaborated with researchers at Stanford University to develop a “talk meter,” sort of a Fitbit for the voice, for its tutoring site. Thanks to advances in artificial intelligence, the researchers could separate the audio of the tutors from that of the students and calculate the ratio of tutor-to-student speech.

In initial pilot tests, the talk meter was posted on the tutor’s video screen for the entire one-hour tutoring session, but tutors found that too distracting. The study was revised so that the meter pops up every 20 minutes or three times during the session. When the student is talking less than 25 percent of the time, the meter goes red, indicating that improvement is needed. When the student is talking more than half the time, the meter turns green. In between, it’s yellow. 

More than 700 tutors and 1,200 of their students were randomly assigned to one of three groups: one where the tutors were shown the talk meter, another where both tutors and students were shown the talk meter, and a third “control” group which wasn’t shown the talk meter at all for comparison.

When just the tutors saw the talk meter, they tended to curtail their explanations and talk much less. But despite their efforts to prod their tutees to talk more, students increased their talking only by 7 percent. 

When students were also shown the talk meter, the dynamic changed. Students increased their talking by 18 percent. Introverts especially started speaking up, according to interviews with the tutors. 

The results show how teaching and learning is a two-way street. It’s not just about coaching teachers to be better at their craft. We also need to coach students to be better learners. 

“It’s not all the teacher’s responsibility to change student behavior,” said Dorottya Demszky, an assistant professor in education data science at Stanford University and lead author of the study. “I think it’s genuinely, super transformative to think of the student as part of it as well.”

The study hasn’t yet been published in a peer-reviewed journal and is currently a draft paper, “Does Feedback on Talk Time Increase Student Engagement? Evidence from a Randomized Controlled Trial on a Math Tutoring Platform,” so it may still be revised. It is slated to be presented at the March 2024 annual conference of the Society of Learning Analytics in Kyoto, Japan. 

In analyzing the sound files, Demszky noticed that students tended to work on their practice problems with the tutor more silently in both the control and tutor-only talk meter groups. But students started to verbalize their steps aloud once they saw the talk meter. Students were filling more of the silences.

In interviews with the researchers, students said the meter made the tutoring session feel like a game.  One student said, “It’s like a competition. So if you talk more, it’s like, I think you’re better at it.” Another noted:  “When I see that it’s red, I get a little bit sad and then I keep on talking, then I see it yellow, and then I keep on talking more. Then I see it green and then I’m super happy.” 

Some students found the meter distracting.  “It can get annoying because sometimes when I’m trying to look at a question, it just appears, and then sometimes I can’t get rid of it,” one said.

Tutors had mixed reactions, too. For many, the talk meter was a helpful reminder not to be long-winded in their explanations and to ask more probing, open-ended questions. Some tutors said they felt pressured to reach a 50-50 ratio and that they were unnaturally holding back from speaking. One tutor pointed out that it’s not always desirable for a student to talk so much. When you’re introducing a new concept or the student is really lost and struggling, it may be better for the teacher to speak more. 

Surprisingly, kids didn’t just fill the air with silly talk to move the gauge. Demszky’s team analyzed the transcripts in a subset of the tutoring sessions and found that students were genuinely talking about their math work and expressing their reasoning. The use of math terms increased by 42 percent.

Unfortunately, there are several drawbacks to the study design. We don’t know if students’ math achievement improved from the talk meter. The problem was that students of different ages were learning different things in different grades and different countries and there was no single, standardized test to give them all. 

Another confounding factor is that students who saw the talk meter were also given extra information sessions and worksheets about the benefits of talking more. So we can’t tell from this experiment if the talk meter made the difference or if the information on the value of talking aloud would have been enough to get them to talk more.

Demszky is working on developing a talk meter app that can be used in traditional classrooms to encourage more student participation. She hopes teachers will share talk meter results with their students. “I think you could involve the students a little more: ‘It seems like some of you weren’t participating. Or it seems like my questions were very closed ended? How can we work on this together?’”

But she said she’s treading carefully because she is aware that there can be unintended consequences with measurement apps. She wants to give feedback not only on how much students are talking but also on the quality of what they are talking about. And natural language processing still has trouble with English in foreign accents and background noise. Beyond the technological hurdles, there are psychological ones too.

 “Not everyone wants a Fitbit or a tool that gives them metrics and feedback,” Demszky acknowledges.

This story about student participation was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Proof Points newsletter.

The post PROOF POINTS: How to get teachers to talk less and students more appeared first on The Hechinger Report.