By Bogdan Yamkovenko, Efficacy Research Manager, and Anna Bergstrom, Senior Donor Communications Manager at Khan Academy
Learning is our North Star
The Khan Academy platform is grounded in learning science and mastery-based pedagogy. We are committed to ensuring anyone, anywhere has access to an education that enables students to learn at their own pace, fill knowledge gaps, and accelerate learning. We are equally committed to understanding the effectiveness of our product on learning outcomes. Our team of efficacy researchers regularly conducts studies to test the efficacy of our platform and determine the impact of our work on student learning. Efficacy study results inform our product improvements and implementation guidelines and are also shared with the larger academic community, educators, policymakers, and community leaders to help them make more informed, data-driven decisions.
I. Latest Efficacy Study Results:
Khan Academy has conducted multiple efficacy studies with districts that use our KAD (Khan Academy Districts) and MAP Accelerator1 products. According to our recent study of ~350K students in grades 3-8 during the 2022-23 school year2:
- Using Khan Academy for 30+ minutes per week, or 18+ hours over the duration of the school year, is associated with ~20% greater-than-expected learning gains, as measured by performance on the nationally normed MAP Growth Assessment. Findings are consistent across demographic groups. (See figure 1).
- Approximately 9% (~33K) of students in the sample reached the recommended dosage of 30+ minutes per week (or 18+ hours per school year).
- The effect size in this study is .36.
While it’s true that the students in the study sample who spent the most time on the Khan Academy platform—18+ hours per school year—experienced the greatest learning gains, students who spent less time than the recommended dosage also experienced increased learning. For example, the 15% (55K) of students in the sample who used Khan Academy 9-18 hours over the duration of the school year experienced ~7% greater-than-expected learning gains.
1 MAP Accelerator is a personalized learning tool that uses MAP® Growth™ scores to offer personalized learning pathways for individual students. MAP® Growth™ is a dynamic, adaptive assessment that measures students’ achievement and growth in K–12 math, reading, language usage, and science. More information on MAP Accelerator can be accessed here.
2 This study included ~350K students in grades 3-8 who completed fall and spring MAP Growth Assessments during the 2022-23 school year.
Figure 1
Year Over Year Comparison: This is the third consecutive year that Khan Academy has conducted the aforementioned study, and today there are ~221K students for whom we have collected two or more years of data. Comparing these ~221K students to themselves over time enables us to perform a deeper, more rigorous analysis that rules out factors like the impact of a highly motivated student or an above-average teacher on student learning outcomes. This longitudinal analysis examines the following:
- How student learning gains change year to year as students increase or decrease usage of Khan Academy
- How gains on the MAP Growth test change year to year as students use Khan Academy to learn more or fewer skills relative to the prior year, where skills are learned to proficient or above. (Note that on the Khan Academy platform, skill levels range from attempted, familiar, and proficient to mastered. Students demonstrate which skills they have learned and their current skill level by answering questions correctly on exercises, quizzes, unit tests, or mastery challenges.)
Results from this rigorous longitudinal study show:
- Each additional skill practiced to proficient or mastered3 results in ~.5pp in learning gains relative to a student’s own results from the prior year. Read more on the Khan Academy blog. (See figure 2).
- Approximately 52% (~115K) of students in the sample increased the number of skills learned to proficient or mastered year over year, whereas ~48% (106K) decreased the number of skills learned compared to the prior year. Students who increased the number of skills learned to proficient or mastered saw incremental learning gains, whereas those who decreased their usage saw incremental declines.
- The amount of incremental learning gains depends on individual students’ learning targets and the total number of additional skills learned to proficient or mastered year-to-year. For example, students who increase the number of skills they practice to proficient or mastered by 1-15 skills (~ 30% of the sample, ~66K students) tend to see, on average, 5pp gains. Those who increase their practice by 15-30 skills (~10% of the sample, 20K students) tend to see, on average, 15pp gains. Approximately 4% (9.5K) of students in our longitudinal study sample increased the number of skills they learned to proficient or mastered by 60 or more as compared to the previous year. These students typically see ~30pp increase in gains. Such gains for an average student in the sample translate to about 20%-30% increase in learning.
3 On the Khan Academy platform, skill levels range from attempted, familiar, and proficient to mastered. Students demonstrate which skills they have learned and their current skill level by answering questions correctly on exercises, quizzes, tests, or mastery challenges.
Figure 2
Longitudinal studies like this one are considered rigorous, relatively rare, and quasi-experimental. Few edtech studies have participants that number in the hundreds of thousands, and only a randomized controlled trial (RCT) is considered more rigorous. Additionally, in this study we included all students regardless of how much Khan Academy they used, which helps address one of the more common edtech study criticisms that edtech tools only work for motivated students. The results of this more rigorous longitudinal research give us a causal link between learning gains and Khan Academy usage, both time spent and skills practiced to proficient or mastered.
II. Illustrative Example – Using Khan Academy to Close Student Learning Gaps:
To put these results into perspective, consider that an average 5th-grade student is expected to gain ~10 points between their fall and spring MAP Growth tests, according to NWEA (the test maker). However, students typically gain less than their target, so this student gains 8 points. Over time, if this typical rate of learning continues, it’s likely that an average student’s learning gap will increase.
However, using Khan Academy to supplement classroom learning, a hypothetically average student can meet or even exceed their learning target by becoming proficient in more skills than achieved the prior year in the following way:
Consider a 5th-grade student who works on one unit of Khan Academy’s 5th-grade math course and completes very few skills—10 skills out of the 130 skills in the course—beyond what they did in other curricula/resources. This student ends the school year with gains on the MAP Growth Assessment that are 80% of their 5th-grade target, or ~8 points higher than what they scored on their fall assessment but 2 points less than their targeted level of growth. Now, suppose in the next year they use Khan Academy to supplement what they’re learning in their core curriculum and become proficient in 50 skills in 6th grade—that’s 40 more skills compared to what they learned on Khan Academy the year before (when they learned 10 skills on Khan Academy). Based on our research finding that each additional skill learned to proficiency or mastery results in .5pp increase in learning, we would expect this student to move 20 percentage points closer to their target. This is a significant amount of learning: 20% incremental learning from skills practice on Khan Academy combined with 80% of this student’s individual target at the end of grade 5 means this student would likely meet 100% of their learning target in their 6th-grade year. If the same student becomes proficient in even more skills in the same year beyond the 40 noted in this example, they could even exceed their learning target.
III. Organizational Targets and Implementation Guidelines Informed by Efficacy Research:
At Khan Academy, we extrapolate from efficacy research findings to set organizational targets that drive improved student outcomes and inform implementation guidelines for educators that are designed to address the learning gap. Given that most students are only reaching ~85% of their target growth, setting implementation guidelines to accelerate learning by 30% means students have a clear path to meet or even exceed their learning target. We are focused on getting as many students as possible to engage deeply with Khan Academy in order to drive improved outcomes (both time and skills to proficient), so we set organizational targets for the following metrics and provide educators with the following practical implementation guidelines:
- Yearly Very Active Learners (YVALs): 18+ hours over the school year or 30+ minutes per week | This guideline is informed by multiple years of efficacy research that show students who achieve this deep use of Khan Academy experience ~20-30% higher-than-expected learning gains. Approximately 9% (~33K) of students in our 2022-2023 school year study sample became YVALs.
Yearly Proficient Learners (YPL): 60+ skills to proficient over the duration of the school yearor 2+ skills to proficient per week | This implementation guideline is informed by efficacy research findings that show year-over-year increase in skills learned to proficient or mastered improves learning by ~.5pp. However, tracking year-over-year skill gains to proficient or mastered isn’t practical in many cases. As a result, we are recommending 60+ skills as a simplified and practical proxy. While students, especially those who are significantly behind, can see 30% gains with even less practice, we set 60+ skills as a rigorous benchmark. This year we have begun to introduce the Yearly Proficient Learners metric to educators in our Districts Partnerships Program.
Looking Ahead
We will continue conducting additional studies of our core platform, including a study that is underway in partnership with the University of Toronto and Stanford that takes additional steps beyond our existing research to isolate the impact of Khan Academy on learning outcomes. Additionally, studies are underway to understand how Khanmigo, Khan Academy’s artificial intelligence tutor for students and assistant for teachers, impacts learning outcomes. Khanmigo-focused efficacy studies will focus on the extent to which Khanmigo impacts learning by determining how it helps learners progress through Khan Academy content and the impact of Khanmigo on students’ test scores outside of the Khan Academy platform.