Independently evaluated report reveals Learning MATTERS’ differentiated T2 & T3 intervention accelerated literacy progress.
Thursday 16th May 2024
In 2022, Learning MATTERS was chosen as one of three providers to trial literacy and communication (lit/comm) support packages, aimed at accelerating support for learners in Years 2 to 8 with Tier 2 and 3 needs. The trial, funded by Te Mahau, within the Ministry of Education, has been independently evaluated by a consortium of Ripple Research, Design and Evaluation, and Kōtātā Insight.
Our understanding of the trial's objective was not to compare packages with other providers but rather to evaluate the effectiveness of targeted interventions providing accelerated support for learners from Years 2 to 8 with Tier 2 (T2) and Tier 3 (T3) needs, employing a diagnostic, explicit, systematic, and cumulative approach. The purpose of the trial was to obtain data on the impact of interventions for learners who hadn’t previously experienced such approaches and/or had experienced challenges that impacted their progress and access to wider learning.
While the report cannot be used for provider comparison, it serves as evidence that the approach employed by Learning MATTERS is highly effective at accelerating achievement for children classified as needing T2 or T3 support. The full report can be found here.
Focusing on Learning MATTERS’ participation in the trial, our methodology facilitated a comprehensive analysis of overall impact, unveiling statistically proven accelerated progress at both T2 and T3 interventions.
Unpacking the Methodology
With just over two terms available, Learning MATTERS chose to provide initial training with face-to-face professional learning which was tailored to equip educators with the techniques and resources to confidently use the iDeaL Approach*. Following this, educators were supported with coaching and mentoring sessions, both in-person and remotely.
With a focus on Years 4 to 6*, the trial strategically targeted students who had previously struggled academically, aiming to intervene effectively to improve their literacy skills while nurturing their self-esteem. To accomplish this, a whole picture of each student was required. Learning MATTERS’ methodology involved universal screening of all Year 4-6 students within the trial schools, ensuring a true understanding of students requiring T2 and T3 interventions. This data served as a foundation for determining the allocation of T2 and T3 cohorts. Trial teachers were guided by and adhered to Learning MATTERS’ process for allocating learning support for T2 and T3 literacy intervention. This enabled control groups to be formed that mirrored a set of learners with similar needs to those in the trial which allowed a determination of the impact of the intervention. Students’ progress was tracked throughout the trial to understand the changes in literacy skills that might occur without the intervention. Results from the screening and initial progress monitoring served as a baseline measure, crucial for attributing any observed changes in literacy skills to the intervention. This also provided teachers with the knowledge and skills to implement and interpret assessment data.
The Literacy and Communication Package Trial Final Evaluation Report states, 'This was the strongest research design for the purposes of statistical impact analysis' (Page 4).
In the 15 schools served in this study by Learning MATTERS, T2 teachers conducted three 20-minute sessions per week for T2 participants in the classroom, while T3 teachers delivered four 30-minute sessions per week for T3 participants outside the classroom, spanning 19 weeks.
Tier 2 instruction focused on utilising the iDeaL Online Platform, delivering spelling lessons to students. Tier 3 sessions encompassed explicit teaching of decoding and encoding skills, incorporating both spelling and reading of decodable texts. The iDeaL Online Platform was employed by both T2 and T3 teachers to support their teaching throughout the 19-week teaching timeframe.
The approach implemented was systematic, diagnostic, and explicit. It would appear from the graph below that the iDeaL Approach is an aspect of the Learning MATTERS trial delivery that had a significant impact.
Entries below taken from Learning MATTERS’ journalling observations.
This graph signals the cumulative effect of mutually reinforcing strategies. The initial training, the iDeaL online portal, and learning materials are foundational to teacher PLD and show a similar frequency of attribution. Above and beyond that, the impact of coaching, mentoring, and guided reflection significantly influences teacher confidence and capability. Then as schools and teachers become more familiar with all aspects of the iDeaL Approach model, the whole approach becomes increasingly influential.
The T3 teachers were deliberately granted adequate release time, reflecting a commitment to sustainability and the enhancement of T3 teachers' skills by the school leadership. The T3 teachers supported the most vulnerable learners within the cohort. Over 18 weeks, this strategic allocation of resources likely played a significant role in positively impacting these students.
Throughout the trial, Learning MATTERS was committed to fidelity of delivery, demonstrated by the completion of 84% of sessions by participating students, as indicated by our data captured. This dedication yielded significant improvements in foundational literacy skills, underscoring the effectiveness of the trial's differentiated approach, particularly advantageous for T3 students.
Drilling into the Data
Learning MATTERS has purposely focused on intervention results compared to the control (baseline progress) to truly identify the impact of the intervention in accelerating literacy progress. Numerical changes that are not statistically valid or need to be inferred will not be discussed here.
Learning MATTERS’ robust trial design and fidelity of delivery demonstrated the most significant response to intervention measured by the change in the Bryant Score and when statistically analysed.
Effect size
The effect size is calculated from the annualised progress estimated from the change in scores at the end of the intervention period. Effect sizes should be assessed against the benchmark of 1.0, indicating one year of learning expectation.
Learning MATTERS’ T2 and T3 intervention groups exhibited higher effect sizes compared to the T2 and T3 control groups, indicating the intervention's efficacy. Conversely, the effect sizes of the control groups aligned closely with those typical of standard classroom learning (1.0).
Refer to Table 6 below, illustrating ‘Summary literacy achievement result by provider – effect sizes’, found on page 37 of the Literacy and Communication Package Trial Final Evaluation Report.
Table 6
Learning MATTERS’ intervention impact above baseline (control) shows accelerated progress and a reduction in the tail of under-achievement
The change in the distribution of achievement scores from the beginning of the trial to the end is shown below (Figure 12, page 40) for both the intervention and the control for Learning MATTERS. The control group (“wheat” colour) shows that even without the intervention there is a growth in learners’ decoding skills. In contrast, there is a substantial difference in learners’ skill development for the intervention group (“teal” colour) over the same period. Initially similar distributions at the trial's commencement support the balanced allocation into treatment and control, as well as endorse the robust trial design. However, by the trial's end, the median for the intervention group notably surpasses that of the control group, with fewer learners at the lower end of the scale.
Refer to Figure 12 below, illustrating the 'Bryant score distribution over three points in time for LM', found on page 40 of the Literacy and Communication Package Trial Final Evaluation Report.
Figure 12
Treatment effect
The treatment effect serves as a primary parameter to quantify the level of impact that can be attributed solely to the intervention. The treatment effect is the gain in decoding skills above what would be expected without the intervention. This is measured with the Adapted Bryant score points.
By statistically 'removing' any random school, student, or time/dose parameters that could potentially artificially influence the raw results, the treatment effect provides a clearer picture of the intervention's efficacy. Table 7 illustrates both the overall increase in decoding skill observed during the trial and the estimated effect of the intervention – the treatment effect.
Refer to Table 7 below, illustrating the 'Overall intervention effect', found on page 42 of the Literacy and Communication Package Trial Final Evaluation Report.
Table 7
Learning MATTERS’ treatment effect results strongly affirm the direct impact of both the T2 and T3 interventions in accelerating progress compared to the control group.
The impact of “Dose”
Key findings show a strong positive correlation between dosage and decoding skill improvement, underlining the crucial role of differentiated practice and targeted/individualised support in fostering literacy growth.
By examining students' actual session attendance, we are able to delve deeper into the relationship between dosage and its impact. Figure 11 illustrates a clear positive correlation between dosage and decoding skill improvement, utilising data from Learning MATTERS. Each data point in the figure represents an individual student, plotting the number of sessions attended against the increase in decoding skills.
Learning MATTERS identifies two key factors influencing dosage impact:
Number and duration of sessions: T3 sessions featured both longer durations and more sessions.
Instructional content and teaching personnel: Variations in who taught and what was taught during T2 and T3 sessions were significant.
Learning MATTERS believes that consistent and increased exposure (both in terms of time and intensity) not only benefited students but also facilitated the professional growth of teachers. This effect was particularly pronounced among T3 teachers, who were afforded dedicated time to refine their teaching skills without classroom responsibilities.
Refer to Figure 11 below, illustrating the 'Impact of dosage on literacy achievement for Learning MATTERS', found on page 39 of the Literacy and Communication Package Trial Final Evaluation Report.
Figure 11
Qualitative insights
Insights gleaned from the Learning MATTERS’ trial journal entries illustrate a picture of improved student engagement, enhanced confidence, and unexpected progress among ESOL students. Leaning MATTERS' attributes these accomplishments to the focus on professional learning, which prioritised explicit teaching methodologies and ongoing coaching, ensuring sustained progress and growth.
Capturing key themes from the trial journal entries, several factors contributed positively to the rates of progress achieved, including:
Teaching sessions provided 3-4 times per week
Increased teacher knowledge
Supporting the development of teacher practice
Implementation of explicit teaching
Access to suitable resources
A systematic and thorough scope and sequence for instructional planning
Repetition and review factored into lesson sequences
Teachers videoing their own practice
Modelling and observations of lessons with feedback and feedforward
Trial teachers adhering mostly to the lesson sequence
Supportive and empathetic relationships developed between the consultant and Trial teachers.
Final reflection
As Learning MATTERS reflects on its trial achievements, we are proud of how the differentiated T2 & T3 intervention accelerated literacy progress. We also recognise the limitations of the short study time frame and requirements for a more longitudinal study to determine if the gains observed are retained.
We would welcome further involvement in research that measures the impact of increased literacy skills over a sustained period of intervention. Further research could explore the specific inclusions (most effective) of;
Suitability of allocations into tiers of instruction
Elements of lesson plans
Levels of teacher practice and correlations to progress made
Monitor the ongoing achievement (or lack thereof) and progress (after cessation of the intervention) against a cohort of peers in a control group.
Following the trial, Learning MATTERS has continued relationships with all 15 schools, sustaining the literacy teaching beyond the trial period.
Learning MATTERS believes there are significant insights from the trial approach and design that should be used to inform policy development, T2 & T3 PLD inclusions for teachers (including modelling and coaching), and frameworks to support classroom teaching and practice for T2 and out-of-the-classroom T3 intervention, including correct allocation into T2 & T3 cohorts.
We extend our gratitude to Dr. Jan Hasbrouck for peer reviewing Learning MATTERS’ summary of the report.
For more information contact:
Rachael Griffin, Communications, Marketing and Sales
Email: marketing@learningmatters.co.nz
* The iDeaL Approach
iDeaL is an acronym for Individualised Diagnostic Explicit Approach to Learning and is utilised by more than 500 kura throughout Aotearoa across Tier 1, 2, and 3 settings.
The iDeaL Teaching Approach is supported by an online Moodle-based platform (the iDeaL Online Platform) which houses professional learning materials, instructional videos, assessment and reporting tools and functionality, a comprehensive suite of systematic and cumulative resources, and leadership support.
** LM chose to focus on the Years 4 – 6, Why?
Students in this age range have had three years at school and not achieved success. It is not too late to intervene and preserve self-esteem.
Budget limitations restricted the student numbers required to get appropriate sample sizes for effective trial results across year levels 2-8.
Learning MATTERS Chit Chat - Literacy Trial
Click here to view our Chit Chat from 2023 where we discussed various aspects of the literacy & communication support package. The conversation delved into the trial's design, implementation strategies, and the tangible impact it had on participants.
Get in touch
If you have any questions, please do not hesitate to reach out to our team.
