evaluation of a notebook program for teachers and principals

In 2018 / 2019 dandolo evaluated a state education department program which provided teachers and principals in government schools with access to notebook computers. One of our consulting analysts, Bokyong Mun interviewed Lotti O’Dea, who was the project manager for the evaluation about some of her highlights.

BM: Can you please give us a quick summary of what the project was about?

LO: dandolo was asked to evaluate a program that provided notebooks to teachers and principals in schools. The program aimed to equip teachers to better plan and teach as well as efficiently complete administrative work. The department wanted us to evaluate the design, implementation and impact of the program, particularly to see if providing the laptops to teachers were having beneficial flow on effects to students as well.

BM: Sounds like a big job! How did you approach the project / what was the team’s methodology going in?

LO: We focused on being comprehensive, particularly with our fieldwork. We carried out interviews, held focus groups at schools. We gathered an unprecedentedly large and rich collection of data about notebook use in schools, and we completed a major survey that was sent to 100 teachers on 100 random days during the year, so 10,000 in total. The survey asked about notebook use in the last 24 hours.

Undertaking all of the fieldwork was hugely valuable in how we understood and built frameworks to analyse our results. Specifically it helped us to understand how teachers in the program were using their laptops. Initially, we thought that any differences in how teachers used them would be linked to characteristics like the type of school, or social economic status and that this would be borne out in our survey data.

But from talking to teachers and students, we found that even in schools teachers were using the laptops quite differently. Our analysis of information we gathered from the surveys reflected this too, and we weren’t able to find a clear trend between how many hours teachers used their notebooks and the socioeconomic status of their school. We used this feedback to instead develop more nuanced profiles of the different user groups of teachers.

Developing these user profiles also provided a framework for outlining any future barriers that may be faced in promoting greater engagement with the program.

BM: What were some of the key conclusions that the team found?

LO: Unsurprisingly, we found that notebooks were indispensable to teachers, and were being used for significant amounts of time. To figure out how the notebooks were beneficial to teaching and student outcomes, we used the SAMR model to map out the different ways that teachers used their notebooks.1 Using the SAMR model meant we were then able to analyse and capture the specific ways that notebooks add value to how teachers go about their roles.

 [slide 4]In terms of improving teaching, the notebooks were most consistently being used to improve efficiency for things like teaching and lesson planning, or communication and administrative tasks.

But we found that around half of teachers didn’t use their notebooks for modification and redefinition activities – or uses that made their teaching more effective, and this was a big area in which we could provide recommendations to the department. Examples of some of these activities we did see included teachers personalising or customising learning to the student, or receiving real-time feedback from students on their engagement and understanding of the lesson.

Building these frameworks also helped us to communicate and illustrate our findings to the department.

BM: What was something in particular that you liked or are proud of about this project?

LO: I found the project interesting because we were able to provide really clear insights into an area where the department had very little visibility before.

An extra aspect of this project that was really valuable, was that the large number of surveys we sent allowed us to do some AB testing on the most effective way of sending out surveys. For example we found there was a higher response rate when we sent surveys out later in the day and where the subject line of the email sounded more urgent. On the other hand, factors like the day of the week, or the length of time between sending out reminders seemed to not to have any affect. These learnings have been really useful for other surveys that dandolo has done on other projects.

BM: What do you think makes this an iconic dandolo project?

LO: dandolo does a lot of work in the education space, and on face value this project can seem like just another evaluation of an education program. But looking beyond the covers we found the project was much more technology heavy and we ended up doing an evaluation that had a unique intersection of technology and education.

When we were working on the project, dandolo was also working on another evaluation project of the software program run in schools by the same state education department. This meant were able to provide joint analysis more broadly on the technology ecosystem in schools, based on our findings from both projects. Taking bits from both projects also allowed us to provide technological insights that were more fundamental and first-principles based, than we might have been able to otherwise. 

1 The SAMR model was created by Dr. Ruben Puentedura.