Impact Summary

UWA researchers have developed an innovative assessment method to enable schools to streamline school assessment processes across the nation. Brightpath is an online tool in which samples of students work can be compared with exemplars and used provide standardised feedback in order to positive affect educational outcomes.

Brightpath has been adopted by approximately 500 schools and used for 300,000 assessments of Australian students. In addition, the researchers have impacted Australian education communities through critical involvement with: (i) the National Assessment Programme—Literacy and Numeracy (NAPLAN) involving over 1 million student participants per year; and (ii) the high-stakes Western Australian Certificate of Education (WACE) affecting 26,000 high school students each year.

Impact Case Study

Impact 1: commercialisation of the brightpath system

In 2014 the company Pairwise was established to develop a new online assessment method that could be used to streamline school assessment processes with the aim of commercialising the program. UWA licensed IP was used to develop the method and create research software for scale development (Pairwise web-application).

Nature of impact: The Brightpath application enables diagnostic assessment, measurement, and reporting in Writing (Information Reports, Essays, Narratives), Oral Language, and Science Investigations. Brightpath scales are also being developed in areas not amenable to large-scale testing, such as Visual Art and Critical Thinking. Brightpath provides information to track student learning over extended periods (i.e. longitudinally). Brightpath enables schools to meet key deficiencies of most assessment processes highlighted in the recent Gonski report “Review to Achieve Educational Excellence in Australian Schools”.

Beneficiaries, end-users and extent: At a classroom level, teachers use software to compare the performance of their students with other classes. At a school level, comparisons are made with all schools using the system. A large number of teachers use the system to produce end-of-semester reports for their students to complement other school-based assessments, in the same fashion that NAPLAN reports are used.

In 2016, approximately 400 schools across Australia were using Brightpath to assess and measure student performance and to produce reports, including school-level reports and individual student reports. Between 2015 and 2016, the number of assessments being made by teachers within the system increased by approximately fourfold. The usage of the system has occurred through collaboration with two main industry partners, the School Curriculum and Standards Authority (SCSA) and Department of Education and Child Development (DECD).

Translation of research: The translation of research outcomes into an accessible form is central to the design of the Brightpath. Once exemplars have been scaled using pairwise comparisons, empirically derived descriptions of the performances are produced by content experts. These are written in a form that is accessible to and interpretable by classroom teachers to enable them to act on the information when teaching students. Exemplars are displayed visually within the application on a scale that is derived from psychometric analysis of the pairwise comparison data. This makes the technical information from scaling using the Bradley-Terry-Luce model (logit scale) simple for teachers to engage with. Displays are used by school leaders and systems/agencies such as SCSA and DECD. An excerpt from an interview with a School Principal follows: “Why? Because I was dealing with NAPLAN data that was sterile, not in real time, and wasn’t owned by my staff … So I saw Brightpath. Brightpath had two answers for me. The two answers were this particular program is going to be owned by my staff. Why? Because the staff were going to generate samples of work from the students. They were going to then take those samples and compare it with a set of exemplars. They were then going to place these assessments on a teacher ruler. That would give them some data that they owned and could then work with, so that we generated a narrative for our school which was saying: Okay, these children are over here, you need to move them there. This is why they achieved that particular grade, and this is what you need to do to move them on.” –Lynda Fisher, Principal Carmel School.

Impact 2: impact on standardised testing and education standards

Professor Andrich and Dr Humphry have had a substantial impact through research and applied work on the National Assessment Programme-Literacy and Numeracy (NAPLAN) and the WA Online Literacy and Numeracy Assessment (OLNA) programme. Professor Andrich is a long-standing member of the Measurement Advisory Group for NAPLAN, which makes recommendations to the Australian Curriculum, Assessment and Reporting Authority (ACARA) about educational measurement directions and immediate decisions. Professor Andrich has been integrally involved in advising and conducting work on approaches to scaling, testing and dealing with misfit to psychometric models, and standard-setting procedures, among others.

Nature of social impact: Between 2011 and 2016, Dr Humphry led the Central Analysis of Data project for the NAPLAN programme (approx. $300K per annum). During this project, common NAPLAN scales were formed for the five assessment areas, vertically aligned across year levels and longitudinally aligned over time. Between 2012 and 2015, Dr Humphry also collaborated with SCSA to set up the standard-setting procedures that produced the minimal standards for OLNA and the scale equating designs to make sure students are assessed relative to the same standard over time. Each year in March and September, SCSA employs the Pairwise comparison application developed by Dr. Humphry at UWA to ensure the new Writing scale for OLNA is aligned with the original scale so that students are accurately assessed against the original standard.

Extent: On NAPLAN scales, results for over 1 million students are reported to systems, schools, teachers, students and parents and are intrinsic to the Department of Education’s objective in conducting NAPLAN: establishing minimum standards in education and identify progress of students at key stages of their educational development. On OLNA scales, to graduate from high school approximately 26,000 year 12 students in WA must now, in addition to other criteria, pass the standards set using the method designed by Humphry. Percentages of students meeting the standards are reported in the publicly available SCSA document Year 12 Student Achievement Data 2017 (School Curriculum and Standards Authority, Year 12 Student Achievement Data 2017, available online).

Associated Research:

Drs Humphry and Heldsinger conducted seminal research that identified deficiencies associated with standard grid-like rubric structures in research that strongly determined the NAPLAN Writing marking guide used since the program’s inception. They subsequently led the development of the innovative two-stage assessment method that forms the basis for the Brightpath web application. Professor Andrich and Dr Humphry have jointly published on standard-setting procedures. Dr Humphry’s research has spanned from 2002 to the present and has largely been conducted within the Pearson Psychometric Laboratory (PPL) at the Graduate School of Education.

The work of Professor Andrich and other researchers within the PPL substantially influenced efforts to develop and implement large-scale testing systems across Australia. Dr Humphry and Professor Andrich have directed funding toward building and maintaining a team of researchers over an extended period of time. Appointments of several staff have been made to support the work, including those of Dr Ida Marais (2007 to present), Dr Joshua McGrane (2010 to 2015), Ken Bredemeyer (PhD candidate-2015 to present), Sonia Sappl (2017 to present), and Dr Sandy Heldsinger (between 2008 and 2014 – prior to commercialisation of Brightpath). This team also works with a broader team of researchers and practitioners locally and internationally.