Challenge

When assessing student work for tasks such as reading comprehension and mathematics, school teachers use multiple choice and short responses. In areas of learning that involve extended performances such as sport, visual arts, oral, musical and dance performances or those involving critical thinking such as information reporting, the process of assessing student work can be challenging. School teachers typically use a rubric or scoring guide to evaluate the quality of students’ responses against set criteria. However, the researchers have identified and published about issues associated with standard grid-like rubrics.

Innovative research on assessment methods began at The University of Western Australia around 2008 with its impetus stemming from a number of observations, including:

• the deficiencies identified in rubric assessment methods
• the lack of ‘fine-grained’ information about student learning provided by existing methods
• a complete lack of valid and reliable assessment methods for some difficult-to-assess learning areas
• concerns from the teaching fraternity about assessment methods such as multiple choice

The goal of the research was to help teachers to obtain valid, reliable, and comparable detailed information in a range of difficult-to-assess areas in a way they can use it to inform their teaching.

Solution

Stephen Humphry, Associate Professor in Educational Assessment, Measurement and Evaluation at UWA and Dr Sandy Heldsinger conducted preliminary research from 2008 to 2013, with grant funding from the Australian Research Council. In 2014, Associate Professor Stephen Humphry and Dr Sandy Heldsinger established the company Pairwise and successfully developed and commercialised the Brightpath software, a breakthrough in the assessment of students in a range of learning areas including those not amenable to large scale testing.

How does it work?

The method of pairwise comparisons goes back over a century. In educational contexts, for each comparison teachers simply judge which of two performances is better. There is a growing body of research in which teachers use long series of pairwise comparisons to assess student work. Although reliable results are produced in a wide range of areas, the process is very time consuming.

Harnessing their reliability in marking and the advanced abilities of teachers, the researchers developed a novel two-stage approach, built on pairwise comparisons:

• Stage 1: Pairwise comparison scales were developed by a small group of teachers for each area. The results were then calibrated and teachers were provided with exemplars. The WA Principals Association were instrumental to this process helping to coordinate work by teachers in many schools to collect student work samples and subsequently compare performances.
• Stage 2: Based on both statistical and qualitative analysis, sets of exemplars are then selected as anchor points on a scale, which is referred to as the “teacher’s ruler”. The teacher’s job then then becomes simpler and more efficient. When assessing work, they simply determine where the child slots in on the scale. Brightpath also supports teachers by using brief descriptors and annotations in conjunction with the exemplars (see figure 1).

By harnessing teacher judgement and involving them when setting up the scales, the outcome is an assessment method that is as straightforward as possible for teachers, accurate, reliable and applicable in a wide range of learning areas.

Impact

980 schools across Australia are now using Brightpath and as at August 2020, over 20,000 teachers have completed over one million assessments within the platform.

Since its release late in 2014, Brightpath has proven to be leading educational reform in Australian education.

In 2019, an expert panel comprising Christine Cawsey, Professor John Hattie and Professor Geoff Masters, reported to the Federal Education Minister that Brightpath was one of the few tools available to schools that ‘aligned with well-constructed learning progressions and capable of providing information about the points students have reached in their learning and the growth they have made over time.’ [1]

Fig. 2 Learning Progressions

An evaluation was undertaken by an independent organisation in 2018 and 2019 that showed significantly greater improvement in NAPLAN Writing results for students in high-usage Brightpath schools relative to a control group. The peer-reviewed literature details the capacity of the method implemented with Brightpath to diagnose a student’s current level of knowledge, skill and ability and to identify next steps.

There is an appetite for such a solution—one that has impact both in Australia and even globally. As well, the impact of this research could translate across other industries, wherever performance assessments are used such as in job interviews, auditions and so forth. For now however, the focus is on the education sector.

The Pairwise and Brightpath story is one in which years of research led to the commercialisation of an assessment approach that has successful outcomes for students, teachers and parents alike.