Brightpath to learning

Challenge

When assessing student work for tasks such as reading comprehension and mathematics, school teachers use multiple choice and short responses. In areas of learning that involve extended performances such as sport, visual arts, oral, musical and dance performances or those involving critical thinking such as information reporting, the process of assessing student work can be challenging. School teachers typically use a rubric or scoring guide to evaluate the quality of students’ responses against set criteria. However, the researchers have identified and published about issues associated with standard grid-like rubrics.

Innovative research on assessment methods began at The University of Western Australia around 2008 with its impetus stemming from a number of observations, including:

• the deficiencies identified in rubric assessment methods
• the lack of ‘fine-grained’ information about student learning provided by existing methods
• a complete lack of valid and reliable assessment methods for some difficult-to-assess learning areas
• concerns from the teaching fraternity about assessment methods such as multiple choice

The goal of the research was to help teachers to obtain valid, reliable, and comparable detailed information in a range of difficult-to-assess areas in a way they can use it to inform their teaching.

Solution

Stephen Humphry, Associate Professor in Educational Assessment, Measurement and Evaluation at UWA and Dr Sandy Heldsinger conducted preliminary research from 2008 to 2013, with grant funding from the Australian Research Council. In 2014, Associate Professor Stephen Humphry and Dr Sandy Heldsinger established the company Pairwise and successfully developed and commercialised the Brightpath software, a breakthrough in the assessment of students in a range of learning areas including those not amenable to large scale testing.

How does it work?

The method of pairwise comparisons goes back over a century. In educational contexts, for each comparison teachers simply judge which of two performances is better. There is a growing body of research in which teachers use long series of pairwise comparisons to assess student work. Although reliable results are produced in a wide range of areas, the process is very time consuming.

Harnessing their reliability in marking and the advanced abilities of teachers, the researchers developed a novel two-stage approach, built on pairwise comparisons:

• Stage 1: Pairwise comparison scales were developed by a small group of teachers for each area. The results were then calibrated and teachers were provided with exemplars. The WA Principals Association were instrumental to this process helping to coordinate work by teachers in many schools to collect student work samples and subsequently compare performances.
• Stage 2: Based on both statistical and qualitative analysis, sets of exemplars are then selected as anchor points on a scale, which is referred to as the “teacher’s ruler”. The teacher’s job then then becomes simpler and more efficient. When assessing work, they simply determine where the child slots in on the scale. Brightpath also supports teachers by using brief descriptors and annotations in conjunction with the exemplars.

By harnessing teacher judgement and involving them when setting up the scales, the outcome is an assessment method that is as straightforward as possible for teachers, accurate, reliable and applicable in a wide range of learning areas.

Impact Pathway

The School Curriculum and Standards Authority, which is responsible for assessments, standards and reporting in WA for Kindergarten to year 12 want comparable grades for student assessments in all learning areas (i.e. B represents the same range of performance levels at each school). The Brightpath program is working towards achieving this goal of comparability across the board, with a particular long-term focus on difficult-to-assess areas such as Oral Language and Critical Reasoning.

The greatest impact is realised by teachers who will have an effective tool that can assess a broad range of areas. Teachers also receive data as feedback which they can use to compare students work and track student learning over a period of time, such as a year, which it is hoped will benefit progression.

The assessment method is currently operating in collaboration with School Curriculum and Standards Authority (SCASA), Western Australian Primary Principals’ Association (WAPPA), Association of Independent Schools of Western Australia (AISWA) and Catholic Education Western Australia (CEOWA).

Within the next five years, Brightpath will be rolled out to all primary and secondary schools in Western Australia including government, private and Catholic schools.

Whilst it will initially operate in Western Australia, Pairwise is also in discussion with stakeholders in other states and territories of Australia.

There is an appetite for such a solution—one that has impact both in Australia and even globally. As well, the impact of this research could translate across other industries, wherever performance assessments are used such as in job interviews, auditions and so forth. For now however, the focus is on the education sector.

The Pairwise and Brightpath story is one in which years of research led to the commercialisation of an assessment approach that has successful outcomes for students, teachers and parents alike.