Phuong Tran, Monash College
The use of many-facet rasch analysis in improving rater reliability in rating writing assessments

When language institutions employ multiple raters to rate students’ written work for assessment purposes, they often face the challenge of ensuring inter-rater reliability and intra-rater reliability. Training raters before marking sessions, moderating papers during the marking sessions, and providing raters with feedback on their rating performance afterwards are common solutions to the challenge. In the process of doing this, institutions usually draw on different types of data to help them make decisions. This presentation reports on the use of many-facet Rasch analysis to support the process at Monash University English Language Centre (MUELC) since 2016. The three years of implementing this approach has given MUELC valuable lessons that are worth sharing.

During this presentation I will cover:

- How the scores that all raters award to all criteria in rating all students’ written work in a writing assessment event are collected and analysed using Facets (Many-facet Rasch measurement) program.
- What the results reveal about the performance of raters, rating criteria, and students.
- How this information can be used to evaluate raters’ consistency and accuracy in marking, identify underperforming raters that need to undergo further training, and specify specific areas of rating on which they need to improve.
- How criteria that tend to be harder to rate consistently or accurately can be identified from the results and targeted in the rater training sessions."

Dr Phuong Tran is Leading Specialist Teacher, Assessment at Monash College in Melbourne, Australia. She leads the Assessment team to develop, validate and assure the quality of assessments across all English language programs. Phuong has a Master of Education (TESOL) from Monash University and a PhD in Education from The University of Melbourne. She has extensive experience in language teaching, teacher training, and assessment design, development and validation. Her areas of interest include assessment development and validation was well as curriculum design and management.