Quant Hub seminar: Investigation of Rater Effects using Graph Theory and Exponential Random Graph Models: examples from high-stakes examination data
This presentation investigates rater effects in high-stakes examinations using Graph Theory (GT) and Exponential Random Graph Models (ERGM). Rater effects, such as biases and inconsistencies, are common challenges in the scoring process, impacting the fairness and validity of assessments. GT and ERGM offer powerful tools to visualize complex rater interactions and reveal hidden patterns that traditional reliability indices, such as Krippendorff’s alpha or Fleiss’s kappa, may not capture.

Using real-world examination data, we demonstrate how graph-based models provide nuanced insights into rater behavior, allowing for the identification of outlier raters and systematic biases. This approach is not intended to replace established methods (e.g., Rasch models) but rather to complement them.

Our empirical analysis shows that GT indices and Rasch model estimates are strongly correlated, highlighting their complementarity. Additionally, certain GT indices can be mathematically linked to Rasch model estimates through the pairwise estimation method, further strengthening their theoretical connection.

Teams link to join online: teams.microsoft.com/l/meetup-join/19%3ameeting_MDFhZjExMjgtNmYxMS00MmUzLTljMDItYTYyNDdlZWQ1Yzc1%40thread.v2/0?context=%7b%22Tid%22%3a%22cc95de1b-97f5-4f93-b4ba-fe68b852cf91%22%2c%22Oid%22%3a%22b33f55d8-6202-46f8-a141-737715faff88%22%7d
Date: 18 November 2024, 13:00 (Monday, 6th week, Michaelmas 2024)
Venue: 15 Norham Gardens, 15 Norham Gardens OX2 6PY
Venue Details: Seminar Room A and Teams
Speaker: Dr. Iasonas Lamprianou (University of Cyprus, Cyprus)
Organising department: Department of Education
Organiser: Lars-Erik Malmberg (University of Oxford)
Part of: Quantitative Methods Hub – Seminar Programme MT 2024
Booking required?: Not required
Booking url: https://www.education.ox.ac.uk/quantitative-methods-hub-seminar-programme-mt-2024/
Cost: Free
Audience: Public
Editor: Kristina Khoo