Coursework: ITP Seminar

Dr. Daniel McCaffrey
Educational Testing Service (ETS)

"Are Your AI Scores Good Enough?"

Use of computers to score performancesfrom standardized evaluations, such as using artificial intelligence (AI) andnatural language processing (NLP) to rate written or spoken responses tostandardized test items, is growing rapidly in popularity. Currently, thesemethods are used in many testing situations to produce scores. For example,tens of millions of responses from elementary and secondary students are scoredusing computer-based automated scoring and states like Ohio (Ohio Department ofEducation, 2018) are moving toward having all student responses from itselementary and secondary school testing program scored by such methods.Moreover, each year millions of responses from high stakes tests such as theGRER, TOEFLR, the Duolingo English Test, the Pearson Test of English and thePearson Test of English Academic are also scored by computer-based automatedmethods.

 Use of AI scoring for assessments invariablyleads to questions about the ability of the scores to support the claims of theitems and the tests and the fairness of the scores. Typically, evaluation ofscores involves statistical analyses of the agreement between AI scores andhuman ratings of the same constructed responses or the accuracy of scores aspredictors of the human ratings. In this talk I will discuss an alternativeframework for evaluating AI scores that focuses on building evidence to supportclaims about the scores. I will discuss how to use statistical analysis of AIscores in this framework and methods for assessing the fairness of scores.