ITP List (home)

December 3, 2021
  • ITP Ed Sciences: Dr. Daniel McCaffrey

    December 3, 2021  12:00 pm - 1:30 pm

    Dr. Daniel McCaffrey
    Educational Testing Service (ETS)

    "Are Your AI Scores Good Enough?"

    Use of computers to score performances from standardized evaluations, such as using artificial intelligence (AI) and natural language processing (NLP) to rate written or spoken responses to standardized test items, is growing rapidly in popularity. Currently, these methods are used in many testing situations to produce scores. For example ,tens of millions of responses from elementary and secondary students are scored using computer-based automated scoring and states like Ohio (Ohio Department ofEducation, 2018) are moving toward having all student responses from its elementary and secondary school testing program scored by such methods.Moreover, each year millions of responses from high stakes tests such as theGRER, TOEFLR, the Duolingo English Test, the Pearson Test of English and thePearson Test of English Academic are also scored by computer-based automated methods.

    Use of AI scoring for assessments invariably leads to questions about the ability of the scores to support the claims of the items and the tests and the fairness of the scores. Typically, evaluation of scores involves statistical analyses of the agreement between AI scores and human ratings of the same constructed responses or the accuracy of scores as predictors of the human ratings. In this talk I will discuss an alternative framework for evaluating AI scores that focuses on building evidence to support claims about the scores. I will discuss how to use statistical analysis of AI scores in this framework and methods for assessing the fairness of scores.

    See more details

December 10, 2021
  • ITP Ed Sciences: Noah Hirschl (ITP Fellow)

    December 10, 2021  12:00 pm - 1:30 pm
    Zoom only

    Noah Hirschl
    ITP Fellow (UW-Madison)

    "Advanced Placement Gatekeeping and Racialized Tracking"

    Racialized tracking is central to sociological explanations for racially stratified educational outcomes. However, school officials’ decision making is of debated importance for explaining racialized tracking. We contribute to this literature by examining the effects of schools’ enrollment policies for Advanced Placement (AP) courses. Using a unique combination of school survey data and administrative data from Wisconsin, we estimate plausibly causal effects of two competing school policies: allowing students complete discretion in AP enrollment, or alternatively, allowing school officials to enforce selection criteria, which we call gatekeeping. We find that gatekeeping has racially disproportionate effects on AP participation. While differences in prior achievement partially explain the especially large negative effects among students of color, gatekeeping produces racial disparities even among students with similar prior achievement. We further find that AP gatekeeping has longer-run effects, discouraging Black students from attending four-year colleges, particularly highly selective institutions.

    See more details