May 1, 2023
"Although transforming discussion quality is central to achieving ambitious education reform goals, analyzing discussion quality in a deep and nuanced way in a large number of classrooms is infeasible for most researchers."
High-quality classroom discussions are crucial to the proper development of reading, writing, and argumentation skills in students. Such student-centered discussions are referred to as "dialogic" in how they encourage students to respond to open-ended and interpretative questions through interactive dialogue with their instructor and peers. A newer and bolder concept than traditional classroom discussions that expect instructors to pose "fact-based" questions with pre-determined answers, dialogic discourse is essential to acheiving ambitious education reform goals. However, analyzing discussion quality in an effective and in-depth manner across a multitude of classrooms is impractical and tedious.
This is where ChatGPT enters the conversation. ChatGPT is a novel, cutting-edge AI-driven tool with untapped potential for automating the analysis of classroom discussions at a larger scale than ever seen before. In the past, artificial intelligence technologies through Natural Language Processing (NLP) have relied on supervised machine learning (ML), a method that demands arduous hand-coding by human annotators, and on the use of language models (LM), which predict the probability of textual data.
The usage of ChatGPT marks a shift to prompt-based learning, where a single LM can be used to generate multiple outputs with fewer amounts of data. ChatGPT thus has the potential to analyze classroom discourse - and human language more generally - in a much shorter amount of time with fewer resources and costs. With this growing potential in mind, LRDC researchers Lindsay Clare Matsumura, Professor, Learning Sciences and Policy Program, School of Education (SOE); Richard Correnti, Associate Professor, SOE, Amanda Godley, Professor, SOE, and Vice Provost for Graduate Studies; and Diane Litman, Professor, School of Computing and Information (SCI), along with two graduate student, will investigate ChatGPT's potential for analyzing and assessing large datasets of classroom-based instructor and student discourse.
Their proposed methodology incorporates various iterative design cycles of prompt engineering, examining the extent to which ChatGPT is able to detect features of high-quality versus low-quality discussions in elementary and high school classrooms in line with human and machine-generated codes. The researchers will additionally build on the findings and corpora from two previous research studies that similarly use feature-based models to examine classroom discussions.
By comparing and contrasting across feature-based models and ChatGPT within elementary and high school classroom discourse and corpora, the researchers will assess the strengths and weaknesses of each approach for large-scale analysis of classroom discussion quality. As a result, they may use these findings to advance further research and formative assessment to support teacher learning.
Crosson, A., C., Correnti, R., Matsumura, L., C., & McKeown, M., G. (2023). Effects of the Triple Q Intervention on Argument Writing: Findings from a Small-Scale Cluster-Randomized Controlled Trial. Journal of Research on Educational Effectiveness.
Correnti, R., Matsumura, L.C., Wang, E., Litman, D., Zhang, H. (2022). Building a validity argument for an automated writing evaluation system (eRevise) as a formative assessment. Computers and Education Open.
Correnti, R., Matsumura, L.C., Walsh, M.W., Zook-Howell, D., & Bickel, D.D. (2021). Effects of Online Content-Focused Coaching on discussion quality and reading achievement: Building theory for how coaching develops teachers' adaptive expertise. Reading Research Quarterly.
Litman, D., Zhang, H., Correnti, R., Matsumura, L.C., & Wang, E. (2021). A fairness evaluation of automated methods for scoring text evidence usage in writing. International Conference on Artificial Intelligence in Education.