Peer review is increasingly as a pedagogical strategy. It is wonderfully complex from a cognitive perspective (in terms of what processes it invokes) and it can be supported with a wide range of simple to complex tools.

We do basic research on how students learn and applied research on what tools best support their learning. Also see: https://sites.google.com/site/swordlrdc/

SWoRD. SWoRD is a web-based reciprocal peer review system. In less fancy terms, students turn their class papers into SWoRD, which then assigns this paper to four to six peers in the class. The peers grade the paper and give advice for how to improve it. Students revise the paper and turn it back in to SWoRD, which distributes the paper to the same peers for final review. SWoRD determines the accuracy of the ratings through a complex process of separating out different kinds of bias in grading. The authors rate the advice given to them in terms of helpfulness. Reviewers get a grade for their work which is one half accuracy and one half helpfulness. In this way, reviewers must work hard and take their task seriously (see video). SWoRD has been used in many different courses (graduate and undergraduate), in many different disciplines and at many different universities. The grades that are produced are just as reliable and accurate as instructor grades, and authors get advice that is possibly more useful than what they would have received from an instructor. Most importantly, SWoRD allows the instructor to assign writing tasks of the most important kind (with feedback and revision) without having to do any grading at all, which means that writing practice can now take place in every class (from small sections of 10 students to large sections of 1000 students). Instructors create an account and setup a course in SWoRD. Students then create their own accounts on SWoRD and sign-up for the class. SWoRD is now licensed by Panther Learning under the name of Peerceptiv.

Teaching Writing and Argumentation with AI-Supported Diagramming and Peer Review. We are investigating the design of intelligent tutoring systems (ITSs) that are aimed at learning in unstructured domains. Such systems are not able to do as much automatically as ITSs working in traditionally narrow and well-structured domains, but rather they need to share responsibilities for scaffolding learning with a teacher and/or peers. In the work proposed, we leverage our expertise in automated natural language understanding, intelligent tutoring systems, machine learning, argumentation (especially in law), complex problem solving, and engineering education, to integrate intelligent tutoring, data mining, machine learning, and language processing to design a socio-technical system (people and machines working together) that helps undergraduates and law students write better argumentative essays. The work of helping learners derive an argument is shared by the computer and peers, as is the work of helping peer reviewers review the writing of others and the work of learners to turn their argument diagrams into well-written documents. Research questions address the roles computers might take on in promoting writing and the technology that enables that, how to distribute scaffolding between an intelligent machine and human agents, how to promote better writing (especially the relationship between diagramming and writing), and how to promote learning through peer review of the writing of others.

Improving Learning from Peer Review with NLP and ITS Technique. One result of prior research with SWoRD is an enormous database of written materials that are ripe for analysis and exploitation in support of research on natural language processing (NLP), intelligent tutoring systems (ITS), cognitive science, educational data mining, and improving learning from peer review. In this project we will both analyze existing SWoRD-generated data, and develop an improved version of SWoRD for use in further experimentation. In particular, we will explore using SWoRD to teach substantive skills in domains involving ill-defined problems, and will explore techniques for automatically identifying key concepts and flagging issue understanding. Second, given a SWoRD toolkit of what can be accomplished robustly with peer interactions, we will explore the use of natural language processing to automatically support and improve those interactions. Finally, we will develop a new version of the SWoRD program that incorporates improved features and control facilities, and that incorporates Artificial Intelligence techniques to improve learning in a variety of ways.

An Intelligent Ecosystem for Science Writing Instruction. The ability to express scientific ideas in both written and oral form is an important 21st century skill. Teachers, employers, and college faculty lament the inability of many high school graduates to write clearly. This deficit in writing is due in part because teachers do not have the time to provide appropriate, timely feedback to students on their writing. This project would help teachers help students achieve these skills through automating an effective feedback process, in ways that are customized to particular disciplines and local classroom needs, particularly in high needs districts. The project will contribute to knowledge about how students learn to write and how computer assisted systems can support this learning. This project will develop and test three tools: 1) Teaching resources organized as developmental trajectories for teachers to use (e.g. from more simple to more complex; with diagnostics and strategies for addressing particular challenges); 2) A teacher dashboard that uses Artificial Intelligence tools to provide timely formative assessment to teachers by highlighting problem areas in their students' writing and peer reviews; and 3) An online teacher resource exchange to rapidly grow the set of appropriate assignments that can be used with this approach, critically filtered by student performance metrics. The project builds on a current system called SWoRD, which supports student peer reviewing in many disciplines within and beyond science. Working with six lead teachers and larger set of pilot teachers, the project will develop a trajectory of effective writing assignments in Biology, Chemistry, and Physics. In year three, there will be a summative evaluation with 90 teachers.