Evaluation for Learning Group

Team Members 

Jennifer Iriti, Research Scientist

Courtney Long, Instructional Developer

Ram Goli, Undergraduate Intern

Chi Fan Lai, Undergraduate Intern

Research

Summary

The Evaluation for Learning group (EFL) is an evaluation, program planning, and social science research project specializing in K-12 education and based at the University of Pittsburgh's Learning Research and Development Center. Directed by Dr. Jennifer Iriti, the EFL works with local and national institutions on the design, documentation, evaluation, and strategic assessment of a wide variety of educational programs and reform efforts. An overarching theme in the EFL work is using evaluative inquiry to support improved decision-making by educational leaders, practitioners, and policy shapers. The resulting products are intended to be useful to specific sponsoring decision makers and organizations, as well as to generate knowledge for the broader research and evaluation communities.

Current & Recent Projects

Winchester Thurston City as Our Campus Study
Winchester Thurston is a PK-12 independent school located in Shadyside. The school has pioneered the development of a community-based learning approach that it calls “City as Our Campus (CAOC).” Teachers design instructional experiences that leverage resources in the city and, often, includes students leaving campus and/or engaging in collaborative work with organizations/experts outside Winchester Thurston. The Evaluation for Learning group is studying the CAOC model in order to document the critical components of the approach, understand the role it plays in teachers’ practice and student experiences, and identify what it takes to sustain and scale it. Ultimately, the research will be shared at a convening of other independent schools from across the country who are interested in initiating or growing community-based learning at their own schools. Key partners: Gary Neils and Adam Nye

Katz Experience-Based Learning Measurement System Development
Like many MBA programs across the country, the Katz School of Business incorporate experience-based learning into its programming. While business executives and university-based instructors often intuitively know that experience-based learning is powerful for preparing MBA students, there is little evidence to date to support this claim. This project is a collaborative development effort to build a valid but practical measurement infrastructure for the experience-based education component of the two-year MBA program so that faculty and staff are better able to support students’ growth toward targeted outcomes and to assess the value-added of the experience-based learning model that is central to the program’s pedagogical approach. Through this context-specific work, we seek to develop a set of standards and indicators for experience-based learning, a popular and valued (but unmeasured) approach to MBA programming nationally. Key partners: William Valenta, Joe Pieri, Jim Kimpel,Ron Magnuson, Susan Cohen, Kiersten Maryott, & Howard Sniderman

National Science Foundation
INLCUDES: Diversifying Undergraduate Participation in STEM Programs. Many universities offer precollege STEM programs that can increase disadvantaged student preparation for and success in undergraduate studies. These programs enable students to participate in research experiences that impart scientific skills and increase self-efficacy and engagement. Students gain a better understanding of how STEM fields can be used to solve real-world problems, become exposed to STEM career pathways, and participate in faculty mentoring that fosters self-identity as a scientist. However, participating minority students, even when demonstrating significant learning gains, may not be accepted for admission to the institutions that host these programs due to policies that filter out applicants based on SAT scores and high school GPAs instead of prioritizing more culturally relevant criteria. The objective of this pilot is to facilitate pathways for disadvantaged students to enter STEM university programs and eventually STEM careers. To accomplish this, the pilot has four specific aims: (1) create a community engagement framework to help recruit underserved students to precollege STEM programs, (2) develop a STEM Success Matrix that identifies student competencies acquired in precollege programs that prepare students for collegiate success in STEM, (3) credential precollege programs based on their ability to prepare students in alignment with the STEM Success Matrix, and (4) develop student badging based on precollege learning gains that will be used in holistic admissions review at research universities. A key task is to evaluate the efficacy and scalability of this process for admissions. Key partners: Alison Slinskey-Legg, Alaine Allen, Lori Delale-O'Connor, David Boone, Rebecca Gonda, Mackenzie Ball, LaTrenda Sherrill, Sunnana Chand, Kellie Kane, Kashif Henderson, and Lina Dostilio.

Pittsburgh Promise Monitoring
Annual indicators for Scholar retention, persistence, and degree attainment.
This ongoing, annual work involves the comprehensive analysis of all data available to understand patterns and trends in college going both of those who actually use the Pittsburgh Promise as well as the rest of the graduates from Pittsburgh Public Schools. This work is used to identify problems or practice or areas in need of additional intervention, including the performance of higher education institutions. Most recently, the work is focused on the development of data placemats to bring action-focused data on college going to school-based leaders. Key partners: Saleem Ghubril and Shelley Scherer

Pittsburgh Promise Impact Evaluation- enrollment, persistence, workforce
This work is focused on using rigorous quantitative methodologies to understand and quantify the impact of the Pittsburgh Promise on postsecondary enrollment, persistence, degree attainment, and workforce participation. The impact of the Promise on postsecondary enrollment and persistence into the second year has been completed and published in Education Finance Policy. That work was generously funded by the Lumina Foundation. The impact of the Promise on workforce participation is currently in the feasibility stage on a grant funded by the Strada Education Network and administered by the Upjohn Institute for Employment Research. Key partners: Lindsay Page, Michelle Miller-Adams, Brad Hershbein, Gary Ritter, & Celeste Carruthers

Better Math Teaching Network
The Nellie Mae Education Foundation has funded the American Institutes for Research’s (AIR) to launch and operate a Networked Improvement Community (NIC) in New England focused on student-centered learning in high school algebra as a potential solution to the problem of high rates of high school math students disengaged in mathematics learning. In the fall of 2015, the network began working with a pilot set of teachers and math leaders who learned about improvement science and began implementing their own Plan-Do-Study-Act cycles in January 2016. The Evaluation for Learning team is part of the developmental evaluation team (led by Dr. Jennifer Russell) that is providing support to the NIC and to the funder to understand NIC development broadly and shape the evolution of its work. Key partners: Jennifer Russell, Jennifer Sherer, Kirk Walters, Toni Smith, & Nellie Mae Education Foundation

Student-centered Assessment Network
The Nellie Mae Education Foundation has funded the American Institutes for Research’s (AIR) to launch and operate a Networked Improvement Community (NIC) in New England focused on student-centered assessment practices. The network engages teams of teachers at partner schools with improvement science methods and student-centered assessment practices. The Evaluation for Learning team is part of the developmental evaluation team (led by Dr. Jennifer Russell) that is providing support to the NIC and to the funder to understand NIC development broadly and shape the evolution of its work. Key partners: Jennifer Russell, Jennifer Sherer, Steve Plank & Nellie Mae Education Foundation

Spencer Foundation- Researcher-Practitioner Partnership Documentation
This strand of work is focused on documenting a researcher-practitioner partnership intended to leverage research insights from the development of a holistic measurement systems for mathematics teaching practice. As researchers and practitioners work together to understand and activate learnings from recent research work funded through IES and NSF grants, the documentation team will capture the considerations, enablers, barriers, and tensions in the negotiation and enactment of the joint work. Key partners: Rip Correnti, Mary Kay Stein, Jennifer Russell, Ally Shay Thomas, Tennessee Department of Education, & Memphis IZone.

Summer Dreamers Academy Evaluation Infrastructure Development
The Pittsburgh Public Schools seeks to build an internal framework and infrastructure to help the organization more effectively use data for shaping policy and practice decisions. This work will focus on the districts’ goals and programming around summer learning to pilot the tools and routines to gather, analyze, and aggregate appropriate data to inform key district questions. Key partners: Dara Ware Allen, Christine Cray, James Doyle, and Tylor Hart