Holistic Evaluation of Teaching at UCLA

Introduction to the Project

UCLA’s Center for the Advancement of Teaching has initiated an exciting new project to improve the way UCLA evaluates teaching. Across our campus, there is widespread recognition that excellent teaching is too multifaceted and nuanced to be documented using a single source of evidence like student surveys. Thus, we have developed a framework that we refer to as Holistic Evaluation of Teaching: a way to think about and document teaching that includes self-reflection, student input, and peer review. Importantly, this new approach emphasizes faculty professional development and efforts to improve teaching. It is adaptable to the particular teaching challenges and cultures of each unit across campus, and it has been developed with broad input from faculty, the Senate, and academic leadership.

Excellent Teaching and Student Success

The motivation for this project is simple: to improve student learning and students’ academic experience by helping faculty implement excellent teaching practices. We believe this can be accomplished in part by establishing a culture in which faculty continually examine their pedagogy and strive to innovate and improve, with the goal of enabling all students — across the diversity of their backgrounds, levels of preparation, and access points (freshman vs. transfer entry) — to learn.

Better Documentation of Teaching Excellence

The academic personnel reward structure must align with the goal of teaching excellence, so that faculty see that their efforts are both appreciated and rewarded. Here, CAP currently faces a challenge with lack of robust evidence for the quality of teaching. CAP members have noted that peer evaluations and letters from students are almost uniformly positive and therefore not helpful. Student evaluations are the only metric that displays a range of opinions. Unfortunately, while there is value in getting input from students, research shows that students are not the best judge of their own learning and that student evaluations are subject to bias based on gender, ethnicity, and other factors.

Working in tandem with the HET initiative, the Center for Educational Assessment is revising the end-of-quarter surveys that students complete for each class. Students using the new surveys are asked about their experience of evidence-based instructional practices, and they are encouraged to share feedback that instructors can use to improve the course. These revisions aim, among other things, to make these surveys a more robust form of evidence in the evaluation of teaching.

Because evaluation of teaching is associated with the high-stakes decisions of merits and promotions, it is not surprising that individuals and departments tend to speak in glowing terms about the candidate’s teaching, without serious attention to solving problems that may exist. We believe it is preferable to take a developmental approach in which the candidate is rewarded for professional development and efforts to improve teaching—particularly efforts that research suggests can enhance student learning.

This is consistent with CAP’s Teaching page, which states that “The articulation of a candidate’s pedagogical approach, the description of new teaching techniques tried out and their results, the efforts to constructively and imaginatively address problematic issues cited by students, all testify to the candidate’s engagement in the educational mission.”

Insights, Support, and Guidance from the UCLA Community

Those developing Holistic Evaluation of Teaching initiatives at other universities have made it clear: the work must be grounded in the priorities and perspectives of the campus community. For this reason, an initial phase of our work (see timeline below) was to conduct interviews with about thirty faculty across campus, including several current and past chairs and deans. These individuals represent a range of disciplines, career stages, and racial/ethnic backgrounds. What we learned from these interviews is at the heart of the HET framework.

We are grateful to have incorporated insights from Interim Executive Vice Chancellor and Provost Michael Levine, now Vice Chancellor, Academic Personnel, who says of the project: “I believe this is a useful approach to the evaluation of teaching, especially in terms of providing information about pedagogical improvement.” Additionally, we are grateful for the insights of Interim Vice Provost, Academic Personnel Kathleen Komar and members of the Academic Senate’s Council on Academic Personnel and Committee on Teaching. Our work is also informed by insights from our Advisory Committee (faculty from across campus), our Steering Committee (leaders from other campus teaching-and-learning centers), the experience of other UC campuses, and our engagement with the Bay View Alliance and their TEval project. We are grateful for the generous financial support and institutional endorsement of the current and immediate past College Deans: Miguel García-Garibay, Darnell Hunt, Tracy Johnson, David Schaberg, and Abel Valenzuela Jr.

Timeline

←       Ongoing consultation with the Senate and academic leadership       →

Stage 1

F 2021-W 2022

Groundwork

Understand HET research base, HET efforts at other universities

Convene HET advisory, steering committees

Stage 2

W-Sp 2022

Needs Assessment, Initial Design

Interviews with 30 UCLA stakeholders

Input from advisory and steering committees

Design framework

Stage 3

Sp-Su 2022

Revise Framework, Preparation for Pilot

Revise framework based on feedback from campus leaders

Design departmental adaptation process, identify collaborating departments

Stage 4

F 2022-Sp 2024

Implementation Pilot

HET adaptation and adoption in select departments

Stage 5

F 2024-Sp 2026

Implementation Scale-Up

Wider HET adaptation and adoption across campus

Project Team

Glory Tobiason, Co-Lead

Dr. Glory Tobiason is Clinical Faculty in Education and a Research Scientist at the National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Working under the aegis of CAT, Dr. Tobiason staffs and co-leads the Holistic Evaluation of Teaching project. She has a background in formative assessment, standards-based instructional design, and teacher quality. She also developed UCLA’s Peer-Assisted Reflections on Student Learning (PAROSL) program.

Adrienne Lavine, Co-Lead

Adrienne Lavine served as the Associate Vice Provost, Center for the Advancement of Teaching from July 1, 2017 to September 30, 2022, and she is a faculty member in Mechanical and Aerospace Engineering. She has served as chair of the Academic Senate and as chair of her department. She was the Director of Education and Outreach for the NSF-sponsored Center for Scalable and Integrated Nano-Manufacturing, directed the Clare Boothe Luce Undergraduate Scholars Program, and chaired the SEAS ABET Accreditation Committee in 2002-2006. She was awarded the Henry and Susan Samueli Teaching Award in her department and is the co-author of an internationally respected heat transfer textbook.

Michelle Chen, Program Manager

Michelle Chen is a project manager in the Center for the Advancement of Teaching (CAT). In addition to HET, projects and programs she manages include the Instructional Improvement Grants, Peer Assisted Reflections on Student Learning, Distinguished Teaching Awards, and Night to Honor Teaching event. She also supports various faculty committees and provides administrative support for CAT’s departmental operations.  

Marc Levis-Fitzgerald, Liaison to HET from Center for Educational Assessment

Dr. Marc Levis-Fitzgerald is the Director of the Center for Educational Assessment. His research interests include curriculum reform and evaluation, student and faculty development, and institutional transformation. He is particularly interested in documenting the experience of undergraduate students, and he directs the development and implementation of the UCLA Senior Survey. He heads several undergraduate science initiative assessment projects funded by NSF and HHMI. In addition, he leads program review efforts related to the Transfer Alliance Program, a partnership between UCLA and community colleges in California.

Kumiko Haas, Liaison to HET from Instructional Improvement Programs

Dr. Kumiko Haas works with individual faculty and oversees the development and implementation of programs, services and activities across all UCLA schools and colleges in support of innovative instructional improvement. She develops and teaches intensive training programs, workshops, and seminars on issues of pedagogy, course design, and instructional methods for new and experienced instructors. Her current responsibilities include development and implementation of faculty engagement programs and teaching assistant training program. These include the New Faculty Teaching Engagement (NFTE), the Collegium of University Teaching Fellows Program (CUTF), TA Training Program (TATP), and Test of Oral Proficiency (TOP) and oversight of the Instructional Improvement Grant Programs (IIP Grants). She also oversees the UCLA Community Based Learning Programs.

Because of the importance of this project, CAT is devoting significant resources of its staff. To date, the following individuals have contributed to the project: Anita Han, Molly Jacobs, Amy Liu, Erin Sparck, Lucia Tabarez, and Brit Toven-Lindsey.

Advisory Committee Members

Paul Barber, Ecology and Evolutionary Biology

Elizabeth Bjork, Psychology

Clarence Braddock III, David Geffen School of Medicine

Jessica Collett, Sociology

Al Courey, Chemistry and Biochemistry

David MacFadyen, Comparative Literature

Victoria Marks, World Arts and Cultures / Dance

Jose-Felipe Martinez, Education

Muriel McClendon, History

Christopher Mott, English

Josh Samani, Physics and Astronomy

Steering Committee Members

Lisa Felipe, Excellence in Pedagogy and Innovative Classrooms

Rachel Kennison, Center for Education Innovation and Learning in the Sciences

Tami Kremer-Sadlik, Social Science IDP

Shanna Shaked, Center for Education Innovation and Learning in the Sciences

Sirui Wang, UCLA Online Teaching and Learning

Juliet Williams, Social Science IDP

Cohort 1

We are working with our first cohort of departments in AY 22-23. During this year, a faculty working group from each department meets regularly with a HET facilitator to tailor the HET approach, making sure it reflects the priorities and disciplinary context of their department.

Life Science Core Education Department

Beth Lazazzera (department chair): “This initiative to adopt a more holistic method to evaluate teaching by faculty is desperately needed and will help create a system that can reward faculty contributions to teaching. As the LS Core Education Department has a large group of outstanding educators, we felt we could help this process become more widespread by being involved early in the HET initiative.”

Working Group:

Debra Pires

Deb Pires

Jukka Keranen

Jukka Keranen

Rana Khankan

Rana Khankan

Jay Phelan

Jay Phelan

Sociology

Abigail Saguy (department chair): “As a department that prizes teaching excellence, we welcome the opportunity to customize new ways of assessing effective teaching that go beyond student reviews and encourage self-reflection.”

Working Group:

Jessica Collett

Jessica Collett

Chris Herring

Chris Herring

Megan Sweeney

Megan Sweeney

Stefan Timmermans

Stefan Timmermans

School of Nursing

Lauren Clark (Associate Dean of Academic Programs): “In nursing, we know that one kind of evaluation doesn’t tell the full story about teaching effectiveness. We have big classes, small classes, clinical, and lecture classes. Developing holistic evaluation methods lets us appraise our teaching performance more fully and fairly.”

Working Group:

Theresa Brown

Theresa Brown

MarySue Heilemann

MarySue Heilemann

Charlene Niemi

Charlene Niemi

Dorothy Wiley

Dorothy Wiley

Linguistics

Megha Sundara (department chair): TBD

Working Group:

Dylan Bumford

Dylan Bumford

Jesse Harris

Jesse Harris

Stefan Keine

Stefan Keine

Megha Sundara

Megha Sundara

Kie Zuraw

Kie Zuraw

Circle with three sections with the names of the three elements of HET in each section: "Dimensions of Excellent Teaching," "Sources of Evidence," and "Evaluation Lenses."

HET Framework

HET is organized into a 3-part framework, which includes dimensions of excellent teaching, sources of evidence, and evaluation lenses. Based on research, examples of other universities, and feedback from UCLA stakeholders, we have designed “off-the-shelf” versions of each of these three elements. These are versions that reflect discipline-agnostic principles about how people learn and common features of student-centered, inclusive pedagogy. (These are described in detail in this document.)

But of course, disciplines and departments have their own pedagogical practices, priorities, cultures, etc. To reflect those, departments that implement HET may choose to adjust the off-the-shelf versions of the three elements.

Conducting A Review Using the HET Framework

Using provided HET guidance and examples, instructors write their teaching statement and assemble their teaching portfolio. Evaluators review these materials, interpret them, and make a judgment about the instructor’s teaching, using the provided HET rubric(s). They describe their judgment in a departmental letter that becomes part of the case. Evaluators may also provide private, no-stakes feedback to the instructor to help them reflect on and improve their teaching.

Dimensions of Excellent Teaching in HET

In the HET framework, excellent teaching is understood to comprise four principles. Excellent teaching (i) engages students, (ii) is equitable, (iii) is learning centered and responsive, and (iv) strives to improve.

Engaging Students

stick figure next to open door

DEFINITION. Teachers engage students by creating courses and lessons that are inviting, coherent, and clearly organized. They interact with students and support their learning beyond the delivery of lectures. Courses and lessons include appropriate content, reflect scholarship about how people learn, and are designed to engage students as active participants.

This includes the following components:

  • Being appropriately available to students and communicating respectfully and promptly;
  • Communicating (e.g., in syllabus, course website) student responsibilities, grading conventions, and course logistics (e.g., where to find materials, how to communicate with the instructor) in an accessible, clear, and explicit way; and
  • Selecting appropriate course content and designing activities that reflect scholarship about how people learn: actively; with frequent opportunities to practice and apply; and with appropriate challenge, scaffolding, and support.

Equitable

DEFINITION. Equitable teaching means designing instruction, interacting with students, and maintaining learning environments in ways that enable all students to learn. Instructors convey that all students are capable scholars and valued members of an academic community and acknowledge differences as contributions to the learning community.

This includes the following components:

  • Designing learning activities to be accessible for all students, including creating and maintaining structures that allow all students to contribute;
  • Incorporating student voice and choice into learning activities; 
  • Modeling a commitment to inclusive teaching in decisions, comments, and behaviors; and
  • Ensuring that course content reflects diversity and surfaces issues of equity and social justice that are relevant to the discipline.

Learning Centered and Responsive

stick figure with two gears above head

DEFINITION. Learning-centered and responsive teaching means instructors design lessons and courses starting with what they intend students to learn, rather than which content they will present. Assessment makes gaps between students’ current and intended learning visible; after formative assessment, feedback and subsequent instruction narrow those gaps. Instructors regularly refine teaching materials based on what they observe about student learning and feedback from students.

This includes the following components:

  • Making the intended learning for courses and lessons explicit and aligning materials, activities, and assessments to that intended learning;
  • Conducting frequent, low-stakes, formative assessment to make student thinking visible and adjusting subsequent instruction and giving students feedback in response;
  • Designing synchronous lessons with enough flexibility and student participation that decisions and adaptations can be made in real time, in response to how learning is unfolding in a particular lesson, for particular students; and
  • Asking students and observing how syllabi, assignments, activities, assessments, readings, etc. support learning, and then refining these materials.

Striving to Improve

stick figure holding briefcase walking up stairs

DEFINITION. Teachers who strive to improve recognize that teaching expertise is not static, and it is always possible to learn more about teaching. Over time, there are shifts in student demographics, background learning, and interests; the social context in which we teach; what we know about how people learn; and disciplinary content (in some fields). Excellent teachers keep an eye out for ways their pedagogy may also need to shift, and they support colleagues to do the same.

This includes the following components:

  • Reflecting on the success of current teaching strategies and adjusting them to better serve students;
  • Participating in ongoing professional development;
  • Observing or being observed by a colleague in order to give or get feedback about teaching; and
  • Contributing to conversations about and efforts to improve teaching and learning in the department, on campus, and beyond.

Sources of Evidence in HET

graphic of a document representing teaching statement and a folder with papers labeled "Teaching Record," "SETs," "Additional Evidence" representing teaching portfolio

Teaching Statement

This brief document serves two purposes. First, it serves as a roadmap for the contents of the teaching portfolio. Second, it is an opportunity for the instructor to self-evaluate and reflect on their teaching. As with current practice in many departments, this can be part of the candidate’s overall self-evaluation letter; the HET framework provides guidance and structure to assist faculty with creating a more substantive teaching statement.

Teaching Portfolio

The teaching portfolio, includes the following, which are consistent with current requirements at UCLA:

  • A Teaching Record (a form that summarizes courses developed and taught); 
  • SETs; and
  • One or more of the following additional sources of evidence:
    • Evidence from students beyond SETs (e.g., student work, feedback about a course that the instructor has acted upon, etc.),
    • Evidence from peers (e.g., description of classroom observation or review of course materials; description of collaborative curricular planning of related courses),
    • Teaching artifacts (e.g., syllabi, lesson materials, major assignments or assessments, websites, etc.), and
    • Record of Efforts to Improve Teaching (a form that summarizes improvement efforts, including but not limited to participation in professional development; participation in formative peer observation; service on a curriculum or assessment committee; teaching-related publications, presentations, or grant applications; etc.).

Evaluation Lenses in HET

Proficiency LensInnovation Lens
graphic of map with drop pin“Where are you on a map of excellent teaching?”graphic of hilly road with trees“How have you moved further along the road to excellent teaching?”
PurposeEvaluate the state of an instructor’s teaching (i.e., how proficient they are at teaching)Evaluate the instructor’s efforts to improve their teaching–that is, the extent to which they have innovated in their practice
What instructors doProvide evidence of the static quality of their teachingProvide evidence of growth over time in their teaching

Departments are provided with guidance for combining two evaluation “lenses”–two ways to view and judge an instructor’s teaching. Looking through the proficiency lens, evaluators consider the state of an instructor’s teaching (i.e., how proficient they are at teaching). The innovation lens helps evaluators render a judgment about the instructor’s efforts to improve their teaching–that is, the extent to which they have innovated in their practice. By focusing on changes in teaching practice and efforts to improve, the innovation lens prioritizes growth over time, rather than static quality of teaching.

Departments tailor these lenses and decide how best to combine them. HET-developed evaluation rubrics​, which departments may choose to customize, help evaluators render judgments about an instructor’s teaching, which they summarize in a narrative.

The teaching statement and evidence in the teaching portfolio are organized to illustrate proficiency in the first three dimensions of excellent teaching and innovation as part of the fourth dimension.