Remember that major changes affect all involved, and when the impact on faculty, staff, and students is monitored, implementation can be successful. To determine the impact of systemic changes, continuous improvement to scale mathematics pathways should include opportunities to gather, analyze, and respond to the voices of all stakeholders.  

The Stages of Concern process (referred to in Essential Action 3 and Essential Action 6) is one way to assess and respond to uncertainties, attitudes, and perceptions of implementing and scaling change. Gathering faculty and staff voices might be done through interviews, open-ended concerns statements, and/or the Stages of Concern questionnaire (American Institutes for Research, 2010). Additionally, leaders should consider the following questions: 

• Do faculty and/or staff have leadership and ownership roles in the change process? If so, what are the stakeholders’ levels of involvement in those roles?
• What are the faculty’s and staff’s levels of preparation and readiness to redesign mathematics courses and/or its corequisite supports?
• How are full-time and part-time faculty getting necessary professional development to support change?

Monitoring the effects of implementation on students might include interviews, observation of academic advising and classroom interactions, a mathematics pathways informational awareness survey, or focus groups to assess students’ understanding of mathematics pathways and their impact towards students’ academic and career goals. Further, a campus-wide attitudinal survey can gauge awareness, impact, and effect of implementation on students.

The leadership team should plan for an ongoing feedback loop in the continuous improvement cycle. In the paradigm shift from an instructor-focused to student-focused environment, self-reflection supports all involved stakeholders to be a part of the change process and, ultimately, informs and leads to improved student success.

Student data collected, analyzed, and interrogated in Essential Action 4 should be part of interim evaluation processes to gauge progress toward the institution’s goals, and, if needed, make interim adjustments to support reaching the goals. Using the evaluation plan (developed in Essential Action 6), the leadership team should identify leading data indicator questions about student persistence, progression, and completion.

Rather than merely relying on lagging indicators of success, such as completion and transfer rates, the Community College Research Center (CCRC) recommends that institutions use a set of early momentum key performance indicators (KPIs), also called leading metrics of student performance, to assess the early impacts of whole institution reform (Jenkins et al., 2018). These leading metrics are timely and actionable to help predict an outcome, allowing institutions to make necessary interim adjustments to their scaling efforts.

The early momentum KPIs are based on metrics of first-year student performance that are associated with higher completion rates over a longer period of time (Jenkins & Bailey, 2017). Recommended early momentum KPIs to monitor fidelity of implementation and identify early signs of progress towards goal achievement include:

• Gains in credit momentum,
• Gains in gateway course completion and persistence, and
• Retention.

These recommended early momentum KPIs should be disaggregated by race/ethnicity, gender, age, socioeconomic status (e.g., Pell eligibility), and first-generation status to identify potential issues or barriers.

Additional considerations for data collection and analyses for interim adjustments include assessing the effectiveness and expansion of multiple measures for student placement, understanding how well students do in subsequent mathematics courses in their pathways (e.g., Pathway to Calculus), reviewing effectiveness of academic advising, and evaluating course redesign effectiveness for gateway math courses and corequisite supports.

With a clear picture of where a given institution is in its implementation of mathematics pathways, the leadership team can make decisions about adjustments if the implementation is not going as planned.

Understanding data and how they move us to action are important to the continuous improvement process. Consider which stakeholder groups are involved in the data collection and analysis, and think about their roles and responsibilities to their higher education communities to share the data.

Both presenting and discussing data can be a challenge, particularly for stakeholders who have not been involved in the implementation or data collection process. To help them understand the data:

• Use asset-framing (The Skillman Foundation, 2018) to communicate data as a tool to advance equity.
• Limit the amount of data to the research question(s) being answered.
• Present data in small chunks that an audience can understand in a short timeframe.
• Use a headline to inform the audience of what they should focus on in any data presentation (Phillips & Horowitz, 2017).
• Guide the data discussion with meaningful questions that engage the audience—for example, “Is the information accurate?” “What jumps out and why?” and “Does the information challenge current assumptions about this population?”
• Ask a representative of the Institutional Research (IR) office staff to address questions about the data and to advise on future data collection.

Ultimately, creating a culture of inquiry and evidence relies on data being shared broadly, consistently, and transparently with the college community.

An effective way to share that information is through data stories, which help build engagement and empathy toward institutions’ efforts to scale mathematics pathways. When facilitating data-driven discussions, these stories present the facts with additional context. See p. 69 of Achieving the Dream’s Data Discovery Guide for tips on writing a data story.

Clear and timely communication and engagement occur across all stages of implementation and is essential to successful mathematics pathways. Transparency is critical to building a sense of ownership among stakeholders. It is not enough to simply disseminate results from the evaluation plan. Faculty and staff should understand the process and how the evaluation results are used to refine goals and improve implementation. It is also important to find ways in which people can give meaningful input.

Revisit Essential Action 3 to create an updated communication and engagement plan that builds broader understanding of the impact of mathematics pathways. This updated plan should reflect what was learned from evaluation results to ensure bidirectional communication across stakeholder groups and to inform decision-making in the change process.

While working towards systemic change, it is easy to become too focused on the work and overlook celebrating milestones and other successes. Public celebrations highlight the commitment to mathematics pathways and the importance that the move toward this normative practice plays in the success of students. The leadership team can brainstorm on what should be celebrated and ways to celebrate throughout the implementation process.

Celebrating success and honoring those who contributed to the success are an important part of sustaining individual commitment. They also cycle back to Essential Action 1 by communicating the institutional commitment to fully scale mathematics pathways. Refer to steps in Essential Action 1 that connect the work to the overall mission and strategic plan of the institution.

“If you don’t celebrate progress as you go, then you lose people’s desire to make continuous improvement. You have to say, ‘We have done so well. Look at where we are. Look at the number of students that have gotten through, that would never have gotten through before. The number of students that now have an opportunity to graduate—they never would have been given that opportunity before, because they never would have finished their math’” (Charles A. Dana Center, 2020).

See Continuous Improvement in Mathematics Departments: Some Thoughts from the Fieldview full resourceDownloadFile, which iterates the importance of celebrating success and identifies various approaches.

Summative evaluation is conducted near, or at, the end of an implementation cycle to determine whether implementation efforts achieved the goals and outcomes established in Essential Action 5. It further informs institutional stakeholders of overall implementation learnings that spur changes in structures, policies, processes, and/or practices that are barriers to student success.

In addition to the data considerations noted in Step 3 above, program outcomes and program effectiveness are vital to understand the impact of implementation efforts. This “big picture” summative evaluation examines both leading (e.g., credit momentum) and lagging (e.g., graduation rates, transfer rates) metrics of student performance within the perspective of the student journey to-and-through mathematics pathways (see “pathways perspective” in Essential Action 8). Effective use of measurable leading metrics of student performance over a relatively short time frame should strongly predict longer-term successes revealed in lagging metrics (Belfield, Jenkins, & Fink, 2019).

The leadership team should review holistically its implementation plan and evaluation plan developed in Essential Action 6. This summative evaluation identifies what was actually done when implementing math pathways, and why goals and outcomes were or were not met. Typical questions include:

• To what extent were the goals and outcomes of the initiative met?
• How successful were the implementation strategies and activities in meeting the goals and outcomes? If interim adjustments were made, how successful were the adjustments?
• Did the initiative impact the intended target audience?
• What strategies encouraged buy-in and commitment from diverse stakeholders?
• Can we adjust or replicate this success in our new continuous improvement plan? What will we continue to do or what will we change and why?

Key findings from summative evaluation help the leadership team understand its new institutional context and support stakeholders in the necessary transition into another cycle of continuous improvement.

As the leadership team uses data to learn from change, it will leverage learnings and the new institutional context to review and revise goals and outcomes. These revised goals and outcomes are the focal point of the institution’s transition to a subsequent cycle of the continuous improvement process to scale mathematics pathways and advance equitable access, learning, and outcomes. In this transition, it is imperative to recognize the ongoing needs of faculty, staff, and students—which change as well.

“When continuous improvement becomes part of our culture, it stops feeling so burdensome. Well-designed structures and processes can make continuous improvement part of our everyday work” (Getz, 2018).