Building Capacities That Can Outlive A Pay For Success Project: ICS’s Coaching Model Of Technical Assistance
By Mary C. Garvey, Associate Director of Pay for Success Financing
“If you’ve done one Pay for Success project, you’ve done one Pay for Success project.” This saying in the Pay for Success (PFS) movement speaks to how different each PFS project, and feasibility study, can be. Doing one project has not necessarily meant the next one will be easier.
Part of the issue, ICS has hypothesized, may be that the PFS field has perhaps focused on project development and execution more than building capacities along the way. The emphasis has often been on getting the project to launch, not on training government staff on the underlying activities themselves.
Recognizing this challenge and opportunity, ICS has taken a unique approach to many of its feasibility study engagements. Instead of serving as a traditional consultant that does the work itself though in consultation with the government, we often partner with local or state government staff to train and guide them to conduct the feasibility study themselves.
This coaching model stands apart from other PFS TA approaches. While it is true that others partner with government staff and some jurisdictions receive embedded Fellows, our coaching model is the only of which ICS is aware that builds capacities in government staff such that they are the primary ones conducting the analyses – and that provides that coaching remotely.
We write this blog post to share information about this model so that the field can learn from our experiences and consider how this approach might prove useful for other efforts to advance our shared, ultimate aim – improving outcomes for years to come.
What’s Involved
The nature of the ICS coaching model requires there to be a minimum staff time commitment from five vital staff roles on the ground within a jurisdiction – the Project Champion, the Project Manager, the Data Manager, the Budget Official, and an Attorney. The Project Champion is a senior executive branch official who prioritizes and oversee the jurisdiction’s PFS initiative, dedicating around 5% of their time to the feasibility study. Ideally, the Project Manager has an early childhood background and/or experience in data and program analysis to gather and analyze data, and conducting extensive research. As the person responsible for the bulk of the feasibility study work that the jurisdiction completes, the Project Manager should be a full-time employee who dedicates 60-75% of his or her time to the study and is available in a similar capacity to carry the project forward post-feasibility. The Data Manager role can be filled by one or more people with access to relevant data, such as data on child welfare, Medicaid data, and school district records, with an equivalent of 25% of one person’s time dedicate to conducting analyses. An attorney is required on an as-needed basis to address any legal concerns (e.g., data-sharing agreements) to ensure that the jurisdiction can complete the work efficiently, given the legal issues that inevitably arise even during the feasibility stage. Finally, the Budget Official dedicates a nominal amount of time to review and advise on items such as cost savings analyses.
Our coaching model of TA relies heavily on constant communication as well as ongoing training and strategy sessions all done via email, weekly check-in calls, monthly interactive educational webinars, and site visits every few months. In addition, we have created instructional memo templates for each of the key steps of the feasibility study process. We use weekly check-in calls as a time to discuss the progress on the current feasibility study step, review the current memo draft, strategize about stakeholder engagement, and train on and plan for the upcoming feasibility study step. This usually involves walking staff members through the associated memo template. The memo-drafting process is quite intensive, often requiring three or more iterations. Because there is almost always more work to be done, email communications serve an important role in keeping the process on-track in between those check-ins. Our monthly webinars are often themed and provide an opportunity for each jurisdiction to present on progress and solicit feedback for the current feasibility study step. During each webinar, ICS staff also introduce and begin training on the next step.
While we tend to have the same goals for site visits to consulting and coaching jurisdictions, site visits to our coaching jurisdictions generally entail more intensive collaborative and instructional working sessions. We plan for three site visits per jurisdiction. Our meetings during the first site visit generally involve planning sessions with jurisdiction staff, data-gathering discussions with local service providers sometimes (on-site at their place of service whenever possible), and PFS presentations to stakeholders followed by question and answer sessions. The second site visit often involves strategy sessions and follow-up with service providers to aid in expansion-planning. The final site visit occurs at the end of the feasibility study process. We return to the jurisdictions to support them as they present feasibility study findings to key stakeholders and strategize around next steps post-feasibility.
Benefits and Challenges
We have completed two cohorts working with a total of eight SIF-supported jurisdictions for PFS feasibility studies; most them have opted for the coaching model. To date, the coaching model has yielded significant benefits. Our coaching creates a deep understanding of the feasibility study process and the requirements of outcome-based financing. TA recipients have noted that the process empowered their local service providers and led them to take concrete steps towards improving their implementation. Jurisdiction staff develop and fine-tune key skillsets; imparting such knowledge instead of doing work for them is an important step toward program sustainability and the ability to positively impact early childhood outcomes in an increasingly outcome-based world long after our feasibility study has ended. For example, by the end of the process, jurisdiction staff will have conducted thorough literature reviews of early childhood research, identified outcomes well-suited to PFS, interpreted cost-benefit analyses and meta-analyses, and estimated the impact on outcomes that a program expansion may have. Given that, in our experience, some jurisdiction staff may never have analyzed a research study before, the completion of these steps and analyses amount to an impressive feat – and one that will continue to pay dividends beyond PFS.
As is to be expected, the coaching model has not been without its challenges as well. Getting the necessary staff on-board by the start of TA has proven difficult at times. The sometimes-transient nature of staff roles at partnering agencies has previously left us without a Project Manager for a period of time in more than one jurisdiction. Even when staff are fully in place, coaching staff through the feasibility study process is inherently time-consuming. It can be difficult to stay on track with the scheduled timeline. The challenges of the past two years coaching around PFS feasibility studies have led to lessons learned and the development of smart practices for moving forward. While we did become better at determining how long each feasibility study step would take after the first cohort, we also learned the importance of building in buffer time and suggest that any TA provider considering a coaching model do the same. The intensity of the coaching model cannot be overstated!
To help identify and manage the challenges that arise, regular evaluation of our TA has been critical. It is primarily through our monthly TA evaluations that we have learned exactly which aspects of the memo templates required further explanation and which webinar topics were the most and least useful as well as the most and least interesting. Our decision halfway through the first cohort to do less presenting and leave more time (up to 30 minutes) for unscripted open discussion amongst the jurisdictions during our webinars, for example, was a direct result of feedback in evaluations. It is important to be open to constructive feedback. Another lesson learned through trial and error is that the coaching model is not always the right fit for jurisdictions that choose it. Because staffing circumstances can quickly change and workload expectations can be misjudged, there have been times when a jurisdiction switched from our coaching model to a more traditional consulting model during the feasibility study process. Perhaps our most important lesson is that it is necessary for us to remain flexible and teachable as coaches with the understanding that our coaching can ultimately improve outcomes for children.
Impact
Reflecting on both the challenges and benefits embedded in our coaching model, we are heartened to see its impact on the ground. Kim Goll, Director of the Orange County Commission on Children and Families, shared her appreciation for the coaching experience:
“When we started this process, our end goal was to determine whether a PFS financing model was feasible for the Neighborhood Resource Network Program. What we learned through the rigorous exercise of doing a deep dive into the research and evaluation data of our evidence based interventions allowed us to better understand our program models and the effectiveness those programs.
The discipline of researching and writing the ICS memos allowed us to scrutinize the program outcomes and develop a deeper understanding about the financial impact of each outcome and who might be most interested in investing in the particular outcomes. ICS staff carefully crafted the feasibility memo outlines, questions, and technical assistance so that each one builds on the previous one, resulting in a triangulation effect of whether a PFS financing contract is feasible, yet the overall benefits of this exercise far exceed that.”
In the final evaluation survey for our most recent cohort, 100% of respondents agreed that, as a result of our TA, their organization or agency has the ability to have a greater impact on children and families and better understands the outcomes their programs are producing. We are tremendously proud of this indicator. Whether we are coaching or consulting, this is the kind of result ICS strives to achieve on the road to improving outcomes for children. We look forward to helping even more communities develop their own capacities to undertake the analyses of PFS feasibility to benefit PFS and non-PFS projects to come – all in service to the youngest in our midst.