Excerpted from the SEP by the Children’s Institute, a subgrantee of the Edna McConnell Clark Foundation.
Program managers at Children’s Institute Inc. (CII) have expressed a strong interest in including an implementation study in our evaluation to complement the fidelity assessments. We also believe that an implementation study will be important in order to provide an assessment of the overall quality of CII’s program and to better understand how the EBTs fit into the overall structure of service delivery.
These paragraphs outline specific plans for measuring program implementation fidelity.
In addition to the fidelity assessment questions that will be addressed (see [previous section]), our plans include at least two rounds of site visits to answer the implementation-related research questions outlined in [another section]. During these visits, we will conduct semi-structured interviews with program managers and staff, observe program activities, and meet with other local service providers and key stakeholders. We will also augment the findings from the site visits with what was learned in conducting the fidelity assessments since implementation is a key focus of those assessments. In addition, we will work with CII to conduct analysis of CII’s MIS to better understand how youth flow through program services and to determine what services in addition to the EBTs are being provided to youth.
Our current plan is to visit four campuses during the first visit (Otis Booth, Watts, Torrance, and Central LA) since these campuses offer the three EBT’s that will be assessed. During the second round, we will revisit these same four campuses, and add the Long Beach campus, which only offers TF-CBT.
Interviews will be conducted in group settings as much as possible (e.g. a group of clinicians that provide FFT will be interviewed together) to maximize evaluation staff time on site and minimize disruption to program operations.
Implementation Fidelity
These sentences detail the data collection methods that will be used in the implementation evaluation.
Consistent with the primary implementation questions outlined in [a previous section], measures of implementation fidelity of the EBTs will be assessed utilizing semi-structured interviews. Interviews with senior staff, including the president and program directors, will inform how the organization’s leadership developed and implemented the systems CII uses to promote fidelity. Interviews with program managers will shed light on the overall client referral, enrollment, and assessment process (primary implementation questions 1 and 2) and the staff recruitment and training process (primary implementation question 5). If the evaluation design used a comparison group, a similar discussion of data collection to understand any diffusion of the program could be included here.If the evaluation design used a comparison group, a similar discussion of data collection to understand any diffusion of the program could be included here.Supervisors and staff (e.g. clinicians that provide EBTs and their supervisors) will be interviewed to better understand how client services are documented and tracked (primary implementation question 3), how EBTs fit within the overall service delivery structure (primary implementation question 4), how CII addresses decisions around which EBT to utilize or not utilize, and how difficulties during the course of treatment are resolved (primary implementation questions 5, 6 and 7). Since semi-structured interviews will be conducted across all of CII’s sites, the team will be able to assess whether there is variation across the five sites (primary implementation question 8).
In addition, participant demographics and participation data will be collected from the CII MIS to assess the level at which the program was successful in providing services to the intended target population.
Organizational Implementation Assessment
The research team and CII are interested in better understanding CII’s overall program model, as well as how the EBTs fit within the organization, especially as it relates to CII’s continuum of care and its trauma-lens. Moreover, CII is interested in the value added of their other services, such as youth development and family support services. In order to address these secondary implementation questions, the semi-structured interviews described above will also address the following. Interviews with program managers and other stakeholders in the community (e.g. staff from the Los Angeles Department of Social Services, a key referring agency) will help the evaluators better understand issues related to the local context and infrastructure (secondary implementation question 9) and what other services are available in the community (secondary implementation question 10). Interviews with program managers, supervisors, and staff will also help the evaluators better understand CII’s overall organization structure, service delivery, and how the trauma-lens factors into services (secondary implementation question 11) as well as the overall strengths and challenges of the CII model (secondary implementation question 12). The protocols for the semi-structure interviews will be developed approximately six to nine months prior to the first round of site visits.
