*Excerpted from the SEP by the National Fund for Workforce Solutions, Jobs for the Future.*

If a between-groups design is planned, an Intent-to-Treat (ITT) analysis is described, which compares outcomes between those assigned program services and those not assigned.

*Descriptive Analyses of Evaluation Sample.* The matching approach, as described above, will enable us to produce matched comparison groups for unemployed and incumbent workers who participate in training. Prior to conducting any impact analyses, IMPAQ will develop descriptive analyses for treatment and matched comparison group members based on information available in the state UI administrative data. These analyses will provide an overview of the baseline characteristics of treatment and matched comparison group members, including:

• Socioeconomic characteristics from ES data (e.g., gender, race, ethnicity, education);

• Employment history from ES data (e.g., industry of prior employment, occupation of prior employment, tenure with prior employer, prior self-employment experience);

• Wage history from Wage Records (e.g., wages in each of 8 quarters prior to program entry, wage growth prior to program entry, number of quarters with positive wages prior to program entry);

• Prior participation in training programs from WIA data (e.g., participated in WIA training, WIA services received); and

• Receipt of UI benefits from UI data (e.g., prior receipt of benefits, number of IMPAQeks on UI, benefit amounts collected).

These descriptive analyses, which will be produced separately for unemployed and incumbent workers, are necessary to provide an overall characterization of NFWS/SIF participants and their matched comparison cases. Additionally, these analyses will provide evidence on whether the matching approach was effective in identifying comparison cases that are observationally similar to the treatment cases. To show that matching was done effectively, IMPAQ will perform the following tests:

The steps in the analysis, and the way in which the evaluation team will examine both the characteristics of the group and the outcomes are clearly related.

• *Treatment-comparison group comparisons in observed characteristics* – We will produce means comparisons of each available characteristic in the ES, WIA, and UI data. Using t-tests, IMPAQ will assess if there are any statistically significant differences in characteristics between treatment and matched comparison group members. If matching was done effectively, IMPAQ will not detect any statistically significant differences in characteristics between treatment and comparison cases.

• *Estimate likelihood of treatment group assignment* – We will estimate a linear regression model for each state, using the treatment and matched comparison group members. The dependent variable in this model is the likelihood of being a program participant and independent variables include all available characteristics in the ES, WIA, and UI data. If matching was done effectively, the estimated parameters will not be statistically significant.

Additionally, IMPAQ will produce descriptive statistics of treatment and matched comparison group outcomes in the 15-month period following program entry. To analyze labor market outcomes, IMPAQ will use the Wage Record data to construct the following outcomes in terms of quarters:

• *Likelihood of Employment* – Equals 1 if worker had positive wages in a given quarter, 0 otherwise. This measure will identify if treatment and control group members were employed in each of the 5 quarters following program entry.

• *Likelihood of Retaining Employment* – Two measures of employment retention will be constructed: 1) equals 1 if worker had positive wages in specified consecutive quarters, 0 otherwise, and 2) equals 1 if worker had positive wages in specified consecutive quarters from the same employer, 0 otherwise. These measures will identify whether workers were able to obtain and retain employment following program entry.

• *Quarterly Earnings* – Equals the quarterly wage amounts earned in each of the five quarters after program entry.

• *Earnings Growth* – Equals the change in quarterly earnings in each of the five quarters after program entry. This outcome measures the wage growth experienced by workers following program entry.

• *Industry of Employment* – Equals 1 if the worker found employment in the industry focus of the program, 0 otherwise. This measure will identify whether individuals were employed in the industry focus of the program.

This section outlines the model, the variables to be used in it, and the assumption upon which the model is based. It also indicates that a *treatment-on-treated* approach will be used.

In addition, IMPAQ will rely on WIA and UI data, as available, to construct the following outcomes:

• *Likelihood of UI Receipt* – Equals 1 if worker collected UI benefits following program entry, 0 otherwise.

• *Likelihood of Receiving WIA Training* – Equals 1 if worker received WIA training following program entry, 0 otherwise.

• *Likelihood of Receiving Educational/Training Credential* – Equals 1 if worker received an educational/training credential as a result of WIA or other training, 0 otherwise.

Descriptive analyses of these measures will enable us to observe patterns in the labor market and other outcomes for treatment and control group members in the 15-month period following program entry.

The statistical model used to estimate the program effect is fully specified and all variables in the model (and their coefficients) are defined.

*Impact Analyses*. To estimate the impact of NFWS/SIF programs on participant outcomes, IMPAQ will examine outcome differences between the treatment group (program participants) and the comparison group. These impacts are formally termed the *impacts of the treatment on the treated*. To estimate program impacts with increased statistical efficiency, IMPAQ will use multivariate regression models, which control for available socioeconomic, employment, and other characteristics. The impact analysis regression models can be expressed by the following equation:

*Y = α ∙ T + X ∙ ß + EMP ∙ γ + OTHER ∙ δ + SITE ∙ ε + u*

This model will be estimated on the combined sample of treatment and matched comparison groups. The dependent variable in this model (*Y*) is the participant outcome of interest (e.g., likelihood of employment, likelihood of retaining employment, quarterly earnings). Control variables include the following:

• *T*, which equals 1 if the individual was in the treatment group and 0 otherwise;

• *X*, which includes all available individual socioeconomic characteristics (e.g., gender, race, age, education) and a constant term;

• *EMP*, which includes variables capturing individual employment and wage history (e.g., industry and occupation of prior employment, tenure with prior employer, wages in each of the 8 quarters prior to program entry);

• *OTHER*, which includes variables capturing prior participation in WIA training and receipt of UI benefits, as available;

• *SITE*, which includes NFWS/SIF site characteristics (e.g., site fixed effects, industry focus of training, number/types of services received); and

• *u*, which is a zero mean disturbance term.

The steps in the analysis, and the way in which the evaluation team will examine both the characteristics of the group and the outcomes are clearly related.

The parameter of interest in this model is α, the regression-adjusted treatment effect of the NFWS/SIF program on the outcome of interest. This parameter represents a rigorous estimate of the impact of receiving NFWS/SIF program services. The model will be estimated separately for unemployed and incumbent workers, for each available outcome of interest. To account for differences in the outcomes distribution across sites, IMPAQ will produce robust standard errors clustered by site – this will ensure that standard errors of estimated parameters are accurately estimated. Once IMPAQ estimate these models, standard errors will be used to produce t-tests to determine if the estimated program impact is statistically significant.

How the statistical analysis is aligned such that the unit of analysis corresponds to the unit of assignment is described.

We will also identify if there were differential impacts by key individual characteristics (e.g., gender, age, education). To do so, IMPAQ will include interactions between these characteristics and the treatment indicator. The parameters of these interactions will capture differential program impacts; t-tests will be used to determine their statistical significance. Similar analyses will be performed to assess differential impacts based on site characteristics, including: 1) the industry focus of the training, 2) types of services received, and 3) number of services received.