Excerpted from the SEP by the National Fund for Workforce Solutions program, a subgrantee of Jobs for the Future
The unit of matching is clearly identified.
The quasi-experimental evaluation study will provide rigorous estimates of the impact of NFWS/SIF programs on participant employment outcomes. Using a quasi-experimental approach, IMPAQ will estimate program impacts by comparing the outcomes of program participants (treatment group) to the outcomes of non-participants who are observationally equivalent to program participants (comparison group).
The steps outline the matching procedures, and greater detail is provided below.
Implementing this approach involves the following steps:
1. Collect state administrative data from states in which the evaluation sites are located
2. Merge participant-level and ES administrative data files, appending demographic and program participation information from the UI and WIA data, as available.
3. Apply matching methods using state administrative data to construct appropriate comparison groups, comprised of non-participants with the same characteristics and resided in the same labor market area as program participants. Comparison groups will be constructed separately for unemployed and incumbent workers.
4. Construct common outcome measures for the treatment and the comparison group members based on state administrative data.
5. Produce rigorous estimates of program impacts through outcomes comparisons between the treatment and the comparison group.
Methods used to form the proposed comparison group are described such that the validity of the matching is explained.
Construct Comparison Groups Using Matching Methods.
One key component of this evaluation is to construct matched comparison groups for unemployed and incumbent workers who are program participants in the NFWS/SIF evaluation sites and who entered the program from April 2011 through February 2012. Our matching approach will ensure that program participants are compared to a sample of non-participants with the same observed characteristics and work history, and who reside within the same labor market area.
Although not included, the following excerpt would be a good place to add a reference from literature.
Matching methods have emerged as a reliable approach for producing rigorous evaluations of workforce programs, particularly when a random assignment design is not feasible. Matching methods rely on the conditional independence assumption (CIA): the outcome (the outcome of the individual not participating in the program) is independent of program participation, controlling for observed characteristics. The implication is that non-participants who are observationally comparable to participants can be used as a comparison group for the evaluation. Matching methods provide credible impact estimates when: 1) the data include large samples of non-participants and 2) matching is performed based on rich information on participant and non-participant characteristics, employment activities over the two years prior to program entry, and local labor market.
The treatment group in this study is program participants in selected NFWS/SIF sites. To construct matched comparison groups, IMPAQ will rely on ES data, which provide rich information on the characteristics and work history of all individuals who sought state employment services. ES is particularly appropriate to identify comparison groups for a number of reasons. First, ES data includes large samples and their numbers will exceed those of evaluation site participants. Second, the ES population includes all unemployed workers seeking employment assistance and incumbent workers seeking training/education services. A matched comparison sample based on the ES data will be nearly observationally identical to program participants; matched comparison groups will be constructed separately for unemployed and for incumbent workers.
In this study, IMPAQ will use the Propensity Score Matching (PSM) method. PSM techniques facilitate construction of a sample of nonparticipants with characteristics that closely correspond to those of program participants. We will implement PSM using the following steps:
Step 1: Merge data – Participant data provided by evaluation sites will be merged with ES and Wage Record data from the state in which sites operate based on participant SSN. The merged data will include all available characteristics and outcomes of participants and non-participants. UI receipt and WIA participation from the UI and WIA data will also be merged with these data, as available.
Step 2: Produce propensity score – We will use a logit model to estimate the likelihood of program participation based on available control variables: 1) socioeconomic characteristics at program entry, work history, and prior services/training participation (ES data), 2) prior wages and employment (Wage Records), 3) prior UI receipt (UI data), and 4) prior participation in WIA training (WIA data). We will then use the model results to produce a propensity score for each participant and non-participant in the data; the propensity score is equal to the predicted probability of program participation based on observed individual characteristics.
Step 3: Use propensity score to match participants with non-participants – Pair-wise matching (one comparison case is matched with each participant) was once the accepted method, but recent work shows that radius matching, where one treatment case is matched to multiple comparison cases, provides the most efficient estimates. We will use radius matching to match participant cases to one or more comparison cases that have identical or nearly identical propensity scores.
This section indicates how internal validity concerns will be addressed in the matching procedure.
Step 4: Test comparison sample and modify specification if necessary – Once matching is achieved, it is necessary to test if participants and comparison individuals share similar characteristics. These tests involve comparisons of variable means and standard deviations between the treatment and the comparison group. If treatment-comparison differences in characteristics are detected, IMPAQ will modify the logit model specification (e.g., include polynomials to capture nonlinearities or multiplicative terms to capture variable interactions) to eliminate such differences and ensure that a successful matching is achieved
There is no content in this SEP example that identifies with the currently selected SEP Guidance Checklist element.