Excerpted from the SEP for the Latin America Youth Center program, a subgrantee of Venture Philanthropy Partners
An impact study [of the Latin America Youth Center program] will use a random assignment design to compare participants with nonparticipants in terms of academic and employment outcomes, and other areas of life central to the successful transition to adulthood. The study will last 40 months. The Promotores Pathway Model (PPM) impact study will:
1) Provide a detailed understanding of how the PPM works and assess programmatic achievements and challenges;
2) Help explain how and why the PPM does or does not achieve desired impacts on participants; and
3) Improve practice and advance the field’s understanding of effective strategies for reconnecting seriously disconnected youth.
The first several paragraphs identify the unit of randomization and also explicitly relate the randomization procedures that will be used. The details are further explained below. The program group and control conditions are also thoroughly laid out.
For the research design of the impact study, LAYC and P/PV will utilize a random assignment experimental design, commonly referred to as a randomized control trial (RCT). As P/PV writes in their original application for the external evaluation, ‘In a random assignment evaluation, individuals eligible for a program are randomly assigned to a treatment group that is allowed to participate in the program and a control group that is not. Thus, for all intents and purposes the two groups can be seen as identical before participation in the Promotores Pathway.’ Using this design, any differences in outcomes observed between the two groups can be presumed to be caused by participation (or lack of) in the Promotores Pathway.
The intake procedure for the Promotores Pathway creates an ideal situation for randomization of potential participants. Each youth that is referred to the Promotores Pathway is required to go through an intake procedure where demographic information is collected. In addition, each youth will complete LAYC’s Risk Screening Tool, which contains 25 yes/no questions asking about participant behaviors, current educational and employment status, and risky behaviors. Each of the questions and answer categories in the Risk Screening Tool has weights – allowing LAYC to compile a total risk score for each youth. All youth that receive a risk score that is equal to, or exceeds, 3 is at the cutoff point deemed eligible for the Promotores Pathway.
Any concerns that proposed strategies or approaches will lead to nonequivalent groups are discussed.
Once a youth receives a qualifying risk score, they are then required to complete the consent process for the impact evaluation. This includes consenting to the randomization process, agreeing to take all baseline and follow-up surveys, and understanding that they are part of a research study. Those youth that do not consent to the study are placed on a Promotores wait list, and are referred to other LAYC services that they may be eligible for.
The program group and, to the extent possible, the control group conditions are described.
Procedures to conduct the random assignment, including who implemented the random assignments, how the procedures were implemented, and procedures used to verify the probability of assignment groups, are described and generated by random numbers.
If a youth provides his/her consent, then they are entered into the randomization process (commonly referred to as the Promotores lottery). The Director of the Promotores Pathway submits a group of three to five youth two times a week (Wednesday and Friday) to P/PV for randomization. P/PV then randomizes the youth into the control or treatment groups, and the results of the randomization are then provided to the Director.
Treatment group youth are then matched with a Promotor. Control group youth are placed on a Promotores Pathway waiting list, and will remain on the waiting list until the end of the evaluation. No youth on the waiting list will receive services through the Promotores Pathway until the evaluation is completed.
If a youth is randomized to the control group, the Promotor will inform the youth of the result, and at the same time make referrals to other LAYC programs when appropriate. These referrals are based on the results of the Risk Screening Tool. For example, if the youth self-reports a potential mental health problem, the Promotor will refer the youth to LAYC‘s mental health treatment services. However, the Promotor simply makes this referral by providing contact information and does not assist the youth in fulfilling the referral in any way. This is in contrast to how the Promotores work with treatment group youth – where a Promotor will (initially) set up meetings as part of the referral process, and transport (and accompany) youth to the referral programs.
P/PV describes the random assignment procedures in the following way:
‘To further ensure the integrity and consistency of random assignment, and to facilitate monitoring of sample buildup, we recommend that sample members be randomly assigned by a third party. The study team has enlisted the services of Ewald & Wasserman Research Consultants, LLC (E&W), a survey research firm with experience in managing random assignment and tracking and conducting survey interviews with hard-to-reach populations.
LAYC staff will gather required information (including consent, contact information and required data fields from the PPM intake data) and submit that information to E&W once they are sure an individual is eligible and a strong candidate for PPM. E&W staff will review materials for completeness and notify the staff regarding any missing information, illegible notations and other omissions that might compromise the integrity of that respondent‘s intake. E&W will carefully monitor the intake data for both duplicate records due to multiple entries and other criteria that might implicate a reason to exclude a potential participant from the sample.
This paragraph explains the blocking procedure that will be used.
After intake forms have been reviewed and any missing entries obtained, E&W will assign the sample member to one of the two groups, treatment or control. The assignment algorithm will follow a 2:1 ratio. The random assignment algorithm will use blocking to ensure stable assignment ratios for each site (the PPM is offered at three different LAYC locations – DC, Silver Spring, and Langley Park). The algorithm will also verify that an applicant has not already been assigned (for example, in the case that a control group member reapplies) and will result in a string of no more than two control or two treatment group assignments in a row.
Following assignment, E&W will fax or email each sample member’s treatment or control group designation back to LAYC and the research team within 24 hours. At that point, those designated as treatment group members can begin to receive services, and those designated as control members will be notified of their status and informed of alternative options.’
LAYC’s Director of the Promotores Pathway (and only the Director) submits a group consisting of 3-5 youth for randomization two times a week. By submitting youth for randomization in groups, potential threats to the integrity of the randomization are minimized.
The randomization process has worked quite well in reality. LAYC submits potential youth for randomization twice a week to E&W, and the results are provided back to LAYC via e-mail usually within an hour of submission (the maximum length of time between submission and randomization results has been 24 hours). E&W, P/PV, and LAYC all keep master lists of the randomization results for all submitted participants.